Monthly Archives: August 2013

Tuning Linux (Debian) in a vSphere VM – Part 2 – Virtual Hardware

tuxcloudAlright – so in Part 1 we covered LVM, partitioning and the basic installation of our VM.  Now its time to tackle a few key points as it pertains to virtual hardware.  Now there isn't a whole lot to cover here and honesty I'm just going to graze the surface on some of these topics.  You will find a bunch of links to some KB articles and white papers where you can go pretty deep on some of these topics so feel free to read them as well.  Again, leave all of your feedback and comments in the sections outlined below – it's a learning process for us all πŸ™‚  So, let's get into it!

Use the virtualized hardware!

VMXNET3 right!  Hopefully by now you know of the advantages of using the VMXNET 3 adapter – most of those same advantages apply to Linux as well.  I’d definitely recommend using this.  Since one of the benefits of VMXNET is the RSS or Multiqueuing it makes sense to use it, however there are a few things to keep in mind when utilizing this with the VMXNET3 driver; Any modern Linux kernel will now have it’s own built in VMXNET3 driver/module and for the most part (I think by default) your VM will use it, even if you have VMware Tools installed. – and this is fine.  There are times however where you might want to use the driver which is included with VMware Tools.  If you are running an older Linux kernel then the version of VMXNET3 you have might not support RSS or Multiqueueing, therefore you might want to use the VMware version which does.  To do so you can run the following command to replace the kernel driver with the VMware provided driver from the VMware Tools install package.

./vmware-install.pl –clobber-kernel-modules=vmxnet3

Go head and check out KB 2020567 for a bit more information on this.   Just remember, newer kernel = do nothing, older kernel = run the command above.

Who wouldn't want to be paravirtual?

Hey sounds cool right!  I know I'd want to be paravirtual!  Well your virtual SCSI controller does too!  PVSCSI has been the recommended adapter of choice for a little while now and can provide you with a multitude of benefits such as higher throughput at a lower CPU cost!  That being said, you need to be sure you are on a supported OS list!  When it boils down to it, so long as you are running at least 2.6.33 of the Linux kernel and have the vmw_pvscsi driver you should be ok, but, for those that like to be officially supported, the list is here.

Manage that memory!

memoryIts always best practice to right-size your VMs.  Keeping an eye on what CPU and memory resources are being utilized, knowing your workloads and sizing it appropriately – a Linux VM is no different (Hey, we are VMs too you know).  There is one setting from within a Linux VM that can be changed and tweaked to help though – swappiness!  Yup I said it, swappiness!  As always, open source and Linux never fail to amuse me with their terms and names for things!  So swappiness is essentially an integer value that determines when to move memory pages from physical memory( which is really virtual now, just mapped to physical) to your local swap.  Sounds cool eh but there is a catch!  Linux will move pages to swap even if you have physical memory available and this is completely separated from the fact that vSphere will also do its' own memory swapping if need be.  To cut to the chase a high swappiness value indicates that processes are more likely to to be swapped, whereas a low value indicates the kernel will leave them be.  The default value I've seen the most is 60 – normally I like to change this to somewhere around 15-20.  This is done by executing the following command

Echo 15 > /proc/sys/vm/swappiness

This works great for the current running state but to persist across reboots you will need to add the "Vm.swappiness=15" somewhere inside of /etc/sysctl.conf

Useless Hardware – Get rid of it!

LinuxPerf-Part2-fd0errorDo you honestly ever plan on using that floppy disk?  Didn't think so – get rid of it!  This is a process I try and apply to all my VMs not just the Linux ones.  Unneeded hardware can tie up not only vSphere resources but your OS resources as well.  So in terms of floppy it's not just as simple as just removing it from the VMs settings – you will need to also disable it from the BIOS of the VM.  If not you will always see that crazy floppy error that flashes up when grub is booting (the one on the right).  And even further, remove and blacklist the floppy module and prevent it from even loading into the Linux OS.  If you perform an 'lsmod | grep floppy' you will see that even though you have removed it from the VM and disabled it in the BIOS, the module still loads.  To completely blacklist a module from loading simply add it to a blacklist file in /etc/modprobe.d/ – In the case of the floppy we can execute the following list of commands to remove all traces of that pesky disk!

Echo "blacklist floppy" | tee /etc/modprobe.d/blacklist-floppy.conf
Rmmod floppy
Update-initramfs -u

So, that's it for now!  As with anything you do in production be sure to test it first!  Try different swappiness values, keep an eye on your resource usage and adjust as needed!  If you have any other tips/tricks for a Linux VM please leave them in the comments section below!  I'd love to hear them!  Next up in this series we will look at a bunch of random things like timekeeping, the i/o scheduler, as well as some other modules that we probably don't require when running virtualized!  Stay tuned and thanks for reading!

Part 1 – Installation

Part 2 – Virtual Hardware

Part 3 – /usr/bin/random

Tuning Linux (Debian) in a vSphere VM – Part 1 – Installation

tuxcloudLAMP services, web servers, firewalls, proxies, load balancing; some of the many use cases that the Linux OS along with its' packages delivers.  In fact, you would probably be pretty hard up to walk into any major enterprise and not find at the very least one flavor of a Linux distro running live in production whether it be a physical appliance or a virtual machine.  The latter however is becoming more and more prevalent.  The evolution to a software defined data center is heavily reliant on having those edge services virtualized, those internal web servers and proxies virtualized and that is why I decided to share a little of what I do when virtualizing Linux, mainly Debian and the processes and practices I take to ensure I get the most out of the OS.  So welcome to Part 1 (Installation) of my 3 part series covering all that is virtualizing Linux!  Be sure to share any tips you might have as well in the comments section as you will see that I'm no guru when it comes to Linux – I know enough to get by – but let's make this a community effort.

LVM for your VM?

So first off let me reiterate that I'm no expert when it comes to LVM and/or Linux disk partitioning – in fact this is one of the reasons I decided to include the installation on this series – to learn more.  I had never even used LVM on a Linux VM before writing this post but I’m sure I will now.  You can see below you are presented with a few different options during the disk partitioning process.

LinuxPerf-Part1-1-LVM

Now before going to far into LVM let me first thank Tomi Hakala ( blog / twitter ) and Romain Decker ( blog / twitter ) for giving me a bit of an introduction on what and how LVM is.  I certainly recommend checking out Tomi’s post on LVM here as I’m probably just going to regurgitate a little of what he wrote.   

Whether you decide to use LVM or not really depends on your use case but from all of the reading and research I’ve done I kind of came to the conclusion of when I would use it and it solely depends on whether or not you plan on using separate partitions inside your guest.    Honestly, if you plan on using one disk only in the virtual machine and you are not worried about separate partitions filling up and taking down everything or you don’t have any plans on mounting different partitions with different mount options or file systems then LVM might not be for you.  But if you are worried about things like a single partition taking down the system, expanding partitions on the fly, or want to simply mount /tmp as noexec (explained later) then you might want to consider using LVM.    My normal setup usually has my OS and individual partitions like swap, var, and home on one disk, and then I add another to simply contain data for whatever application it is I’m running.  Therefore, since I’m going to setup some individual partitions and I will want to easily expand them on the fly I’m going to use LVM by selecting ‘Guided – use entire disk and setup LVM' – That being said, when/if I do add the second disk for data I would not use LVM on it since I could easily enough expand it with the native Linux (resize2fs) or gparted tools.

Why can't partitioning just be easy?

LinuxPerf-Part1-2-Partition

Now we are into the partition questions.  As I said before I plan on having separate partitions for /home, /usr, /var, and /tmp – Why?  Well, this way I can set a few different mount options and do it on a per partition basis – meaning, if this was say a web server would I really need to be executing binaries on /tmp – probably not, so by breaking it out into partitions I can mount /tmp with the noexec option to prevent that.  If it was one big partition this option wouldn’t be available to me.  Also, if I planned on selecting ‘All files in one partition’ then I probably would of chose to not use LVM as I could just as easily expand the disk with VMware and resize2fs.  Another advantage of using separate partitions, say an application goes squirrelly and fills up your system with logs in /var – with a /var being mounted separately it will only be that partition that is affected while the others remain functional.  In the case of having only one partition, the whole server would be affected.  So again, for these reasons I’m going to select ‘Separate /home, /usr, /var, and /tmp’.

As mentioned above you may want to mount different partitions with different options as they are laid out in /etc/fstab – Now there are a lot of options (all here) when mounting partitions – too many to go over in this post but here’s a few to get your started

Noatime – The linux file system will keep track of a ton of different things as it pertains to time and an files (creation, modified, accessed).  Certainly when it has to update the access time every single time a file is accessed you can understand that it will generate an extra I/O.  If you don’t need this functionality, mount the partition with noatime and it will not occur.

Noexec/Exec – Disabled the execution of binaries on the specified file system.  Usually done on a /tmp mount in order to harden security.

Sync/Async – determines whether IO is written to disk in a synchronous or asynchronous fashion.

Remember: Any changes to the options in /etc/fstab will not occur until you have either rebooted or remounted the file system in question.

Swap Shop!

LinuxPerf-Part1-3-PartitionOverview

So now we see an overview of how Debian has determined the best way to lay out our partitions.  For the most part I normally just accept these defaults with one exception.  You may want to increase the size of the swap partition.  There are a lot of recommendations that somewhat conflict with each other around this subject.  It used to be that swap should be at least twice the size as the amount of physical memory (IE 256MB RAM would result in 512MB swap partition).   I tend to let this slide a bit but when I can I try to at the very least set the size of the swap partition to the same as the VMkernel swap file, which is the total amount of memory assigned to the VM minus the VMs reservation. (IE – a 4GB VM with a reservation of 3GB would result in a swap file totaling 1GB.)  Best advice, just pay attention to the amount of in-guest swapping that is occurring as well as the amount, if any, of ballooning (if you are heavily over committed) and you can always resize down the road.

The rest of the installation is pretty much your standard Linux install so I’ll leave that up to you…

Watch for Part 2 when we talk about virtual hardware and VMware Tools!  Also again, please leave any suggestions or comments below – they are always appreciated!  

Part 1 – Installation

Part 2 – Virtual Hardware

Part 3 – /usr/bin/random