Tag Archives: ESXi

Tuning Linux (Debian) in a vSphere VM – Part 3 – /dev/random

tuxcloudSo here we are – Part 3 and the final part of the Linux series.  The title of this part is /usr/bin/random because well the content will be indeed quite random.  I couldn't think of a way to classify the content of this post into a single category!  So get ready for a hodge podge of fixes, modifications and configurations that have nothing in common and no similarities whatsoever.  I apologize up front for the flow (or lack thereof) of this post but hopefully someone will find something useful in it.

What time is it?

Almost every OS is very dependent in having accurate time and use different hardware and software techniques to do so.  When something is virtualized this adds yet another layer in between the OS and the hardware and creates some challenges as it pertains to timekeeping.  I’m not going to go through a time keeping lesson – there’s a great VMware whitepaper here that goes very deep into the topic.  Instead I'll just try to summarize what I've learned over the years as it relates to Linux and timekeeping.  Depending on the Linux distribution and kernel version you are running you may need to add some boot parameters to either the grub or LILO menu in order to disable the High Precision Event Timer (HPET) and set the proper clock managers.  Most new releases of Linux distributions (within the last couple of years) don’t require any changes, but for instance, if you are running Debian 4.x you would need to append “divider=10 clocksource=acpi_pm” to your kernel boot line.  For a full list of available options have a look at KB1006427 from VMware.

The I/O Schedule

ESXi has its' own special sauce as it pertains to scheduling disk I/O – and so does Linux.  Linux has some different I/O schedulers built into the OS, such as NOOP (noop), Completely Fair Queuing (cfq), Anticipatory (anticipatory), and Deadline (deadline).  By default, as of kernel 2.6, most Linux distributions use CFQ as their I/O scheduler of choice – not really a problem in the physical world but as a guest OS can cause some performance degradation.  As stated earlier, ESXi has its' own I/O scheduling, so does it really make sense to schedule I/O at the guest OS level, and then at the hypervisor level?  Probably not!  That's why there is a VMware KB article that states to switch your I/O scheduler to noop or deadline.  Honesty I would switch to noop as it does nothing to optimize and disk I/O, which would allow the hypervisor to do its thing.  Here's how!

You can change the scheduler during runtime by echoing to the proper disk, IE for sda we would use

echo noop > /sys/block/sda/queue/scheduler

However, to permanently switch you need to add an elevator switch to your grub kernel entry in grub or your Linux entry in grub2.  The above will reset back to cfq on reboot.  To permanently do this your kernel entry in the grub menu should look similar to the following (menu.lst for Grub and grub.cfg for Grub2).  Below is Grub2.

linux   /boot/vmlinuz-2.6.32-5-686 root=UUID=b62a38cf-8917-484a-9a96-d5a74beb8d59 ro  quiet elevator=noop

Copy the floppy!

In Part 2 we went over how to completely get rid of the floppy drive.  Part of those instructions included blacklisting the floppy module inside of /etc/modprobe.d/ – well guess what?  There are a slew of other modules that are loaded by default that you will probably never use when running a virtualized instance of Linux.  Below is a list of modules that I often blacklist – Sure, there are the one off cases where you will need one or two of these modules loaded so just pick and chose for yourself…

floppy – Take a guess, yup you got it, the floppy drive 🙂

mptctl – This monitors your RAID configuration.  I don't normally RAID inside of my Linux guests so this is really not needed – also this will spam up your messages log quite a bit as well.

pcspkr, snd_pcm, snd_page_alloc, snd_timer, snd, soundcore – These all have to do with sound which I'm not even sure is possible.  Disable them!

coretemp – if you don't care how hot your vCPU's are running you are safe to disable this.  If you do care, then, well, I'm not sure what to tell you 🙂

parport, parport_pc – these have to do with your parrallel ports.  I've never used these and always blacklist them.

Virtual Consoles – Do you even use them?

If you ware wondering how can I even use them inside of vSphere – check out my post here.  If you don't use them, why leave them enabled?  Disabling them is pretty easy, just comment out the tty# lines in /etc/inittab – I always tend to leave the lines in the file and just place a '#' in front of the ones I don't need – below you can see an example of my initttab files.  As you can see I left one console activated.

1:2345:respawn:/sbin/getty 38400 tty1
2:23:respawn:/sbin/getty 38400 tty2
#3:23:respawn:/sbin/getty 38400 tty3
#4:23:respawn:/sbin/getty 38400 tty4
#5:23:respawn:/sbin/getty 38400 tty5
#6:23:respawn:/sbin/getty 38400 tty6

So that's really all I can think of that in terms of tips and tricks that I do when deploying a Linux guest on vSphere.  I'm sure there are many others.  Linux and vSphere both have a ton of advanced settings and are very flexible when it comes to tuning for performance so if you know of any that I've missed I'd love to hear about them, just leave them in the comments box below.  By no means am I a Linux expert and I'm always eager to learn 🙂  Thanks for reading!

Part 1 – Installation

Part 2 – Virtual Hardware

Part 3 – /usr/bin/random

Some updates to the PowerCLI Shutdown VMs Script – Now w/ a virtualized vCenter option

shutdown-button-mdOver that past few months I've been getting quite a few emails and comments as it pertains to the scripts I have shared to shutdown and power on VMs in the event of power loss.  I assume since the storm weather is kicking in people are realizing that they need to have some kind of protection in regards to gracefully getting the VMs down.  Now I've had the (privilege?) of watching my script run multiple times in a production environment and it has gotten to the point where it is working flawlessly.  One caveat to the script though – whichever Windows box is executing the PowerShell must stay powered on longer than your vCenter Server.

That being said the script would be pretty useless if you had a virtualized vCenter Server running on the same cluster as your production VMs that your are shutting down – this is what all of the emails and comments that I've been getting have been based around.  So, without further ado I've added a few updates to the script to better support those with a virtualized vCenter.  A few explanations below…

So first off (Line 50) I've added a variable called $vCenterVMName and assigned this a default value of NA.  This variable should hold the VM Name of your vCenter Server.  I've tried to keep just one version of the script supporting both virtualized and non-virtualized vCenter's thus is why the default to "NA".  So if you don't have a virtualized vCenter just leave the variable set to NA and the script will run as it always had, otherwise if you do, put its' VM name in here.

Secondly down on Lines 107 to 1118 I do a few things.  First, check to see if you have changed that default value and if you have we need to figure out which ESXi host vCenter is running on in order to shut it down last.  This is what's done here.

Then, in each of the shutdown loops (Line 125 and Line 162) I've added an if statement.  Basically it just checks to see if the VM it is processing is the vCenter Server.  If so, it doesn't process the shutdown.

Lastly I've done a couple more things.  On Line 186 I've added a check while shutting down the hosts.  If the host that is being processed is the host that vCenter is running on I want to leave it till the end, so it is skipped during this loop.  Then, the finally the block of code starting on Line 196, if needed, will send the shutdown command to the host that is running vCenter Server.

As you may of noticed I've added a some logging as well, nothing bit, just a record of what happened in a txt file rather than relying on Write-Host.  The script in its' entirety is below or you can download it here.  Watch for updates to the power on script to support the vritualized vCenter as well.  Just not sure how I'm going to tackle that one yet 🙂

# Power off VMs (poweroffvms.ps1)
# This is a complete update to my previous shutdown script.  No longer do we
# loop through the esx hosts and shut down VMs per host.  We now just shut them
# all down up front.  Also, no more $vmstoleaveon variable.  I'm assuming here 
# that your vCenter is not virtual.  If it is you can most likely grab version
# 1 of this script and take the code to leave the vCenter host till last.  Maybe
# someday I'll get around to updating it to merge the two...but for now, this is
# it!
# This script does have a 'partner' script that powers the VMs back on, you can
# grab that script at https://blog.mwpreston.net/shares/
# Created By: Mike Preston, 2012 - With a whole lot of help from Eric Wright 
#                                  (@discoposse)
# Variables:  $mysecret - a secret word to actually make the script run, stops 
#                         the script from running when double click DISASTER
#             $vcenter - The IP/DNS of your vCenter Server
#             $username/password - credentials for your vCenter Server
#             $filename - path to csv file to store powered on vms
			  used for the poweronvms.ps1 script.
#             $cluster - Name of specific cluster to target within vCenter
#             $datacenter - Name of specific datacenter to target within vCenter
# Usage: ./poweroffvms.ps1 "keyword"
#        Intended to be ran in the command section of the APC Powerchute Network
#        Shutdown program before the shutdown sequence has started.
Add-PSSnapin VMware.VimAutomation.Core
#some variables
$vcenter = "vcenterServer"
$username = "user"
$password = "password"
$cluster = "clusterName"
$datacenter = "datacenterName"
$filename = "c:\path\to\poweredonvms.csv"
$mysecret = "kerfuffle"
$date = ( get-date ).ToString('yyyyMMdd')
$logfile = New-Item -type file "C:\ShutdownLog-$date.txt"
#for use with a virtualized vCenter - bring it down last :) - if you don't need this functionality
#simply leave the variable set to NA and the script will work as it always had
$vCenterVMName = "NA"
Write-Host "A Powerchute network shutdown command has been sent to the vCenter Server." -Foregroundcolor yellow
Write-Host "This script will shutdown all of the VMs and hosts located in $datacenter" -Foregroundcolor yellow
Write-Host "Upon completion, this server ($vcenter) will also be shutdown gracefully" -Foregroundcolor yellow
Write-Host ""
Write-Host "This script has also has a counterpart (powervmsbackon.ps1) which should" -Foregroundcolor yellow
Write-Host "be setup to run during the startup of this machine ($vcenter). " -Foregroundcolor yellow
Write-Host "This will ensure that your VMs are powered back on once power is restored!" -Foregroundcolor yellow
Sleep 5
Add-Content $logfile "mwpreston.net PowerOff Script Engaged"
Add-Content $logfile ""
if ($keyword -ne $mysecret) 
    Write-Host "You haven't passed the proper detonation sequence...ABORTING THE SCRIPT" -ForegroundColor red
    Add-Content $logfile "You haven't passed the proper detonation sequence...ABORTING THE SCRIPT"
#connect to vcenter
Write-Host "Connecting to vCenter - $vcenter ...." -nonewline
Add-Content $logfile "Connecting to vCenter - $vcenter"
$success = Connect-VIServer $vcenter -username $username -Password $password
if ($success) { Write-Host "Connected!" -Foregroundcolor Green }
    Write-Host "Something is wrong, Aborting script" -Foregroundcolor Red
    Add-Content $logfile "Something is wrong, Aborting script"
Write-Host ""
Add-Content $logfile  ""
#Get a list of all powered on VMs - used for powering back on....
Get-VM -Location $cluster | where-object {$_.PowerState -eq "PoweredOn" } | Select Name | Export-CSV $filename
#change DRS Automation level to partially automated...
Write-Host "Changing cluster DRS Automation Level to Partially Automated" -Foregroundcolor green
Get-Cluster $cluster | Set-Cluster -DrsAutomation PartiallyAutomated -confirm:$false 
#change the HA Level
Write-Host ""
Write-Host "Disabling HA on the cluster..." -Foregroundcolor green
Write-Host ""
Add-Content $logfile "Disabling HA on the cluster..."
Add-Content $logfile ""
Get-Cluster $cluster | Set-Cluster -HAEnabled:$false -confirm:$false 
#get VMs again (we will do this again instead of parsing the file in case a VM was powered in the nanosecond that it took to get here.... :)
Write-Host ""
Write-Host "Retrieving a list of powered on guests...." -Foregroundcolor Green
Write-Host ""
Add-Content $logfile "Retrieving a list of powered on guests...."
Add-Content $logfile ""
$poweredonguests = Get-VM -Location $cluster | where-object {$_.PowerState -eq "PoweredOn" }
Add-Content $logfile "Checking to see if vCenter is virtualized"
#retrieve host info for vCenter
if ($vcenterVMName -ne "NA") 
    Add-Content $logfile "vCenter is indeed virtualized, getting ESXi host hosting vCenter Server"
    $vCenterHost = (Get-VM $vCenterVMName).Host.Name 
    Add-Content $logfile "$vCenterVMName currently running on $vCenterHost - will process this last"
    Add-Content $logfile "vCenter is not virtualized"
Add-Content $logfile "Proceeding with VM PowerOff"
Add-Content $logfile ""
#and now, let's start powering off some guests....
ForEach ( $guest in $poweredonguests ) 
    if ($guest.Name -ne $vCenterVMName)
        Write-Host "Processing $guest ...." -ForegroundColor Green
        Write-Host "Checking for VMware tools install" -Foregroundcolor Green
        $guestinfo = get-view -Id $guest.ID
        if ($guestinfo.config.Tools.ToolsVersion -eq 0)
            Write-Host "No VMware tools detected in $guest , hard power this one" -ForegroundColor Yellow
            Add-Content $logfile "$guest.Name - no VMware tools, hard power off"
            Stop-VM $guest -confirm:$false
           write-host "VMware tools detected.  I will attempt to gracefully shutdown $guest"
           Add-Content $logfile "$guest.Name - VMware tools installed, gracefull shutdown"
           $vmshutdown = $guest | shutdown-VMGuest -Confirm:$false
#Lets wait a minute or so for shutdowns to complete
Write-Host ""
Write-Host "Giving VMs 2 minutes before resulting in hard poweroff"
Write-Host ""
Add-Content $logfile ""
Add-Content $logfile "Waiting a couple minutes then hard powering off all remaining VMs"
Add-Content $logfile ""
Sleep 120
#Now, let's go back through again to see if anything is still powered on and shut it down if it is
Write-Host "Beginning Phase 2 - anything left on....night night..." -ForegroundColor red
Write-Host ""
#get our list of guests still powered on...
$poweredonguests = Get-VM -Location $cluster | where-object {$_.PowerState -eq "PoweredOn" }
ForEach ( $guest in $poweredonguests ) 
    if ($guest.Name -ne $vCenterVMName)
        Write-Host "Processing $guest ...." -ForegroundColor Green
        #no checking for toosl, we just need to blast it down...
        write-host "Shutting down $guest - I don't care, it just needs to be off..." -ForegroundColor Yellow
        Add-Content $logfile "$guest.Name - Hard Power Off"
        Stop-VM $guest -confirm:$false
#wait 30 seconds
Write-Host "Waiting 30 seconds and then proceding with host power off"
Write-Host ""
Add-Content $logfile ""
Add-Content $logfile "Processing power off of all hosts now"
Sleep 30
#and now its time to slam down the hosts - I've chosen to go by datacenter here but you could put the cluster
#  There are some standalone hosts in the datacenter that I would also like to shutdown, those vms are set to
# start and stop with the host, so i can just shut those hosts down and they will take care of the vm shutdown ;)
$esxhosts = Get-VMHost -Location $cluster
foreach ($esxhost in $esxhosts)
    if ($esxhost.Name -ne $vCenterHost)
        #Shutem all down
        Write-Host "Shutting down $esxhost" -ForegroundColor Green
        Add-Content $logfile "Shutting down $esxhost"
        $esxhost | Foreach {Get-View $_.ID} | Foreach {$_.ShutdownHost_Task($TRUE)}
#finally shut down the host which vCenter resides on if using that functionality
if ($vcenterVMName -ne "NA") 
    Add-Content $logfile ""
    Add-Content $logfile "Processing $vcenterHost - shutting down " 
    Get-VMhost $vCenterHost | Foreach {Get-View $_.ID} | Foreach {$_.ShutdownHost_Task($TRUE)} 
Add-Content $logfile ""
Add-Content $logfile "All done!"
#that's a wrap

So there you have it!  Hopefully this will help some of you out with your shutdown endeavors.  Feel free to make any changes you need in order to make it work for your environment and let me know how it goes – keep those comments, concerns, thoughts and emails coming in!

vSphere USB Passthru and Autoconnect Devices and PowerCLI

usbWait!  I thought I had UPS plugged into my host and setup as passthru to my VM already!  Why can't I see it now?  What happened?  Who moved that external drive I had connected to my Veeam console to seed an offsite backup?  Ever find yourself asking yourself any of these questions…I certainly have!  Due to circumstances out of my control I have a few hosts that tend to be "out in the wild".  Available and insecure, readily accessible to the hundreds of people walking by it each day.  At time to time either someone trips over a cord, some deliberately unplugs something, or equipment needs to be moved and gets plugged in to different ports upon reconnection.

Not as many options in the GUI

As is with most other products, using the GUI to configure something sometimes doesn't give you all the options that you need.  Essentially when configuring USB Passthru to a VM from within either the vSphere Client or the vSphere Web Client you the device needs to be plugged in and it gets assigned to the VM based on the host USB port that it is connected to.  Again in most cases this is fine but in my situation I needed to be able to have this device connected to the VM no matter what port it was connected to.  Turned out after reading some documentation around the vSphere API as well as having a great discussion with Luc Dekens on the VMTN forums there is indeed a way to do exactly what I needed to do.

Community to the rescue!

So, in the API reference for the VirtualUSBUSBBackingInfo object it states the following

To identify the USB device, you specify an autoconnect pattern for the deviceName. The virtual machine can connect to the USB device if the ESX server can find a USB device described by the autoconnect pattern. The autoconnect pattern consists of name:value pairs. You can use any combination of the following fields.

  • path – USB connection path on the host
  • pid – idProduct field in the USB device descriptor
  • vid – idVendor field in the USB device descriptor
  • hostId – unique ID for the host
  • speed – device speed (low, full, or high)

Perfect, this is exactly what I was looking for.  Basically I can have the USB device autoconnect to the VM by initially connecting by using ANY combination of the above parameters as the deviceName.  First off, path is out of the question, it's what is going to changed when the device is plugged into a different port.  So I decided to use pid, vid, and hostId.  Therefore, if a device with the specified product id and vendor id is plugged into a host with the specified hostId it will automatically pass this through to the VM in which I assign it to!   Awesome!  One problem, I still don't have a clue what the pid, vid, and hostId are, nor do I know the PowerCLI syntax to add the device.

PIDS and VIDS and More…

So how do you find out the pid and vid of the device you want to add?  Well, there's a KB for that…kinda, KB1648 mentions how to do it.  Basically go to in the Windows registry HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\USB\ and search around and you'll find it 🙂  In my case i was using an APC SmartUPS 1500 which had a pid and vid of 0003 and 051D respectively.

The final piece of information we need is the hostId.  By hostId the documentation really means the actual hardware id that is stored within the BIOS of your host.  Accessing ExtensionData.Hardware.Systeminfo.Uuid will actually retrieve this for you.  As you will see in the script below there is certainty some formatting issues that need to be resolved with it, but it most certainly works 🙂

Putting it all together

Now that we have all the information we need it's time to fire up PowerCLI to get this task scripted (I needed to do it 40 times 🙂 ).  I'm not going to lie here, I had no idea how to do this with PowerCLI but by using Onyx I got a great start.

As you can see in the script below a few things happen.  First you need to specify some credentials to and your vCenter location as well as the pid, vid, hostname of the host that will be connected to the device as well as the VM name in which you want to pass the device through.  Lines 17 and 18 get some information in regards to the target VM and lines 21 and 22 get that hardware uuid from the host.  Lines 24 through 36 is the reformatting of the uuid that I described earlier.  You can see the format in the API reference.  The rest of the script does the actually setup of the USB device, this is the code, minus a few changes here and there, that Onyx spit out for me.

The Script

  1. Add-PSSnapin VMware.VIMAutomation.Core  
  3. $vcserver = "vcenterserver"  
  4. $vcusername = "username"  
  5. $vcpassword = "password"  
  7. $myhost = "Host that USB is attached to"  
  8. $myvm = "VM which to pass USB through to"  
  10. $ppid = "PID of USB Device"  
  11. $vvid = "VID of USB Device"  
  13. Connect-VIserver $vcserver -user $vcusername -pass $vcpassword  
  16. #get id of VM  
  17. $vm = get-vm $myvm  
  18. $vmid = $vm.ID  
  20. #get host uuid from BIOS  
  21. $vmhost = get-vmhost $myhost  
  22. $vmhostId =  $vmhost.ExtensionData.Hardware.SystemInfo.Uuid  
  24. #reformat vmhostID to the proper format for autoconnect string  
  25. $vmhostd = $vmhostid.Replace("-","")  
  26. $section1 = $vmhostid.substring(0,16)  
  27. $section2 = $vmhostid.substring(16)  
  28. $newsec1 = (&{for ($i = 0;$i -lt $section1.length;$i += 2)  
  29.    {  
  30.      $section1.substring($i,2)  
  31.    }}) -join '\ '  
  32. $newsec2 = (&{for ($i = 0;$i -lt $section2.length;$i += 2)  
  33.    {  
  34.      $section2.substring($i,2)  
  35.    }}) -join '\ '  
  36. $hostId = "$newsec1-$newsec2"  
  39. #create usb device and add it to the VM.  
  40. $spec = New-Object VMware.Vim.VirtualMachineConfigSpec  
  41. $spec.deviceChange = New-Object VMware.Vim.VirtualDeviceConfigSpec[] (1)  
  42. $spec.deviceChange[0] = New-Object VMware.Vim.VirtualDeviceConfigSpec  
  43. $spec.deviceChange[0].operation = "add"  
  44. $spec.deviceChange[0].device = New-Object VMware.Vim.VirtualUSB  
  45. $spec.deviceChange[0].device.key = -100  
  46. $spec.deviceChange[0].device.backing = New-Object VMware.Vim.VirtualUSBUSBBackingInfo  
  47. $spec.deviceChange[0].device.backing.deviceName = "pid:$ppid vid:$vvid hostId:$hostId"  
  48. $spec.deviceChange[0].device.connectable = New-Object VMware.Vim.VirtualDeviceConnectInfo  
  49. $spec.deviceChange[0].device.connectable.startConnected = $true  
  50. $spec.deviceChange[0].device.connectable.allowGuestControl = $false  
  51. $spec.deviceChange[0].device.connectable.connected = $true  
  52. $spec.deviceChange[0].device.connected = $false  
  54. $_this = Get-View -Id "$vmid"  
  55. $_this.ReconfigVM_Task($spec)  

So there you have it!  You can unplug and plug this USB device in to your hearts delight.  ESXi should pick up the device no matter what port it is plugged into and pass it on to your VM every time!  Certainly this isn't something that you will do everyday, but for those that have hosts sitting out in the open, may be a handy configuration to have set up in their environment.  As always, I'm not the greatest scripter in the world, so any comments, suggestions, improvements, concerns, thoughts are most definitely appreciated.

My wishlist for the vSphere Web Client .Next

Genie-1First off I want to state that by no means do I have anything to do with VMware and/or the development of any of the following.  Also, VMware has nothing to do with me and to my knowledge are not working on any of the following items, but you never know 🙂

So with that out of the way lets get started.  I’ve had enough time to use the web client provided with vSphere 5.1 to know that it is most certainly a powerful tool.  As of late I’ve been experimenting with adding my own vCenter Orchestrator workflows and executing these from the Web Client – This is, along with some other features of the web client that I have blogged about (tagging and searching),  very cool and very nice to have.  That being said, I’m a geek, and I’m never satisfied, always wanting more – so here’s what I call a wishlist for the WebClient.Next.

vCOps / Capacity IQ overprosioned VMs identification.

vm-elephantAlthough in many environments almost every single VM may be over provisioned I think it would still be nice to be able to quickly identify those that are heavily over provisioned (maybe by a metric? that the user defines).  So maybe the elephant is a bit much, but to have the VM icon change, or an alarm raised directly within the web client may help to drive home the importance of right sizing your VMs.  It may expose this information to more people, more application owners that may have access to vCenter but not vCOPs.  But hey, why stop there?  Why not make some recommendations when editing the said VMs settings on where that “right-size” is.  Maybe another little row on the RAM selection to identify vCOps recommendations, or maybe a simple note in the CPU selection window.


More vCOps integration

vcopsreports2I’m not going to lie, it has been fabulous being able to view those vCOps badges from within the web client.  The less I need to go to another tab the better 🙂  That being said I want more!!!  It would be nice to have some of the vCOps reports available from within the web client and mapped to inventory items contextually….  Just as we can now start a vCenter Orchestrator workflow by right-clicking on a cluster it would be nice to generate a capacity report in a similar fashion.


More vendor plugins

plugObviously this one isn’t on VMware, I believe that the foundation is already there, but I’d love to see more and more vendors and partners take advantage of it and develop plugins for the web client.  For instance, those backup vendors that have had integration within the c# client, as well as those that haven’t had any integration at all.  I can certainly see advantages to right clicking a VM and being able to directly add it to a backup job.  I understand that vDP already has this, but there are a lot of people using third party solutions….


The longer I wait to publish this the more of my items are being delivered.  Veeam has just announced their support and integration into the vSphere Web Client in the next release of Backup and Replication.

Update Manager and Single Host Management

Yup, this is one that I’m positive is on the list for almost everyone I know.  We certainly need a way to perform those Update Manager scans and remediations from within the Web Client.  Having to switch back and forth between clients, well, how do I put it, kinda sucks!  Also the ability to manage a single host.  I’m sure we are not the only customer in the world that has those one-off hosts that are not connected to a vCenter – it would certainly be nice to be able to have the same interface available to manage these as well.  This one I would assume is a shoe in as there will be no more C# client next release.

Tags into PowerCLI

powerclilogoOK OK, this one really has nothing to do with the web client but still, something I wanted to stress.  The ability to assign tags to inventory items in vSphere 5.1 is one of the most useful and practical features of the release in my opinion.  It’s easier to sort through massive amounts of VMs, tag them by service, location, sla, however you really want.  Certainly granting the ability to tag and retrieve VMs by tag from PowerCLI can only complement that feature as there are a lot of scripts I find myself wanting to write, but being held back by the absence of and tag cmdlets.


Speaking of Tags and Update Manager

Wouldn’t it be nice to remediate hosts based on how they have been tagged?  Hmm, OK, well, maybe that isn’t that useful…BUT, what would be cool is having the ability to schedule certain VMware Tools/Compatibility updates based on  a virtual machine tag.  I’ve had countless times where I’ve needed to move certain VMs into a temporary folder in order to have them participate in a VMware tools upgrade that following night, then, move them back and try to remember where they were. By simply tagging these VMs and running my scheduled tasks against the tags would make this process a little easier.

I’m sure I had more, however over the weeks I’ve been compiling this list I’ve just forgotten to write some of them down…  With that said, is there anything you are looking for in vSphere.Next?  Comments, concerns, opinions, rants, all are welcome below….

Unitrends Enterprise Backup – In the lab!!!

Unitrends has been inside the backup industry for over 20 years!  That’s a long time to be a key player inside of one industry.  How do they do it?  By staying on top of the ever so changing nature of backup.  That’s why I decided to do this sponsored review of Unitrends latest offering, Unitrends Enterprise Backup.   Again, this review is sponsored but it most certainly contains my words and thoughts about the product.  Now that that’s been cleared up let me say that they make it pretty easy for you evaluate the product yourself.  Personally I went and grabbed a free NFR license that they offer to all Microsoft and VMware certificate holders.  Even if you simply belong to a VMUG you can get yourself an NFR license.  While the Unitrends Enterprise Backups deployment route requires Hyper-V or VMware, the appliance itself can backup and protect both your physical and virtual environment, but for the purpose of this review I only tested on a virtual infrastructure.


This couldn’t have been easier!!!  Unitrends has taken the virtual appliance route when it comes to installation, meaning a simple File->Deploy OVF Template and following a quick step wizard is all you really need to do to get Unitrends into your environment.  This honestly took around 10 minutes from start to finish to deploy into my environment.   Just be sure you meet the minimum system requirements (100GB free space, 2 CPUs, 4GB RAM) and deploy the appliance.  There is one single appliance to deploy whether you plan on using the free, trial, NFR, or complete enterprise solution.


Once the appliance has been deployed into your environment it’s now time to get it configured and ready to go.  Like most virtual appliances you must first configure IP and DNS settings inside of the VM console.  Unitrends provides a nifty little console menu to simplify this operation.  Once you have an IP configured the rest of the configuration can be done through the web interface by opening up a browser and pointing to the IP of your Unitrends appliance.  Upon the first login to the appliance Unitrends displays a setup wizard which takes you through most all of the steps that you need to configure to get up and running.  The steps of this process are outlined below

  • Accept EULA
  • Set Date/Time Parameters
  • Configure Hostname
  • Configure SMTP server
  • Change default console authentication (root password)
  • Add any additional administrative users to the system
  • Select an installation type – Just a note here you will have the option to either install as a local backup system or as a vault.  The difference being that vault actually acts as a replication target for a local backup system, giving you the ability to replicate your locally backed up files to another Unitrends installation in an offsite location.
  • Add additional storage to the Unitrends system.  If you left all of the defaults when deploying the defaults you should have roughly 80GB of storage to use as a backup target, here you can add additional storage to the appliance.
  • Install any required agents – If you are solely using this product to backup VMware VMs (or strictly current Windows OSes or Hyper-V) then this step can be skipped, however if you plan to backup any Linux or Mac OS based PHYSICAL systems then you have the option to deploy the agents needed at this time.  As with most settings you can always do this later as well – I skipped this for the time being.
  • Add Client – In Unitrends terms a client is a server/computer that you want to protect.  For the sake of protecting a VMware environment a client will be either your vCenter Server or ESXi host.  Here is where I added my vCenter Server by providing the DNS name as well as credentials then clicking ‘Setup’.  (Shown below)
  • Deduplication Options – You have the option here to disable the Unitrends software deduplication if you happen to be using a deduplication appliance or have deduplication enabled on your underlying storage array.

Honestly at this point you have all the configuration you need to start protecting your virtual environment with Unitrends.  Obviously there are a ton more settings and tweaks you can set in the Settings section of the application but I will concentrate more on the core functionality of the product rather than dive into each and every configuration setting.  Unitrends lays this functionality out in their top navigational menu, thus the rest of this review will follow the same structure.


The status section is there to do just what you would think, display the past, present, and future status of your backup system.  Not a whole lot to say about this section other than there are some really nice visualizations that let you see your future schedule for backups as well as the outcomes from previous backups.


Ok, here we go now into the meat and potatoes of this product; its’ ability to backup your virtual infrastructure.  First off you will see a couple of tabs along the top after selecting Backup; 1-Time Backup and Schedule Backup.  As you may have guessed one option performs a 1-time backup and the other allows you to assign a schedule.  For the purposes of this review I will only go over the scheduled backup option.
The first step is to setup a schedule or modify an existing one.  In my case I called this RPO 6 hours and set the Incremental Forever options to occur every 6 hours (brilliant eh? 🙂 )  Since I have selected my vCenter Server from the navigational menu i can see a list of potential VMs to protect in the list box to the left.  This list is dependent and will change based on the item you select in the navigational menu, meaning if it was an Exchange server you may see storage groups or databases, SQL you would see databases, etc…  Again in our case we are dealing solely with VMware so I see a list of VMs.  One cool and useful feature is the ‘Auto Include New VMs’ option.  This allows you to automatically include new VMs into a specified schedule as they are created and registered with vCenter without physically configuring the Unitrends appliance.  A very cool feature to be able to protect those VMs that tend to just show up in your environment.
The Schedule column on the right hand side of the screen is where you define how and when you want your backups to occur.  You will see here options around the minimum and maximum number of days to retain your backups as well as the backup strategy you wish to deploy.  This strategy could be one of the following….
  • Incremental Forever – This strategy essentially performs a full backup on the target and then subsequently performs incremental backups from then on.  Periodically (I think once every two weeks) the original full backup is synchronized and brought up to date using the  incremental backups locally and differentials are generated for the retention points leading up to this.  A great strategy to obtain a near continuous data protection strategy and speed up the time it takes to backup a virtual machine.
  • Full with Incrementals – Basically the same as Incremental Forever however giving you the ability to specify when the full backup occurs (hourly, daily, weekly, monthly).  I believe in this option the full backups are performed on your production environment and not generated from incrementals.
  • Full with Differentials – This would be somewhat the reverse of Full with Incrementals meaning you will always have an up to date full backup, with differential files as your restore points.
  • Custom – you guessed it, play with it and tailor it to how you want it to be.

And guess what?  You’re done!  Depending on the configuration you’ve specified your backups and retention policies should kick off immediately and start protecting your VMs.  The process of creating a backup job or schedule inside of Unitrends is very very simple.  What I like most about this process is how it is laid out in the web interface.  With the exception of advanced settings (SMTP and storage) all of the configuration and setup is done on one simple screen (See below).


So far we have installed, setup, and configured some backups jobs with Unitrends.  Now is the time to put it to the test and do perhaps the most important function that backup solutions provide; the ability to restore.  Unitrends offers a couple type of restores within their product; Restore a computer or Restore a system.  Now these terms sound like they do essentially the same thing but they most certainly do not.  Your most common type of operation and traditional restore is handled through restore a computer.  This will restore a VM from your local backup instance.  The ‘Restore a System’ is actually a DR operation that will restore your system from either an archive or a replicated (vaulted) instance.  I’ve mentioned the vault options earlier in this review and due to limited resources I can’t efficiently test this so for the case of this section I’ll walk through restoring a single VM from Unitrends.
First off if you select your vCenter in the navigation menu while in the restore section you should be presented with a nifty little screen.  What I like most about this is it is very easy to select your targeted restore time through either the visualizations provided or the Recovery Point Times table.  You then have a couple of options on how to restore.
Firstly is the ‘Next (Select Options)’ button.  This is the route to take if your desire is to restore a complete VM.  Once moving into the next screen you simply need to select a desired host and datastore, give your restored VM a name and you are off to the races.  Another noteable and huge feature is Instant Recovery.  This is the ability to publish the backup files directly to an ESX host and run your VM directly from the backup files themselves.  An awesome feature that you may need to use if access is needed immediately.
The second option, ‘Restore Selected Files’  allows you to perform a file level restore.  In order to first do this you must click the ‘Create’ button in order to create an image to restore from.  After this process is complete a network path will be displayed allowing you to connect by a couple of different methods.  Through Windows you could simply browse or map to the selected path.  Unitrends will also prevent this image as an iSCSI target allowing your end users to connect and restore any files they wish to an location they wish.  Once completed be sure to click the ‘Tear Down’ button in order to unmount the image.


So now we are on to the Archive capabilities of Unitrends.  In order to use the archive features you need to have archive storage attached to your appliance.  I facilitated this by adding a second disk to the VM and configuring it under the Storage and Retention settings however it appears you can also use a NAS or iSCSI type solution as well.  So basically the archive capabilities allow you to take either all or portions of your local backups or local appliance directories and setup different retention schedules on them in order to preserve certain recovery points in a different location.
Again like most configuration that is done within Unitrends all of it is on one screen.  You can see from the screen shot that all that needs to be done is selecting what to archive, the type of backups (a lot of options in here), your archive target and some simple options on whether to compress and encrypt your archived data.  You also have the ability to test your archive settings before submitting the archive job in order to ensure you have enough space remaining on your targets.  Archives can be run both on demand as well as scheduled.  There targets can consist of attached vmdks, iSCSI targets, NAS devices as well as tape drives.


Although fairly self explanatory I wanted to touch on this section as I found it to be one of the real shining points around Unitrends Enterprise Backup.  As you can see in the screenshot below Unitrends has a ton of predefined reports that you can generate and view dealing with everything from the outcomes of your backup jobs to your deduplication savings to storage utilization to inidividual reports dealing with applications such as SQL server.


So there you have it!  That in a nutshell is Unitrends Enterprise Backup.  I can’t say that I have really truly done this application justice as there are a lot of features that I didn’t get a chance to test.  Outside of this review Unitrends contains many features such as not only backing up your virtual infrastructure but your physical presence as well all from one interface.  They have solutions for full application item recovery such as Exchange mailboxes and SQL databases and tables.  Unitrends has a very solid, scaleable, easy to use system.  One of the biggest high points for me is just the ease of use.  Having most all of the configuration options for setting up backup jobs on one screen is a great feature allowing administrators to see the full scope of options and settings all at once.  Another awesome thing about Unitrends is the free edition.  Although limited to 4 VMs you get a fully functional product – no catch!  Don’t believe me?  Try it for yourself.   Go grab a fully functional trial at Unitrends.com.

Customize the VMware vSphere Health Check to only email report if there is a hardware issue.

One of the things that I love most about utilizing vCenter Server to manage my hosts is the ability to generate a nice little email alert when any hardware within my hosts decides to go a little offside.  That being said there are a lot of times when I don’t have vCenter and am trying to monitor hardware failures on the free ESXi Hypervisor.  This as always been a challenge of some sorts and usually the solution that I end up using is William Lam’s vSphere Health Check Report on top of the vSphere Management Assistant (VMA).

The vSphere Health Check report is an awesome perl script written by William Lam that generates a very thorough report containing almost everything you would need to know about a host and the VMs that are residing on it.  After the script is done running you get a nice little email such as the following….

As you can see there is a slew of information included in the health check report – and don’t take this the wrong way, this is all great information and very nice to have as a little report waiting for you in your inbox every morning  Howerver, the goal I had was to take this script, add it to a cronjob on my VMA, and modify the code so that I only get an email and the attached report if I have a hardware issue.  So to get started go ahead and get your VMA setup if you havn’t already (I think you can use any version of the remote CLI if you don’t have VMA) and go and download version 5.02 of the VMware vSphere Health Check report (The instructions to configure and install it are on the download page as well).

So, now with the customization, around Line 289 (your line numbers may differ as I’ve been in there hitting enter :)) I’ve added a few variables.  $HOSTISSUES is simply going to be a boolean variable that we will flag as yes if we run into any warnings or errors when parsing the hardware health and $emailMessage is a variable that will house those issues so we can place them directly inside the body of the message.

my $VM_TOOL="yes";
my $VMW_APP="yes";
my $VPX_SETTING="yes";

#mwpreston - additions 
my $HOSTISSUES="no"; 
my $emailMessage="";


So on we go into the meat and bones of the script.  Around line1930 or so you should see where the script actually goes in and checks the health of the hosts.  Basically what I have done here is if the script finds any issues at all from within this section (anything that isn’t green) I will flag $HOSTISSUES to yes and append the issues to $emailMessage.   I’ve highlighted each spot below where I have added code but you can chose to add it where you like, meaning if you don’t care about a memory error, don’t include it there, etc…

if($HOST_HEALTH eq "yes") {
	if($local_host->runtime->healthSystemRuntime) {
		if($local_host->runtime->healthSystemRuntime->hardwareStatusInfo) {
			my $hardwareStatusInfo = $local_host->runtime->healthSystemRuntime->hardwareStatusInfo;
			my ($cpuInfo,$memInfo,$storageInfo);
			$healthHardwareString .= "<tr><th align=\"left\">".$host_name."</th></tr>\n";
			my ($sensor_health_color,$sensor_health) = ("","");;

			if($hardwareStatusInfo->cpuStatusInfo) {
				$cpuInfo = $hardwareStatusInfo->cpuStatusInfo;
				foreach(@$cpuInfo) {
					$sensor_health = $_->status->key;
					if ($sensor_health =~ m/green/i) { 
						$sensor_health_color="<td bgcolor=\"$green\">OK</td>"; 
					elsif ($sensor_health_color =~ m/red/i)	{ 
						$sensor_health_color="<td bgcolor=\"$red\">PROBLEM</td>";
						#MWPRESTON - added following 2 lines
						$emailMessage .= $_->name."\n"; 
					elsif ($sensor_health_color =~ m/yellow/i) { 
						$sensor_health_color="<td bgcolor=\"$yellow\">WARNING</td>"; 
						#MWPRESTON - added following 2 lines
						$emailMessage .= $_->name."\n"; 
					else { 
						$sensor_health_color="<td bgcolor=\"gray\">UNKNOWN</td>"; 
						#MWPRESTON - added following 2 lines
						$emailMessage .= $_->name."\n"; 
					$healthHardwareString .= "<tr><td>".$_->name."</td>".$sensor_health_color."</tr>\n";		
			if($hardwareStatusInfo->memoryStatusInfo) {
				$memInfo = $hardwareStatusInfo->memoryStatusInfo;
				foreach(@$memInfo) {
					$sensor_health = $_->status->key;
					if ($sensor_health =~ m/green/i) { 
						$sensor_health_color="<td bgcolor=\"$green\">OK</td>"; 
					elsif ($sensor_health_color =~ m/red/i) { 
						$sensor_health_color="<td bgcolor=\"$red\">PROBLEM</td>"; 
						#MWPRESTON - added following 2 lines
						$emailMessage .= $_->name."\n"; 
					elsif ($sensor_health_color =~ m/yellow/i) { 
						$sensor_health_color="<td bgcolor=\"$yellow\">WARNING</td>"; 
						#MWPRESTON - added following 2 lines
						$emailMessage .= $_->name."\n"; 
					else { 
						$sensor_health_color="<td bgcolor=\"gray\">UNKNOWN</td>"; 
					$healthHardwareString .= "<tr><td>".$_->name."</td>".$sensor_health_color."</tr>\n";
			if($hardwareStatusInfo->storageStatusInfo) {
				$storageInfo = $hardwareStatusInfo->storageStatusInfo;
				foreach(@$storageInfo) {
					$sensor_health = $_->status->key;
					if ($sensor_health =~ m/green/i) { 
						$sensor_health_color="<td bgcolor=\"$green\">OK</td>"; 
					elsif ($sensor_health_color =~ m/red/i) {
						$sensor_health_color="<td bgcolor=\"$red\">PROBLEM</td>";
						#MWPRESTON - added following 2 lines
						$emailMessage .= $_->name."\n";
					elsif ($sensor_health_color =~ m/yellow/i) { 
						$sensor_health_color="<td bgcolor=\"$yellow\">WARNING</td>";
						#MWPRESTON - added following 2 lines
						$emailMessage .= $_->name."\n";
					else { 
						$sensor_health_color="<td bgcolor=\"gray\">UNKNOWN</td>";
						#MWPRESTON - added following 2 lines
						$emailMessage .= $_->name."\n";
					$healthHardwareString .= "<tr><td>".$_->name."</td>".$sensor_health_color."</tr>\n";
		if($local_host->runtime->healthSystemRuntime->systemHealthInfo) {
			my $sensors = $local_host->runtime->healthSystemRuntime->systemHealthInfo->numericSensorInfo;
			$healthSoftwareString .= "<tr><th align=\"left\">".$host_name."</th></tr>\n";
			my $sensor_health_color = "";
			foreach(sort {$a->name cmp $b->name} @$sensors) {
				my $sensor_health = $_->healthState->key;
				if ($sensor_health =~ m/green/) { 
					$sensor_health_color="<td bgcolor=\"$green\">OK</td>"; 
				elsif ($sensor_health_color =~ m/red/) { 
					$sensor_health_color="<td bgcolor=\"$red\">PROBLEM</td>";
					#MWPRESTON - added following 2 lines
					$emailMessage .= $_->name."\n";
				elsif ($sensor_health_color =~ m/yellow/) { 
					$sensor_health_color="<td bgcolor=\"$yellow\">WARNING</td>"; 
					#MWPRESTON - added following 2 lines
					$emailMessage .= $_->name."\n";
				else { 
					$sensor_health_color="<td bgcolor=\"gray\">UNKNOWN</td>"; 
					#MWPRESTON - added following 2 lines
					$emailMessage .= $_->name."\n";
				my $reading;
				if(defined($_->rateUnits)) {
					$reading =  &restrict_num_decimal_digits(($_->currentReading * (10 ** $_->unitModifier)),3) . " " . $_->baseUnits . "/" . $_->rateUnits;
				} else {
					$reading =  &restrict_num_decimal_digits(($_->currentReading * (10 ** $_->unitModifier)),3) . " " . $_->baseUnits;
				$healthSoftwareString .= "<tr><td>".$_->name."</td><td>".$reading."</td>".$sensor_health_color."</tr>\n";

And finally you need to modify the emailReport function in order to check for HOSTISSUES (since we don’t want to email unless we have an issue) and also attach $emailMessage to the body of the message.  This function is near the top of the script starting at roughly line 325.  My changes are below….

sub emailReport {
	#MWPRESTON - modified if statement to include HOSTISSUES
	if($email eq "yes" and $HOSTISSUES ne "no") {
		my $smtp = Net::SMTP->new($EMAIL_HOST ,Hello => $EMAIL_DOMAIN,Timeout => 30,);

		unless($smtp) {
	                die "Error: Unable to setup connection with email server: \"" . $EMAIL_HOST . "\"!\n";

		open(DATA, $report) || die("Could not open the file");
		my @report = <DATA>;

		my @EMAIL_RECIPIENTS = $smtp->recipient(@EMAIL_TO,{SkipBad => 1});
		my $boundary = 'frontier';
		$smtp->datasend('From: '.$EMAIL_FROM."\n");
		$smtp->datasend('To: '.@EMAIL_TO."\n");
		$smtp->datasend('Subject: VMware vSphere Health Check Report Completed - '.giveMeDate('MDYHMS'). " (" . $system_name . ")\n");
		$smtp->datasend("MIME-Version: 1.0\n");
		$smtp->datasend("Content-type: multipart/mixed;\n\tboundary=\"$boundary\"\n");
		$smtp->datasend("Content-type: text/plain\n");
		$smtp->datasend("Content-Disposition: quoted-printable\n");
		#MWPRESTON - added following 2 lines.
		$smtp->datasend("\nProblems have been detected with the following\n");
		$smtp->datasend("\nReport $report is attached!\n");
		$smtp->datasend("Content-Type: application/text; name=\"$report\"\n");
		$smtp->datasend("Content-Disposition: attachment; filename=\"$report\"\n");

If you would like the complete modified script you can get it here – honestly, I would recommend going and downloading Willams version first  and then make the required changes to it.  Since his script seems to be actively updated and worked on, whereas mine here will probably remain pretty static.  Again, I haven’t thoroughly tested this and it’s a use at your own risk type thing – and I’ve only applied these changes in version 5.02 of Williams script.

Again, any comments, concerns, questions, or if you just need help setting up the script or cron job let me know in the comments box below.

Creating Roles and adding Active Directory Users & Groups to the role through PowerCLI

As of late I’ve been working on a PowerCLI script to assist me with the deployment of around 50 or so ESXi hosts.  For the most part the PowerCLI cmdlets have been pretty self explanatory and pretty simple to use.  Really I just need to setup some DNS, Hostnames, Networks, Datastores, NTP settings, and permissions…you know, everything you would normally do when setting up a host.  The problem I’ve ran into is when I hit that permissions section.

Essentially what I wanted to do was create a new role containing only those permissions that I wanted then simply assign an active directory group to that role.  So the role part, not too bad, pretty simple really…I used the following…

New-VIRole -Name "Elevated VM User" -Privilege (Get-VIPrivilege -ID "VirtualMachine.Interact")
Get-VIRole "Elevated VM User" | Set-VIRole -AddPrivilege (Get-VIPrivilege -ID "ScheduledTask.Run")
Get-VIRole "Elevated VM User" | Set-VIRole -AddPrivilege (Get-VIPrivilege -ID "ScheduledTask.Create")
Get-VIRole "Elevated VM User" | Set-VIRole -AddPrivilege (Get-VIPrivilege -Name "Power")
Get-VIRole "Elevated VM User" | Set-VIRole -AddPrivilege (Get-VIPrivilege -Name "Maintenance")

Basically this just creates my new role (Elevated VM User) and adds a handful of permissions to it.  In order to find out the name for all the priveleges I simply just ran Get-VIPrivelege -Role ‘Admin’.

So now that I had my role created I just need to add an AD Account to that role.  A lot of the documentation led me to the New-VIPermission cmdlet – sounds easy, but running the following New-VIPermission -Role ‘Elevated VM User’ -Principal ‘DOMAIN\Username’ produced nothing but the following errors for me….

New-VIPermission : 9/12/2012 2:47:40 PM New-VIPermission Could not find VIAccount with name ‘Domain\Username’.
At line:1 char:17
+ New-VIPermission <<<< -Role ‘Admin’ -Principal ‘Domain\Username’
+ CategoryInfo : ObjectNotFound: (Domain\Username:String) [New-VIPermission], VimException
+ FullyQualifiedErrorId : Core_ObnSelector_SelectObjectByNameCore_ObjectNotFound,VMware.VimAutomation.ViCore.Cmdlets.Commands.PermissionManagement.NewVIPermission

New-VIPermission : 9/12/2012 2:47:40 PM New-VIPermission Value cannot be found for the mandatory parameter Entity
At line:1 char:17
+ New-VIPermission <<<< -Role ‘Admin’ -Principal ‘Domain\Username’
+ CategoryInfo : NotSpecified: (:) [New-VIPermission], VimException
+ FullyQualifiedErrorId : Core_BaseCmdlet_UnknownError,VMware.VimAutomation.ViCore.Cmdlets.Commands.PermissionManagement.NewVIPermission

Now it seams as if me passing the string of ‘Domain\Username’ is not satisfying whatever object it is that -Principal is looking for.  So after a many googling and twittering the following code is how I ended up assigning Domain\Username to the role I created.

$groupName = "Domain\Username"
$currhost = get-vmhost HOSTNAME | % {Get-View $_.Id}
$authmgr = Get-View AuthorizationManager-ha-authmgr
$perm = New-Object VMware.VIM.Permission
$perm.Principal = $groupName
$perm.group = $true
$perm.propagate = $true
$perm.RoleId = ($authmgr.RoleList | where {$_.Name -eq "Elevated VM User"}).RoleId

And this worked perfect!  As I’ve always said I’m sure there are a million ways to do something and if you think your way is more efficient or better than mine, please share!  This is my first crack at it so all suggestions and comments are always welcome 🙂

PXE booting a scripted ESXi 5.x Install

As of late I’ve been working on an automated deployment solution for ESXi.  Seeing as I will be deploying 50 + identical installations in the near future one thing I don’t want to have to do is follow documentation and screenshots telling me where and when to click.  So, enter the scripted install…and hey, why not throw PXE into the mix as well.

The official VMware documentation does have section dealing with PXE booting a scripted install, but it seems to assume that the VMware installation is going to be the only application included in your PXE boot and TFTP directories.  Now I don’t know about you, but the VMware installation is not the only thing that I want to boot from my PXE server.  Most people will have a menu built with various installation and utilities….so this is a few things I’ve learned over the past week that are not in the documentation…

So first off is the actual menu item in the the pxelinux config file.  Below is the actual KERNEL and APPEND items from the VMware documentation (which won’t work unless you have extracted everything into the root of your TFTP server.  As you can see there is no path preceding mboot, thus assuming it’s in the root.

LABEL install KERNEL mboot.c32 
APPEND -c location of boot.cfg MENU LABEL ESXi-5.0.0-XXXXXX-full ^Installer

Since we are placing our install files inside a utils folder, we need to change this to read as follows.

LABEL ESXi 5 Installation
 MENU LABEL ^11.VMware ESXi 5 Installation
 KERNEL utils/ESXi5/mboot.c32
 APPEND -c utils/ESXi5/boot.cfg ks=http://myKSWebserver/ks/ks.cfg

Also you can see that I have added the ks=location of ks.cfg command to boot.cfg as well.  This is what allows us to do the scripted installation.  I’m not going to delve into all of the ks options that are out there, if you want to you can go to the documentation and check it out for yourself.  The point of this post is just to get you up and running with a scripted pxe install when not residing in the root directory.

We are almost done now, just one more step.  The problem of pointing to the install outside of the root directory is solved, however the actual boot.cfg (located in the root of the install files) also assumes it is in the root directory, so we now have to go and tweak that a bit.  Again, the boot.cfg in its unmodified state is below…

 title=Loading ESXi installer
 modules=/b.b00 --- /useropts.gz --- /k.b00 --- /a.b00 --- /ata-pata.v00 --- /ata-pata.v01 --- /ata-pata.v02 --- /ata-pata.v03 --- /ata-pata.v04 --- /ata-pata.v05 --- /ata-pata.v06 --- /ata-pata.v07 --- /block-cc.v00 --- /ehci-ehc.v00 --- /s.v00 --- /weaselin.i00 --- /ima-qla4.v00 --- /ipmi-ipm.v00 --- /ipmi-ipm.v01 --- /ipmi-ipm.v02 --- /misc-cni.v00 --- /misc-dri.v00 --- /net-be2n.v00 --- /net-bnx2.v00 --- /net-bnx2.v01 --- /net-cnic.v00 --- /net-e100.v00 --- /net-e100.v01 --- /net-enic.v00 --- /net-forc.v00 --- /net-igb.v00 --- /net-ixgb.v00 --- /net-nx-n.v00 --- /net-r816.v00 --- /net-r816.v01 --- /net-s2io.v00 --- /net-sky2.v00 --- /net-tg3.v00 --- /ohci-usb.v00 --- /sata-ahc.v00 --- /sata-ata.v00 --- /sata-sat.v00 --- /sata-sat.v01 --- /sata-sat.v02 --- /sata-sat.v03 --- /scsi-aac.v00 --- /scsi-adp.v00 --- /scsi-aic.v00 --- /scsi-bnx.v00 --- /scsi-fni.v00 --- /scsi-hps.v00 --- /scsi-ips.v00 --- /scsi-lpf.v00 --- /scsi-meg.v00 --- /scsi-meg.v01 --- /scsi-meg.v02 --- /scsi-mpt.v00 --- /scsi-mpt.v01 --- /scsi-mpt.v02 --- /scsi-qla.v00 --- /scsi-qla.v01 --- /scsi-rst.v00 --- /uhci-usb.v00 --- /tools.t00 --- /imgdb.tgz --- /imgpayld.tgz

So there are a few minor tweaks we need to do with this.  First off, we need to add the prefix label and define where our installation files are actually sitting.  Secondly, we need to remove every single slash that is preceding every module.  This will allow our installation to load and boot as directed from a subfolder within the root directory of our TFTP server.  The modified boot.cfg is shown below…

 title=Loading ESXi installer
 modules=b.b00 --- useropts.gz --- k.b00 --- a.b00 --- ata-pata.v00 --- ata-pata.v01 --- ata-pata.v02 --- ata-pata.v03 --- ata-pata.v04 --- ata-pata.v05 --- ata-pata.v06 --- ata-pata.v07 --- block-cc.v00 --- ehci-ehc.v00 --- s.v00 --- weaselin.i00 --- ima-qla4.v00 --- ipmi-ipm.v00 --- ipmi-ipm.v01 --- ipmi-ipm.v02 --- misc-cni.v00 --- misc-dri.v00 --- net-be2n.v00 --- net-bnx2.v00 --- net-bnx2.v01 --- net-cnic.v00 --- net-e100.v00 --- net-e100.v01 --- net-enic.v00 --- net-forc.v00 --- net-igb.v00 --- net-ixgb.v00 --- net-nx-n.v00 --- net-r816.v00 --- net-r816.v01 --- net-s2io.v00 --- net-sky2.v00 --- net-tg3.v00 --- ohci-usb.v00 --- sata-ahc.v00 --- sata-ata.v00 --- sata-sat.v00 --- sata-sat.v01 --- sata-sat.v02 --- sata-sat.v03 --- scsi-aac.v00 --- scsi-adp.v00 --- scsi-aic.v00 --- scsi-bnx.v00 --- scsi-fni.v00 --- scsi-hps.v00 --- scsi-ips.v00 --- scsi-lpf.v00 --- scsi-meg.v00 --- scsi-meg.v01 --- scsi-meg.v02 --- scsi-mpt.v00 --- scsi-mpt.v01 --- scsi-mpt.v02 --- scsi-qla.v00 --- scsi-qla.v01 --- scsi-rst.v00 --- uhci-usb.v00 --- tools.t00 --- lsiprovi.v00 --- imgdb.tgz --- imgpayld.tgz

So there you go, as long as your pxe environment is set up properly you should be able to just boot into the pxe menu, select your ESXi installation label and install ESXi as normal or if you have added the directive for a kickstart installation the scripted install will begin.  If you have any thoughts, questions, comments, concerns or even suggestions on how to do this a better way please let me know in the comments below 😉

Practise makes perfect! More PowerCLI APC Powerchute Network Shutdown Goodness (Now with Power On!)

Picture this, a quaint little city in Eastern Ontario. It’s been at least a couple of months since we have really had any rain. My tomato plants are dying, my lawn is completely brown, but the datacentre, it’s chugging along nicely…Then it hits us, the first rainfall (of any real value) of the summer. Finally maybe things will start to grow again. Chatter around the office turns to the weather yet again, people are smiling, happy, can’t wait to get home see if maybe there was a chance that their grass might have turned a slight shade of green…and then, nothing but darkness and the faint sound of our neighbouring businesses generator kicking on…!

Are you kidding me? It rains once this summer and it knocks the power out? Wow! No big deal though right? We have our Powerchute Network Shutdown all configured to peel through and shut-down all of our physical hardware, and a while back I wrote this nifty little script to shutdown the virtual infrastructure, no problem!

And thus the title for this blog post – Practise makes Perfect! Turns out that I could have been a little more efficient in my script. Initially I was looping through hosts one by one shutting off VMs, then waiting, then checking, then going through for another pass, and then moving on to the next host… Well, with 10 minutes left on battery I don’t have the time to complete this process on our 8 production hosts, let alone an environment of any type of size. Needless to say the VMs and hosts that didn’t get the time of the day with the script fell hard. Thankfully, there was no corruption or issues resulting from it, but still, I don’t like things not going as planned . When I’m standing in the datacenter I like to see all of the blinky lights shut off before those air conditioners stop pumping out coolness.. So out of my failures I give you the new and improved shutdown script (now with Power ON). You can head straight here to pull down both the power off and power on scripts, but they are explained in a bit more detail below as there is a little bit of configuration to get the power on functionality to work….  As well a big thanks goes out to a fellow Toronto VMUGer (VMUGite ??) Eric Wright (blog / twitter ) and his ‘The Art of Shutdown‘ post a couple of months back.  Eric wrote a great script to shut things down and even has a great little ‘test’ mode in his version.  I got a few lines of code and ideas from Eric’s script so go check it out as well

Power Off the VMs

So first off is the shutdown script – you can get the complete version here.

Just a few notes regarding my updates to the script.

if ($keyword -ne $mysecret) 
    Write-Host "You haven't passed the proper detonation sequence...ABORTING THE SCRIPT" -ForegroundColor red

So first off you see here that if you call the script without a parameter or with a parameter that doesn’t match our secret keyword inside the script the whole thing aborts.  This is to stop us from simply double clicking this or accidenttally unleashing a disaster amongst ourselves.

Get-VM -Location $cluster | where-object {$_.PowerState -eq "PoweredOn" } | Select Name | Export-CSV c:\Scripts\Shutdown\PowereredOnVMGuests.csv

Another important addition to the script.  Basically this dumps a list of the powered on VMs to a csv file.  This will be used in the partner power on script once power has been restored.

The rest of the script is pretty straight forward.  Basically sets DRS to manual mode, disables HA, gracefully shuts down the VMs (any without VMware Tools installed get a hard shutdown), waits a couple of minutes and does another pass, then procedes to shutdown the hosts.  Again, the complete poweroff script can be downloaded here.

Power On the VMs

So all is good, VMs got powered off, but what happens when power comes back up.  Even if you do get your hosts powered back on they aren’t going to turn on your VMs automatically.  Now I know there is a start and stop with host setting but when you are in a DRS cluster this doesn’t do you much good as it is a host setting and VMs move between hosts.  Thus the power on script!  So this script will sit on the same box as your shutdown script.  It needs to be outside of your virtual infrastructure.  My vCenter Server is a physical box so I have placed both the shutdown and the power on scripts there.  Also, you will need to have this script be triggered on start-up for this machine.  This is simply do to the fact that I don’t want to be running any scripts in the middle of the night if power is restored…I’d rather walk in to a fully functional datacenter the next morning 🙂

The full script can be downloaded here but below are a few explanations of what happens….

if (Test-Path $filename)

The complete script is wrapped in this if statement.  Meaning if the power off dump file doesn’t exist then the script will not execute.

    while ((Get-Service vpxd).Status -ne "Running")
        Write-Host "." -nonewline
        Sleep 2
        Write-Host "." -nonewline
        Sleep 2
        Write-Host "." -nonewline
        Sleep 2 

This repeating loop basically waits for the vCenter Service to be started before the script can continue.  Since this script should be executed during startup we may have to spend some time here waiting for vCenter to get all its ducks in a row 🙂

    foreach ($iVM in $myImportantVMs)
        Write-Host "Powering on $iVM ..." -nonewline
        Start-VM $iVM
        Sleep 5
        Write-Host "DONE" -Foregroundcolor Green   

Before reading the dump file and getting into the thick of everything we have the option to power on some of the more important VMs first.  So if you have any mail servers, dhcp servers, dns servers, it would be a good idea to put them into $myImportantVMs to have them started first. Once these VMs are on a similar type function will power on the rest of the VMs in the csv file, one by one, every 5 seconds.  You can set the sleep command to whatever you want but 5 seems good for me.

    rename-item "$fileName" "$nameOnly-$DateStamp$extOnly"

This is nearing the end of the script and basically appends the date to the csv file.  This prevents the script from running on next startup – unless you have another power outage of course.

So there you have it – a complete power off script and a complete power on script.  I know I’ve rambled a little bit in the post but I’m just in that sort of mood.  If you need a hand setting things up or have any questions, concerns, improvements, criticism don’t hesitate to leave them in the comments.  I’ll get back to you….  And yes, my tomato plants are all dead!

Dell Storage missing from VMware Health Status

We have probably all been there at one point while messing around with ESXi and certain types of hardware.  You get your ESXi host installed and configured and it seems to work fine until you look at the Health Status section of the host configuration and notice that you can see mostly all of your hardware, except for the peice that is most likely to fail, your storage.  Now normally, well, with HP anyway, you can install the provided offline packages in order to get the hardware to appear, however in my experiences with Dell after installing their Server Administrator software and their Open Manage bundles the storage was still no where to be found.

Well, here's the deal.  The adapter that I was specifically trying to report on was a PERC H700.  Now most of the Dell PERC controllers are based off of a MegaRAID chipset (answer is always a Google search away) and you can confirm this by running 'esxcfg-scsidevs -a' to show the following output about your hba's.

So, off to LSI's site I headed to have a look for a VIB that might give me the information I need to see about my storage.  I ended up pulling down a vib designed for the MegaSAS 9260.  There are many ways to install a vib into ESXi but since I already had established an SSH session on my host I decided to use esxcli.  After transferring the vib file to your host you can install it via the CLI by running the following command.  I thought I was home free here…

esxcli software vib install -v /tmp/name_of_vib

Now this returned the following error…

'Could not find a trusted signer.' – a quick look at the help for for vib install I seen that there was a –no-sig-check option that could be attached to the command.  Honestly, I'm not sure what the implications of running with 'no sig check' are, but I forged ahead anyways.  If i ever get some time I'll try and do a little reading and update this post with some valid information around skipping the signature check.  Either way, running the command again with the –no-sig-check (below) will get the job done for you…

esxcli software vib install -v /tmp/name_of_vib –no-sig-check

So, a quick reboot after the fact and voila!  Disk information.  

So, there you have it, finally I can see my storage 🙂  Once again, comments, concerns, questions are always welcome in the boxes below….

Getting rid of that pesky shell warning in ESXi 5

We all know that we can enable SSH and the ESXi shell from within the vSphere client or through the DCUI.  This is a great feature that lets us get into the ESXi command space and run things like esxtop, esxcli commands, etc…  Problem being, that once these shells are enabled we get that pesky shell/SSH warning displayed in our vSphere client, as well, that all too familiar yellow triangle gets labeled on our host.  Now, I don't like seeing any warnings on my hosts, especially those dealing with something as minor as SSH.  Good thing is, there is a very easy way to remove or suppress these warnings.

First off, the advanced configuration setting to do this is located in the software section under 'Advanced Settings'->UserVars->UserVars.SuppressShellWarning'.  By default this setting will be set to 0, meaning display the warning.  To hide it, simply set this option to 1.

There you go!  Easy enough… if you only have one host!  But what if you had multiple clusters full of multiple hosts…. well, that's where PowerCLI comes into play.  First off, connect to your vCenter server using the Connect-VIServer servername CMDLET.  Once connected, the following command will loop though a given cluster and modify the setting on every host…

foreach ($esxhost in get-VMHost -Location CLUSTERNAME ) { $esxhost | Set-VMHostAdvancedConfiguration -Name UserVars.SuppressShellWarning -Value 1 }

And there you go!  A happy, non warning triangle life for you!