Tag Archives: ESXi

Creating Roles and adding Active Directory Users & Groups to the role through PowerCLI

As of late I’ve been working on a PowerCLI script to assist me with the deployment of around 50 or so ESXi hosts.  For the most part the PowerCLI cmdlets have been pretty self explanatory and pretty simple to use.  Really I just need to setup some DNS, Hostnames, Networks, Datastores, NTP settings, and permissions…you know, everything you would normally do when setting up a host.  The problem I’ve ran into is when I hit that permissions section.

Essentially what I wanted to do was create a new role containing only those permissions that I wanted then simply assign an active directory group to that role.  So the role part, not too bad, pretty simple really…I used the following…

New-VIRole -Name "Elevated VM User" -Privilege (Get-VIPrivilege -ID "VirtualMachine.Interact")
Get-VIRole "Elevated VM User" | Set-VIRole -AddPrivilege (Get-VIPrivilege -ID "ScheduledTask.Run")
Get-VIRole "Elevated VM User" | Set-VIRole -AddPrivilege (Get-VIPrivilege -ID "ScheduledTask.Create")
Get-VIRole "Elevated VM User" | Set-VIRole -AddPrivilege (Get-VIPrivilege -Name "Power")
Get-VIRole "Elevated VM User" | Set-VIRole -AddPrivilege (Get-VIPrivilege -Name "Maintenance")

Basically this just creates my new role (Elevated VM User) and adds a handful of permissions to it.  In order to find out the name for all the priveleges I simply just ran Get-VIPrivelege -Role ‘Admin’.

So now that I had my role created I just need to add an AD Account to that role.  A lot of the documentation led me to the New-VIPermission cmdlet – sounds easy, but running the following New-VIPermission -Role ‘Elevated VM User’ -Principal ‘DOMAIN\Username’ produced nothing but the following errors for me….

New-VIPermission : 9/12/2012 2:47:40 PM New-VIPermission Could not find VIAccount with name ‘Domain\Username’.
At line:1 char:17
+ New-VIPermission <<<< -Role ‘Admin’ -Principal ‘Domain\Username’
+ CategoryInfo : ObjectNotFound: (Domain\Username:String) [New-VIPermission], VimException
+ FullyQualifiedErrorId : Core_ObnSelector_SelectObjectByNameCore_ObjectNotFound,VMware.VimAutomation.ViCore.Cmdlets.Commands.PermissionManagement.NewVIPermission

New-VIPermission : 9/12/2012 2:47:40 PM New-VIPermission Value cannot be found for the mandatory parameter Entity
At line:1 char:17
+ New-VIPermission <<<< -Role ‘Admin’ -Principal ‘Domain\Username’
+ CategoryInfo : NotSpecified: (:) [New-VIPermission], VimException
+ FullyQualifiedErrorId : Core_BaseCmdlet_UnknownError,VMware.VimAutomation.ViCore.Cmdlets.Commands.PermissionManagement.NewVIPermission

Now it seams as if me passing the string of ‘Domain\Username’ is not satisfying whatever object it is that -Principal is looking for.  So after a many googling and twittering the following code is how I ended up assigning Domain\Username to the role I created.

$groupName = "Domain\Username"
$currhost = get-vmhost HOSTNAME | % {Get-View $_.Id}
$authmgr = Get-View AuthorizationManager-ha-authmgr
$perm = New-Object VMware.VIM.Permission
$perm.Principal = $groupName
$perm.group = $true
$perm.propagate = $true
$perm.RoleId = ($authmgr.RoleList | where {$_.Name -eq "Elevated VM User"}).RoleId

And this worked perfect!  As I’ve always said I’m sure there are a million ways to do something and if you think your way is more efficient or better than mine, please share!  This is my first crack at it so all suggestions and comments are always welcome 🙂

PXE booting a scripted ESXi 5.x Install

As of late I’ve been working on an automated deployment solution for ESXi.  Seeing as I will be deploying 50 + identical installations in the near future one thing I don’t want to have to do is follow documentation and screenshots telling me where and when to click.  So, enter the scripted install…and hey, why not throw PXE into the mix as well.

The official VMware documentation does have section dealing with PXE booting a scripted install, but it seems to assume that the VMware installation is going to be the only application included in your PXE boot and TFTP directories.  Now I don’t know about you, but the VMware installation is not the only thing that I want to boot from my PXE server.  Most people will have a menu built with various installation and utilities….so this is a few things I’ve learned over the past week that are not in the documentation…

So first off is the actual menu item in the the pxelinux config file.  Below is the actual KERNEL and APPEND items from the VMware documentation (which won’t work unless you have extracted everything into the root of your TFTP server.  As you can see there is no path preceding mboot, thus assuming it’s in the root.

LABEL install KERNEL mboot.c32 
APPEND -c location of boot.cfg MENU LABEL ESXi-5.0.0-XXXXXX-full ^Installer

Since we are placing our install files inside a utils folder, we need to change this to read as follows.

LABEL ESXi 5 Installation
 MENU LABEL ^11.VMware ESXi 5 Installation
 KERNEL utils/ESXi5/mboot.c32
 APPEND -c utils/ESXi5/boot.cfg ks=http://myKSWebserver/ks/ks.cfg

Also you can see that I have added the ks=location of ks.cfg command to boot.cfg as well.  This is what allows us to do the scripted installation.  I’m not going to delve into all of the ks options that are out there, if you want to you can go to the documentation and check it out for yourself.  The point of this post is just to get you up and running with a scripted pxe install when not residing in the root directory.

We are almost done now, just one more step.  The problem of pointing to the install outside of the root directory is solved, however the actual boot.cfg (located in the root of the install files) also assumes it is in the root directory, so we now have to go and tweak that a bit.  Again, the boot.cfg in its unmodified state is below…

 title=Loading ESXi installer
 modules=/b.b00 --- /useropts.gz --- /k.b00 --- /a.b00 --- /ata-pata.v00 --- /ata-pata.v01 --- /ata-pata.v02 --- /ata-pata.v03 --- /ata-pata.v04 --- /ata-pata.v05 --- /ata-pata.v06 --- /ata-pata.v07 --- /block-cc.v00 --- /ehci-ehc.v00 --- /s.v00 --- /weaselin.i00 --- /ima-qla4.v00 --- /ipmi-ipm.v00 --- /ipmi-ipm.v01 --- /ipmi-ipm.v02 --- /misc-cni.v00 --- /misc-dri.v00 --- /net-be2n.v00 --- /net-bnx2.v00 --- /net-bnx2.v01 --- /net-cnic.v00 --- /net-e100.v00 --- /net-e100.v01 --- /net-enic.v00 --- /net-forc.v00 --- /net-igb.v00 --- /net-ixgb.v00 --- /net-nx-n.v00 --- /net-r816.v00 --- /net-r816.v01 --- /net-s2io.v00 --- /net-sky2.v00 --- /net-tg3.v00 --- /ohci-usb.v00 --- /sata-ahc.v00 --- /sata-ata.v00 --- /sata-sat.v00 --- /sata-sat.v01 --- /sata-sat.v02 --- /sata-sat.v03 --- /scsi-aac.v00 --- /scsi-adp.v00 --- /scsi-aic.v00 --- /scsi-bnx.v00 --- /scsi-fni.v00 --- /scsi-hps.v00 --- /scsi-ips.v00 --- /scsi-lpf.v00 --- /scsi-meg.v00 --- /scsi-meg.v01 --- /scsi-meg.v02 --- /scsi-mpt.v00 --- /scsi-mpt.v01 --- /scsi-mpt.v02 --- /scsi-qla.v00 --- /scsi-qla.v01 --- /scsi-rst.v00 --- /uhci-usb.v00 --- /tools.t00 --- /imgdb.tgz --- /imgpayld.tgz

So there are a few minor tweaks we need to do with this.  First off, we need to add the prefix label and define where our installation files are actually sitting.  Secondly, we need to remove every single slash that is preceding every module.  This will allow our installation to load and boot as directed from a subfolder within the root directory of our TFTP server.  The modified boot.cfg is shown below…

 title=Loading ESXi installer
 modules=b.b00 --- useropts.gz --- k.b00 --- a.b00 --- ata-pata.v00 --- ata-pata.v01 --- ata-pata.v02 --- ata-pata.v03 --- ata-pata.v04 --- ata-pata.v05 --- ata-pata.v06 --- ata-pata.v07 --- block-cc.v00 --- ehci-ehc.v00 --- s.v00 --- weaselin.i00 --- ima-qla4.v00 --- ipmi-ipm.v00 --- ipmi-ipm.v01 --- ipmi-ipm.v02 --- misc-cni.v00 --- misc-dri.v00 --- net-be2n.v00 --- net-bnx2.v00 --- net-bnx2.v01 --- net-cnic.v00 --- net-e100.v00 --- net-e100.v01 --- net-enic.v00 --- net-forc.v00 --- net-igb.v00 --- net-ixgb.v00 --- net-nx-n.v00 --- net-r816.v00 --- net-r816.v01 --- net-s2io.v00 --- net-sky2.v00 --- net-tg3.v00 --- ohci-usb.v00 --- sata-ahc.v00 --- sata-ata.v00 --- sata-sat.v00 --- sata-sat.v01 --- sata-sat.v02 --- sata-sat.v03 --- scsi-aac.v00 --- scsi-adp.v00 --- scsi-aic.v00 --- scsi-bnx.v00 --- scsi-fni.v00 --- scsi-hps.v00 --- scsi-ips.v00 --- scsi-lpf.v00 --- scsi-meg.v00 --- scsi-meg.v01 --- scsi-meg.v02 --- scsi-mpt.v00 --- scsi-mpt.v01 --- scsi-mpt.v02 --- scsi-qla.v00 --- scsi-qla.v01 --- scsi-rst.v00 --- uhci-usb.v00 --- tools.t00 --- lsiprovi.v00 --- imgdb.tgz --- imgpayld.tgz

So there you go, as long as your pxe environment is set up properly you should be able to just boot into the pxe menu, select your ESXi installation label and install ESXi as normal or if you have added the directive for a kickstart installation the scripted install will begin.  If you have any thoughts, questions, comments, concerns or even suggestions on how to do this a better way please let me know in the comments below 😉

Practise makes perfect! More PowerCLI APC Powerchute Network Shutdown Goodness (Now with Power On!)

Picture this, a quaint little city in Eastern Ontario. It’s been at least a couple of months since we have really had any rain. My tomato plants are dying, my lawn is completely brown, but the datacentre, it’s chugging along nicely…Then it hits us, the first rainfall (of any real value) of the summer. Finally maybe things will start to grow again. Chatter around the office turns to the weather yet again, people are smiling, happy, can’t wait to get home see if maybe there was a chance that their grass might have turned a slight shade of green…and then, nothing but darkness and the faint sound of our neighbouring businesses generator kicking on…!

Are you kidding me? It rains once this summer and it knocks the power out? Wow! No big deal though right? We have our Powerchute Network Shutdown all configured to peel through and shut-down all of our physical hardware, and a while back I wrote this nifty little script to shutdown the virtual infrastructure, no problem!

And thus the title for this blog post – Practise makes Perfect! Turns out that I could have been a little more efficient in my script. Initially I was looping through hosts one by one shutting off VMs, then waiting, then checking, then going through for another pass, and then moving on to the next host… Well, with 10 minutes left on battery I don’t have the time to complete this process on our 8 production hosts, let alone an environment of any type of size. Needless to say the VMs and hosts that didn’t get the time of the day with the script fell hard. Thankfully, there was no corruption or issues resulting from it, but still, I don’t like things not going as planned . When I’m standing in the datacenter I like to see all of the blinky lights shut off before those air conditioners stop pumping out coolness.. So out of my failures I give you the new and improved shutdown script (now with Power ON). You can head straight here to pull down both the power off and power on scripts, but they are explained in a bit more detail below as there is a little bit of configuration to get the power on functionality to work….  As well a big thanks goes out to a fellow Toronto VMUGer (VMUGite ??) Eric Wright (blog / twitter ) and his ‘The Art of Shutdown‘ post a couple of months back.  Eric wrote a great script to shut things down and even has a great little ‘test’ mode in his version.  I got a few lines of code and ideas from Eric’s script so go check it out as well

Power Off the VMs

So first off is the shutdown script – you can get the complete version here.

Just a few notes regarding my updates to the script.

if ($keyword -ne $mysecret) 
    Write-Host "You haven't passed the proper detonation sequence...ABORTING THE SCRIPT" -ForegroundColor red

So first off you see here that if you call the script without a parameter or with a parameter that doesn’t match our secret keyword inside the script the whole thing aborts.  This is to stop us from simply double clicking this or accidenttally unleashing a disaster amongst ourselves.

Get-VM -Location $cluster | where-object {$_.PowerState -eq "PoweredOn" } | Select Name | Export-CSV c:\Scripts\Shutdown\PowereredOnVMGuests.csv

Another important addition to the script.  Basically this dumps a list of the powered on VMs to a csv file.  This will be used in the partner power on script once power has been restored.

The rest of the script is pretty straight forward.  Basically sets DRS to manual mode, disables HA, gracefully shuts down the VMs (any without VMware Tools installed get a hard shutdown), waits a couple of minutes and does another pass, then procedes to shutdown the hosts.  Again, the complete poweroff script can be downloaded here.

Power On the VMs

So all is good, VMs got powered off, but what happens when power comes back up.  Even if you do get your hosts powered back on they aren’t going to turn on your VMs automatically.  Now I know there is a start and stop with host setting but when you are in a DRS cluster this doesn’t do you much good as it is a host setting and VMs move between hosts.  Thus the power on script!  So this script will sit on the same box as your shutdown script.  It needs to be outside of your virtual infrastructure.  My vCenter Server is a physical box so I have placed both the shutdown and the power on scripts there.  Also, you will need to have this script be triggered on start-up for this machine.  This is simply do to the fact that I don’t want to be running any scripts in the middle of the night if power is restored…I’d rather walk in to a fully functional datacenter the next morning 🙂

The full script can be downloaded here but below are a few explanations of what happens….

if (Test-Path $filename)

The complete script is wrapped in this if statement.  Meaning if the power off dump file doesn’t exist then the script will not execute.

    while ((Get-Service vpxd).Status -ne "Running")
        Write-Host "." -nonewline
        Sleep 2
        Write-Host "." -nonewline
        Sleep 2
        Write-Host "." -nonewline
        Sleep 2 

This repeating loop basically waits for the vCenter Service to be started before the script can continue.  Since this script should be executed during startup we may have to spend some time here waiting for vCenter to get all its ducks in a row 🙂

    foreach ($iVM in $myImportantVMs)
        Write-Host "Powering on $iVM ..." -nonewline
        Start-VM $iVM
        Sleep 5
        Write-Host "DONE" -Foregroundcolor Green   

Before reading the dump file and getting into the thick of everything we have the option to power on some of the more important VMs first.  So if you have any mail servers, dhcp servers, dns servers, it would be a good idea to put them into $myImportantVMs to have them started first. Once these VMs are on a similar type function will power on the rest of the VMs in the csv file, one by one, every 5 seconds.  You can set the sleep command to whatever you want but 5 seems good for me.

    rename-item "$fileName" "$nameOnly-$DateStamp$extOnly"

This is nearing the end of the script and basically appends the date to the csv file.  This prevents the script from running on next startup – unless you have another power outage of course.

So there you have it – a complete power off script and a complete power on script.  I know I’ve rambled a little bit in the post but I’m just in that sort of mood.  If you need a hand setting things up or have any questions, concerns, improvements, criticism don’t hesitate to leave them in the comments.  I’ll get back to you….  And yes, my tomato plants are all dead!

Dell Storage missing from VMware Health Status

We have probably all been there at one point while messing around with ESXi and certain types of hardware.  You get your ESXi host installed and configured and it seems to work fine until you look at the Health Status section of the host configuration and notice that you can see mostly all of your hardware, except for the peice that is most likely to fail, your storage.  Now normally, well, with HP anyway, you can install the provided offline packages in order to get the hardware to appear, however in my experiences with Dell after installing their Server Administrator software and their Open Manage bundles the storage was still no where to be found.

Well, here's the deal.  The adapter that I was specifically trying to report on was a PERC H700.  Now most of the Dell PERC controllers are based off of a MegaRAID chipset (answer is always a Google search away) and you can confirm this by running 'esxcfg-scsidevs -a' to show the following output about your hba's.

So, off to LSI's site I headed to have a look for a VIB that might give me the information I need to see about my storage.  I ended up pulling down a vib designed for the MegaSAS 9260.  There are many ways to install a vib into ESXi but since I already had established an SSH session on my host I decided to use esxcli.  After transferring the vib file to your host you can install it via the CLI by running the following command.  I thought I was home free here…

esxcli software vib install -v /tmp/name_of_vib

Now this returned the following error…

'Could not find a trusted signer.' – a quick look at the help for for vib install I seen that there was a –no-sig-check option that could be attached to the command.  Honestly, I'm not sure what the implications of running with 'no sig check' are, but I forged ahead anyways.  If i ever get some time I'll try and do a little reading and update this post with some valid information around skipping the signature check.  Either way, running the command again with the –no-sig-check (below) will get the job done for you…

esxcli software vib install -v /tmp/name_of_vib –no-sig-check

So, a quick reboot after the fact and voila!  Disk information.  

So, there you have it, finally I can see my storage 🙂  Once again, comments, concerns, questions are always welcome in the boxes below….

Getting rid of that pesky shell warning in ESXi 5

We all know that we can enable SSH and the ESXi shell from within the vSphere client or through the DCUI.  This is a great feature that lets us get into the ESXi command space and run things like esxtop, esxcli commands, etc…  Problem being, that once these shells are enabled we get that pesky shell/SSH warning displayed in our vSphere client, as well, that all too familiar yellow triangle gets labeled on our host.  Now, I don't like seeing any warnings on my hosts, especially those dealing with something as minor as SSH.  Good thing is, there is a very easy way to remove or suppress these warnings.

First off, the advanced configuration setting to do this is located in the software section under 'Advanced Settings'->UserVars->UserVars.SuppressShellWarning'.  By default this setting will be set to 0, meaning display the warning.  To hide it, simply set this option to 1.

There you go!  Easy enough… if you only have one host!  But what if you had multiple clusters full of multiple hosts…. well, that's where PowerCLI comes into play.  First off, connect to your vCenter server using the Connect-VIServer servername CMDLET.  Once connected, the following command will loop though a given cluster and modify the setting on every host…

foreach ($esxhost in get-VMHost -Location CLUSTERNAME ) { $esxhost | Set-VMHostAdvancedConfiguration -Name UserVars.SuppressShellWarning -Value 1 }

And there you go!  A happy, non warning triangle life for you!

vSphere Syslog Collector – Install and Configure

I've always used vi-logger from within the vSphere Management Assistant to deal with my syslogging of our ESXi servers, that is until our last upgrade to vSphere 5.  The vi-logger command is no more within the 5.0 version of the VMA so I began looking from some alternate solutions.  Now I could of went out and used a Kiwi product or Splunk or configured a Linux box to do our syslogging, however I thought i would give the vSphere Syslog Collector that is bundled with the vCenter installation media a shot.  Honestly I don't find syslog to be a real science.  You centralize the log files, not a big deal, but having a solution all from one vendor is kind of nice.  The vSphere Syslog Collector does exactly what it says; it collects the log files from the ESXi hosts, but it also gives you some status information from within a vCenter plugin as well.  As well, it's a pretty easy install and config as you will see below.

First off mount the ISO of the vCenter installer on the server you would like to act as your collector and select 'VMware Syslog Collector' and click 'Install'.  During the install (and in VMware's documentation) it is called the vSphere Syslog Collector, however on the menu it's called the VMware Syslog Collector.  Let's just say VSC for short to cover off both names…

After accepting the EULA and licensing you should be presented with the Destinations screen.  Here we need to do a couple of things; First, select where you want the collector application to be installed and secondly, where the logs that are collected are going to be stored (Repository directory).  Also, we have the option here to chose how large we want the log files to grow as well as the number of rotations to keep.  I left all of these values at their defaults, except for the repository directory as I wanted to place this on some lower level, cheaper storage.

Next we need to chose a setup type.  I chose to go with VMware vCenter Server installation as I wanted to integrate this with my vCenter instance. Otherwise, you can chose the 'Standalone Installation' option.  

After selecting your setup type, if choosing to integrate with vCenter you will need to provide login credentials to your vCenter Server.  For the most part this should be pretty straightforward.

Next up is ports and protocols.  Again, I left all of these at default, however you may wish to change the ports that the syslog collector operate on.

Then it's just matter of specifying how it should be defined on the network and letting it install…

So that's it, the collector is now installed.  One more step, we need to tell the desired hosts where we want to ship their logs to.  This can be done in a few different ways, all accomplishing the same thing, but, to each his own, here are the methods that I'm aware of.

1. The GUI – for the non command line type people.

Select your desired host which you want to syslog.  Go to Configuration->Advanced Settings (under Software)->Syslog->Global.  From here it is as simple as setting the hostname or IP address of your syslog server in the syslog.Global.LogHost option.

***Updated April 2012***

Also, be sure to open up the syslog ports within the firewall built into ESXi itself.  Go to Configuration->Security Profile and click 'Properties' in the Firewall section.  It can be as simple as just checking the box next to syslog, however if you would like to further secure your environment you can click the 'Firewall' button at the bottom and specify which IP address/networks are allowed to connect through these ports.


2. The ESXi Command Line space

Using the following two commands you can do the exact same thing as explained in #1.

esxcli system syslog config set –loghost=vCenter01
esxcli system syslog reload
Updated – And the firewall commands to open up the correct ports and restrict access to your syslog server.
esxcli network firewall ruleset set –ruleset-id=syslog –enabled=true –allowed-all=false
esxcli network firewall ruleset allowedip add –ruleset-id syslog –ip-address

3. Host Profiles 

For those with larger installations, you can certainly set the syslog information in a host profile and remediate that against your hosts.  Those setting are located within the profile under the 'Advanced Configuration Option and the same 'Syslog.Global.logHost' option.  *** NOTE*** Until you actually create a host profile from a host that has already had this advanced option setup you will not see this option'.


As well, don't forget to set the firewall options for your syslog server in the host profile under the Firewall Configuration -> Ruleset Configuration ->syslog – Ruleset section.

4. PowerCLI

Things begin to get a little fuzzy here.  If you try to run the get and set VMHostSyslogServer cmdlets on ESXi 5 you will receive an error stating that the host isn't supported for those cmdlets, however, they still work, they still setup the syslog server.  The proper way to do this through powershell is using the get and set VMHostAdvancedConfiguration cmdlets examples below.  And once again, I found even this to be a bit quirky in the sense that I couldn't get the set-VMHostAdvancedConfiguration to just accept a -Name and -Value for the setup, but had to use the -NameValue pairing instead.  Also I'm sure someone that knows powershell (not me 🙂 ) can rock this out on one line, but for now, this is what I got.

$sysloginfo = get-VMHostAdvancedConfiguration -Name "syslog.Global.logHost" -VMhost "IP of host that is already setup"
Set-VMHostAdvancedConfiguration -VMHost "IP of host you want to setup" -NameValue $sysloginfo


As for enabling the syslog in the firewall that can be achieved with the following command

Get-VMHostFirewallException -VMhost hostname -name syslog | Set-VMHostFirewallException -Enabled $true

But when it comes to setting the allowed IP I cannot for the life of me find a way to do this…I'll update later if I do, or if you do, please let me know in the comments. 🙂

So there you have it!  A fully functional instance of the vSphere Syslog Collector.  As always comments, questions, concerns, rants – put'em in the comments 🙂