Tag Archives: Powercli

Practise makes perfect! More PowerCLI APC Powerchute Network Shutdown Goodness (Now with Power On!)

Picture this, a quaint little city in Eastern Ontario. It’s been at least a couple of months since we have really had any rain. My tomato plants are dying, my lawn is completely brown, but the datacentre, it’s chugging along nicely…Then it hits us, the first rainfall (of any real value) of the summer. Finally maybe things will start to grow again. Chatter around the office turns to the weather yet again, people are smiling, happy, can’t wait to get home see if maybe there was a chance that their grass might have turned a slight shade of green…and then, nothing but darkness and the faint sound of our neighbouring businesses generator kicking on…!

Are you kidding me? It rains once this summer and it knocks the power out? Wow! No big deal though right? We have our Powerchute Network Shutdown all configured to peel through and shut-down all of our physical hardware, and a while back I wrote this nifty little script to shutdown the virtual infrastructure, no problem!

And thus the title for this blog post – Practise makes Perfect! Turns out that I could have been a little more efficient in my script. Initially I was looping through hosts one by one shutting off VMs, then waiting, then checking, then going through for another pass, and then moving on to the next host… Well, with 10 minutes left on battery I don’t have the time to complete this process on our 8 production hosts, let alone an environment of any type of size. Needless to say the VMs and hosts that didn’t get the time of the day with the script fell hard. Thankfully, there was no corruption or issues resulting from it, but still, I don’t like things not going as planned . When I’m standing in the datacenter I like to see all of the blinky lights shut off before those air conditioners stop pumping out coolness.. So out of my failures I give you the new and improved shutdown script (now with Power ON). You can head straight here to pull down both the power off and power on scripts, but they are explained in a bit more detail below as there is a little bit of configuration to get the power on functionality to work….  As well a big thanks goes out to a fellow Toronto VMUGer (VMUGite ??) Eric Wright (blog / twitter ) and his ‘The Art of Shutdown‘ post a couple of months back.  Eric wrote a great script to shut things down and even has a great little ‘test’ mode in his version.  I got a few lines of code and ideas from Eric’s script so go check it out as well

Power Off the VMs

So first off is the shutdown script – you can get the complete version here.

Just a few notes regarding my updates to the script.

if ($keyword -ne $mysecret) 
{ 
    Write-Host "You haven't passed the proper detonation sequence...ABORTING THE SCRIPT" -ForegroundColor red
    exit
}

So first off you see here that if you call the script without a parameter or with a parameter that doesn’t match our secret keyword inside the script the whole thing aborts.  This is to stop us from simply double clicking this or accidenttally unleashing a disaster amongst ourselves.

Get-VM -Location $cluster | where-object {$_.PowerState -eq "PoweredOn" } | Select Name | Export-CSV c:\Scripts\Shutdown\PowereredOnVMGuests.csv

Another important addition to the script.  Basically this dumps a list of the powered on VMs to a csv file.  This will be used in the partner power on script once power has been restored.

The rest of the script is pretty straight forward.  Basically sets DRS to manual mode, disables HA, gracefully shuts down the VMs (any without VMware Tools installed get a hard shutdown), waits a couple of minutes and does another pass, then procedes to shutdown the hosts.  Again, the complete poweroff script can be downloaded here.

Power On the VMs

So all is good, VMs got powered off, but what happens when power comes back up.  Even if you do get your hosts powered back on they aren’t going to turn on your VMs automatically.  Now I know there is a start and stop with host setting but when you are in a DRS cluster this doesn’t do you much good as it is a host setting and VMs move between hosts.  Thus the power on script!  So this script will sit on the same box as your shutdown script.  It needs to be outside of your virtual infrastructure.  My vCenter Server is a physical box so I have placed both the shutdown and the power on scripts there.  Also, you will need to have this script be triggered on start-up for this machine.  This is simply do to the fact that I don’t want to be running any scripts in the middle of the night if power is restored…I’d rather walk in to a fully functional datacenter the next morning 🙂

The full script can be downloaded here but below are a few explanations of what happens….

if (Test-Path $filename)

The complete script is wrapped in this if statement.  Meaning if the power off dump file doesn’t exist then the script will not execute.

    while ((Get-Service vpxd).Status -ne "Running")
    {
        Write-Host "." -nonewline
        Sleep 2
        Write-Host "." -nonewline
        Sleep 2
        Write-Host "." -nonewline
        Sleep 2 
    }

This repeating loop basically waits for the vCenter Service to be started before the script can continue.  Since this script should be executed during startup we may have to spend some time here waiting for vCenter to get all its ducks in a row 🙂

    foreach ($iVM in $myImportantVMs)
    {
        Write-Host "Powering on $iVM ..." -nonewline
        Start-VM $iVM
        Sleep 5
        Write-Host "DONE" -Foregroundcolor Green   
    }

Before reading the dump file and getting into the thick of everything we have the option to power on some of the more important VMs first.  So if you have any mail servers, dhcp servers, dns servers, it would be a good idea to put them into $myImportantVMs to have them started first. Once these VMs are on a similar type function will power on the rest of the VMs in the csv file, one by one, every 5 seconds.  You can set the sleep command to whatever you want but 5 seems good for me.

    rename-item "$fileName" "$nameOnly-$DateStamp$extOnly"

This is nearing the end of the script and basically appends the date to the csv file.  This prevents the script from running on next startup – unless you have another power outage of course.

So there you have it – a complete power off script and a complete power on script.  I know I’ve rambled a little bit in the post but I’m just in that sort of mood.  If you need a hand setting things up or have any questions, concerns, improvements, criticism don’t hesitate to leave them in the comments.  I’ll get back to you….  And yes, my tomato plants are all dead!

Getting rid of that pesky shell warning in ESXi 5

We all know that we can enable SSH and the ESXi shell from within the vSphere client or through the DCUI.  This is a great feature that lets us get into the ESXi command space and run things like esxtop, esxcli commands, etc…  Problem being, that once these shells are enabled we get that pesky shell/SSH warning displayed in our vSphere client, as well, that all too familiar yellow triangle gets labeled on our host.  Now, I don't like seeing any warnings on my hosts, especially those dealing with something as minor as SSH.  Good thing is, there is a very easy way to remove or suppress these warnings.

First off, the advanced configuration setting to do this is located in the software section under 'Advanced Settings'->UserVars->UserVars.SuppressShellWarning'.  By default this setting will be set to 0, meaning display the warning.  To hide it, simply set this option to 1.

There you go!  Easy enough… if you only have one host!  But what if you had multiple clusters full of multiple hosts…. well, that's where PowerCLI comes into play.  First off, connect to your vCenter server using the Connect-VIServer servername CMDLET.  Once connected, the following command will loop though a given cluster and modify the setting on every host…

foreach ($esxhost in get-VMHost -Location CLUSTERNAME ) { $esxhost | Set-VMHostAdvancedConfiguration -Name UserVars.SuppressShellWarning -Value 1 }

And there you go!  A happy, non warning triangle life for you!

vSphere Syslog Collector – Install and Configure

I've always used vi-logger from within the vSphere Management Assistant to deal with my syslogging of our ESXi servers, that is until our last upgrade to vSphere 5.  The vi-logger command is no more within the 5.0 version of the VMA so I began looking from some alternate solutions.  Now I could of went out and used a Kiwi product or Splunk or configured a Linux box to do our syslogging, however I thought i would give the vSphere Syslog Collector that is bundled with the vCenter installation media a shot.  Honestly I don't find syslog to be a real science.  You centralize the log files, not a big deal, but having a solution all from one vendor is kind of nice.  The vSphere Syslog Collector does exactly what it says; it collects the log files from the ESXi hosts, but it also gives you some status information from within a vCenter plugin as well.  As well, it's a pretty easy install and config as you will see below.

First off mount the ISO of the vCenter installer on the server you would like to act as your collector and select 'VMware Syslog Collector' and click 'Install'.  During the install (and in VMware's documentation) it is called the vSphere Syslog Collector, however on the menu it's called the VMware Syslog Collector.  Let's just say VSC for short to cover off both names…

After accepting the EULA and licensing you should be presented with the Destinations screen.  Here we need to do a couple of things; First, select where you want the collector application to be installed and secondly, where the logs that are collected are going to be stored (Repository directory).  Also, we have the option here to chose how large we want the log files to grow as well as the number of rotations to keep.  I left all of these values at their defaults, except for the repository directory as I wanted to place this on some lower level, cheaper storage.

Next we need to chose a setup type.  I chose to go with VMware vCenter Server installation as I wanted to integrate this with my vCenter instance. Otherwise, you can chose the 'Standalone Installation' option.  

After selecting your setup type, if choosing to integrate with vCenter you will need to provide login credentials to your vCenter Server.  For the most part this should be pretty straightforward.

Next up is ports and protocols.  Again, I left all of these at default, however you may wish to change the ports that the syslog collector operate on.

Then it's just matter of specifying how it should be defined on the network and letting it install…

So that's it, the collector is now installed.  One more step, we need to tell the desired hosts where we want to ship their logs to.  This can be done in a few different ways, all accomplishing the same thing, but, to each his own, here are the methods that I'm aware of.

1. The GUI – for the non command line type people.

Select your desired host which you want to syslog.  Go to Configuration->Advanced Settings (under Software)->Syslog->Global.  From here it is as simple as setting the hostname or IP address of your syslog server in the syslog.Global.LogHost option.

***Updated April 2012***

Also, be sure to open up the syslog ports within the firewall built into ESXi itself.  Go to Configuration->Security Profile and click 'Properties' in the Firewall section.  It can be as simple as just checking the box next to syslog, however if you would like to further secure your environment you can click the 'Firewall' button at the bottom and specify which IP address/networks are allowed to connect through these ports.

 

2. The ESXi Command Line space

Using the following two commands you can do the exact same thing as explained in #1.

esxcli system syslog config set –loghost=vCenter01
esxcli system syslog reload
 
Updated – And the firewall commands to open up the correct ports and restrict access to your syslog server.
 
esxcli network firewall ruleset set –ruleset-id=syslog –enabled=true –allowed-all=false
esxcli network firewall ruleset allowedip add –ruleset-id syslog –ip-address 192.168.42.150
 

3. Host Profiles 

For those with larger installations, you can certainly set the syslog information in a host profile and remediate that against your hosts.  Those setting are located within the profile under the 'Advanced Configuration Option and the same 'Syslog.Global.logHost' option.  *** NOTE*** Until you actually create a host profile from a host that has already had this advanced option setup you will not see this option'.

Updated 

As well, don't forget to set the firewall options for your syslog server in the host profile under the Firewall Configuration -> Ruleset Configuration ->syslog – Ruleset section.

4. PowerCLI

Things begin to get a little fuzzy here.  If you try to run the get and set VMHostSyslogServer cmdlets on ESXi 5 you will receive an error stating that the host isn't supported for those cmdlets, however, they still work, they still setup the syslog server.  The proper way to do this through powershell is using the get and set VMHostAdvancedConfiguration cmdlets examples below.  And once again, I found even this to be a bit quirky in the sense that I couldn't get the set-VMHostAdvancedConfiguration to just accept a -Name and -Value for the setup, but had to use the -NameValue pairing instead.  Also I'm sure someone that knows powershell (not me 🙂 ) can rock this out on one line, but for now, this is what I got.

$sysloginfo = get-VMHostAdvancedConfiguration -Name "syslog.Global.logHost" -VMhost "IP of host that is already setup"
Set-VMHostAdvancedConfiguration -VMHost "IP of host you want to setup" -NameValue $sysloginfo

Updated

As for enabling the syslog in the firewall that can be achieved with the following command

Get-VMHostFirewallException -VMhost hostname -name syslog | Set-VMHostFirewallException -Enabled $true

But when it comes to setting the allowed IP I cannot for the life of me find a way to do this…I'll update later if I do, or if you do, please let me know in the comments. 🙂

So there you have it!  A fully functional instance of the vSphere Syslog Collector.  As always comments, questions, concerns, rants – put'em in the comments 🙂

The Resource Pool Priority Pie Paradox – Part 3 – The Powershell Code

Part 2 – The 4:1 Formula
Part 3 – Powershell Scripts

So in parts 1 and 2 of this series I explained (hopefully) what the resource pool priority pie paradox is and how to punch some key numbers into a formula and figure out what exactly you could set for custom shares on a production and test resource pool in order to come out with that 4:1 ratio.  In this part I'll show you how you can get those numbers from your own environment using the VMware PowerCli without having to do the math yourself.

Just a note, for the script to work you must only have two resource pools, which you assign to variables at the beginning of the script.  Along with the resource pools you need to assign your vCenter server name as well as the cluster name.  You may also need to pump in your credentials if you aren't running this from a trusted client.

Anyways, the script is below, feel free to modify as you please.  If you see a spot where I could be more efficient or if I'm doing anything completely crazy let me know…  You can also download it here if you don't feel much like copying and pasting..:)

#################################################################################
# ResourcePoolPie

#
# This script will poll look at VMs residing in a production and a test resource
# pool and determine how to best set the custom share value in order to maintian
# a 4:1 Production:Test ratio.  Note, you must be setup in a Production and Test
# Environment inside of one cluster.  Its kind of unique to my environment and
# may not apply to everyone elses, but feel free to change and modify what you
# need in order to make it work for you.
#
# The following variables will need to be assigned
#
#   $vcenterserver = ip address of vcenter
#   $clustername = name of cluster containing resource pools
#   $prodname = name of production resource pool
#   $testname = name of test resource pool
#
# Created By: Mike Preston, 2011
#
#
#################################################################################

Add-PSSnapin VMware.VimAutomation.Core
# Assign appropriate values to following variables.
$vcenterserver = ""
$clustername = ""
$prodname = ""
$testname = ""
# establish connection to vcenter server
$connection = Connect-VIServer $vcenterserver
# get cluster information
$clus = get-Cluster -Name $clustername | get-view
# get resource pool information
$prodpool = Get-ResourcePool -Location $clustername -Name $prodname
$testpool = Get-ResourcePool -Location $clustername -Name $testname
# get a list of vms in production and test
$prodvms = get-vm -Location $prodpool | where { $_.PowerState -eq "PoweredOn" }
$testvms = get-vm -Location $testpool | where { $_.PowerState -eq "PoweredOn" }
# initialize counters to zero
$totalprodcpu = 0
$totaltestcpu = 0
$totalprodmem = 0
$totaltestmem = 0
# loop through production pool and total cpu/memory
foreach ($vm in $prodvms)
{
    $totalprodcpu = $totalprodcpu + $vm.NumCPU
    $totalprodmem = $totalprodmem + $vm.MemoryMB
}
# loop through test pool and total cpu/memory
foreach ($vm in $testvms)
{
    $totaltestcpu = $totaltestcpu + $vm.NumCPU
    $totaltestmem = $totaltestmem + $vm.MemoryMB
}
# Begin CPU calculations
Write-Host "CPU Configuration"
Write-Host "=========================================================="
write-host "Production Pool contains" $prodvms.count "Powered On VMs containing" $totalprodcpu "cpus"
write-host "Test Pool contains" $testvms.count "Powered On VMs containing" $totaltestcpu "cpus"
write-host "Cluster has" $clus.summary.effectivecpu "MHZ to hand out"
#populate variables for formula a(4w-3x) = nx

 

$a = $totalprodcpu
$w = $clus.summary.effectivecpu
$n = $totalprodcpu + $totaltestcpu
Write-Host ""
# lets solve x 🙂
Write-Host "Lets plug these numbers into our formula and solve x (Production Share)"
Write-Host "a = total number in production, w = total resources, n = total number in prod and test"
Write-Host "——————————–"
Write-Host "a(4w-3x) = nx) "
Write-Host "$a(4($w) – 3(x)) = $n(x)) "
# lets get some tmp vars initialized
$tmp1 = 4 * $w
Write-Host "$a($tmp1 – 3x) = $n(x)"
$tmp2 = $a * $tmp1
$tmp3 = 3 * $a
Write-Host "$tmp2 – $tmp3(x) = $n(x)"
$tmp4 = $n + $tmp3
Write-Host "$tmp4(x) = $tmp2"
$prodpoolresources = $tmp2 / $tmp4
Write-Host "x = $prodpoolresources"
Write-Host "——————————–"
$prodpoolresources = [Math]::Round($prodpoolresources,0)
$testpoolresources = $w – $prodpoolresources
$prodsharepercpu = $prodpoolresources / $totalprodcpu
$prodsharepercpu = [Math]::Round($prodsharepercpu,0)
$testsharepercpu = $testpoolresources / $totaltestcpu
$testsharepercpu = [Math]::Round($testsharepercpu,0)
# Display recommendations
Write-Host "Recommended Share setting for production $prodpoolresources Mhz split between" $prodvms.count "Vms with" $totalprodcpu "cpus resulting in" $prodsharepercpu "Mhz per cpu"
Write-Host "Recommended Share setting for test $testpoolresources Mhz split between" $testvms.count "Vms with" $totaltestcpu "cpus resulting in" $testsharepercpu "mhz per cpu"
# Begin calculating Memory
Write-Host "=========================================================="
Write-Host "Memory Configuration"
Write-Host "=========================================================="
write-host "Production Pool contains" $prodvms.count "Powered On VMs containing" $totalprodmem "MB of Memory"
write-host "Test Pool contains" $testvms.count "Powered On VMs containing" $totaltestmem "MB of Memory"
write-host "Cluster has" $clus.summary.effectivememory "MB of memory to hand out"
#populate variables for formula a(4w-3x) = nx
$a = $totalprodmem
$w = $clus.summary.effectivememory
$n = $totalprodmem + $totaltestmem
Write-Host ""
# lets solve x 🙂
Write-Host "Lets plug these numbers into our formula and solve x (Production Share)"
Write-Host "——————————–"
Write-Host "a(4w-3x) = nx) "
Write-Host "$a(4($w) – 3(x)) = $n(x)) "
# lets get some tmp vars initialized
$tmp1 = 4 * $w
Write-Host "$a($tmp1 – 3x) = $n(x)"
$tmp2 = $a * $tmp1
$tmp3 = 3 * $a
Write-Host "$tmp2 – $tmp3(x) = $n(x)"
$tmp4 = $n + $tmp3
Write-Host "$tmp4(x) = $tmp2"
$prodpoolresources = $tmp2 / $tmp4
Write-Host "x = $prodpoolresources"
Write-Host "——————————–"
$prodpoolresources = [Math]::Round($prodpoolresources,0)
$testpoolresources = $w – $prodpoolresources
$prodsharepermb = $prodpoolresources / $totalprodmem
$testsharepermb = $testpoolresources / $totaltestmem
# Output results
Write-Host "Recommended Share setting for production $prodpoolresources MB split between" $prodvms.count "Vms with" $totalprodmem " MB of RAM resulting in $prodsharepermb shares per MB"
Write-Host "Recommended Share setting for test $testpoolresources MB split between" $testvms.count "Vms with" $totaltestmem " MB of RAM resulting $testsharepermb shares per MB"
Write-Host "=========================================================="

 

 

Part 2 – The 4:1 Formula
Part 3 – Powershell Scripts

Use PowerCli to shutdown VM’s and Hosts when running on battery.

UPDATE – I’ve had the unfortunate chance of testing this script a few times and have found that its not as efficient as it could be.   If I were you I would check out my updated post Practise makes perfect! More PowerCLI APC Powerchute Network Shutdown Goodness (Now with Power On!) as it completes much faster and has a function to turn the VMs back on once power is restored.  Thanks!

A few months ago, in order to prepare for the release of ESXi and vSphere 5 I went through the motions of migrating all of our ESX hosts to ESXi. I thought I had all my ducks in a row concerning all of our third party solutions that connect to our environment (Veeam, VCops, etc) but one that completely slipped my mind was our APC and vGhetto shutdown scripts that were running inside our vMA. Since our then current solution relied on the ability to ssh into the hosts and ESXi by default doesn’t have an ssh server running, the scripts proved to be somewhat, how shall I say this….useless (and yes, I found out the hard way!!). When looking for alternatives I first went to APC and was delighted to see that they finally had a version of their powerchute network shutdown available for the vMA appliance. What I found out though was that it basically peeled through your list of servers that you have connected to the vMA and shut them down accordingly, which in my case is not the answer. I have multiple ESX(i) instances connected to the vMA which reside in many different locations, so if the power failed at our main datacentre, essentially our vMA would receive the shutdown command and initiate a shutdown sequence on ESX hosts in an offsite location. So, back to the drawing board as that was definitely not the answer. I liked the way that lam??? Ghetto scripts worked originally being that you could specify which hosts and even which vMA to shut down So i decided to recreate, or at least get close too that same functionality in powershell. Below is what I came up with, and the script is very much in its infancy so if you can see some spots for improvement please let me know! Also, I don’t claim to be a powershell expert in any sort of sense so definitely don’t take this directly not production, test it in a lab first or even in comment the shutdown lines and have a look at the output. I placed the script on a windows box outside of our virtual infrastructure, so essentially the physical machine receives the shutdown command, but before shutting itself down it runs the powershell script to gracefully shutdown the virtual infrastructure (vms and hosts) first.

Anyways, here it is and if you have any suggestions , comments or questions don’t hesitate to ask. We can muddle through this together 🙂

#################################################################################
# Shutdown VMs and Hosts
#
# This script will loop through a list of ESXi hosts and initiate shutdown
# commands to the vm’s residing on them. If VMware Tools is installed, the
# script will attempt to do a graceful shutdown. If VMware tools is not
# installed a hard power off will be issued. One note, if you have any VMs that
# you would like to remain on until the end of the process be sure to put their
# names in the $vmstoleaveon variable and also be sure they reside on the last
# host listed in the $hoststoprocess variable.
#
# i.e If I wanted to have VM1 and VM2 stay on till the end I would have to be
# sure that they reside on esxi-03 and my variables would be setup as follows
#
# $vmstoleaveon = “VM1 VM2”
# $listofhosts = “esxi-01 esxi-02 esxi-03”
#
# Created By: Mike Preston, 2011
# Updates: MWP – March 2012 – how not looks to see if any VMs are still on after
#                doing initial shutdown, then runs a hard stop, and enters
#                maintenance mode before shutting down.

 

#
#################################################################################
# list of hosts to process
$listofhosts = “esxi-01”, “esxi-02”, “esxi-03”
#list of vm’s to ‘go down with the ship’ – vms must reside on last host in above list.
$vmstoleave = “VM1”, “VM2”
#loop through each host
Foreach ($esxhost in $listofhosts)
{
$currentesxhost = get-vmhost $esxhost
Write-Host “Processing $currentesxhost”
#loop through each vm on host
Foreach ($VM in ($currentesxhost | Get-VM | where { $_.PowerState -eq “PoweredOn” }))
{
Write-Host “====================================================================”
Write-Host “Processing $vm”
# if this is a vm that is supposed to go down with the ship.
if ($vmstoleave -contains $vm)
{
Write-Host “I am $vm – I will go down with the ship”
}
else
{
Write-Host “Checking VMware Tools….”
$vminfo = get-view -Id $vm.ID
# If we have VMware tools installed
if ($vminfo.config.Tools.ToolsVersion -eq 0)
{
Write-Host “$vm doesn’t have vmware tools installed, hard power this one”
# Hard Power Off
Stop-VM $vm -confirm:$false
}
else
{
write-host “I will attempt to shutdown $vm”
# Power off gracefully
$vmshutdown = $vm | shutdown-VMGuest -Confirm:$false
}
}
Write-Host “====================================================================”
}
    Write-Host “Initiating host shutdown in 40 seconds”
    sleep 40
    #look for other vm’s still powered on.
    Foreach ($VM in ($currentesxhost | Get-VM | where { $_.PowerState -eq “PoweredOn” }))
    {
        Write-Host “====================================================================”
        Write-Host “Processing $vm”
        Stop-VM $vm -confirm:$false
        Write-Host “====================================================================”
    }
    #Shut down the host
    Sleep 20
    Set-VMhost -VMhost $currentesxHost -State Maintenance
    Sleep 15
    $currentesxhost | Foreach {Get-View $_.ID} | Foreach {$_.ShutdownHost_Task($TRUE)}
}
Write-Host “Shutdown Complete”