Picture this, a quaint little city in Eastern Ontario. It’s been at least a couple of months since we have really had any rain. My tomato plants are dying, my lawn is completely brown, but the datacentre, it’s chugging along nicely…Then it hits us, the first rainfall (of any real value) of the summer. Finally maybe things will start to grow again. Chatter around the office turns to the weather yet again, people are smiling, happy, can’t wait to get home see if maybe there was a chance that their grass might have turned a slight shade of green…and then, nothing but darkness and the faint sound of our neighbouring businesses generator kicking on…!
Are you kidding me? It rains once this summer and it knocks the power out? Wow! No big deal though right? We have our Powerchute Network Shutdown all configured to peel through and shut-down all of our physical hardware, and a while back I wrote this nifty little script to shutdown the virtual infrastructure, no problem!
And thus the title for this blog post – Practise makes Perfect! Turns out that I could have been a little more efficient in my script. Initially I was looping through hosts one by one shutting off VMs, then waiting, then checking, then going through for another pass, and then moving on to the next host… Well, with 10 minutes left on battery I don’t have the time to complete this process on our 8 production hosts, let alone an environment of any type of size. Needless to say the VMs and hosts that didn’t get the time of the day with the script fell hard. Thankfully, there was no corruption or issues resulting from it, but still, I don’t like things not going as planned . When I’m standing in the datacenter I like to see all of the blinky lights shut off before those air conditioners stop pumping out coolness.. So out of my failures I give you the new and improved shutdown script (now with Power ON). You can head straight here to pull down both the power off and power on scripts, but they are explained in a bit more detail below as there is a little bit of configuration to get the power on functionality to work…. As well a big thanks goes out to a fellow Toronto VMUGer (VMUGite ??) Eric Wright (blog / twitter ) and his ‘The Art of Shutdown‘ post a couple of months back. Eric wrote a great script to shut things down and even has a great little ‘test’ mode in his version. I got a few lines of code and ideas from Eric’s script so go check it out as well…
Power Off the VMs
So first off is the shutdown script – you can get the complete version here.
Just a few notes regarding my updates to the script.
|
51 52 53 54 55 |
if ($keyword -ne $mysecret) { Write-Host "You haven't passed the proper detonation sequence...ABORTING THE SCRIPT" -ForegroundColor red exit } |
So first off you see here that if you call the script without a parameter or with a parameter that doesn’t match our secret keyword inside the script the whole thing aborts. This is to stop us from simply double clicking this or accidenttally unleashing a disaster amongst ourselves.
|
70 |
Get-VM -Location $cluster | where-object {$_.PowerState -eq "PoweredOn" } | Select Name | Export-CSV c:\Scripts\Shutdown\PowereredOnVMGuests.csv |
Another important addition to the script. Basically this dumps a list of the powered on VMs to a csv file. This will be used in the partner power on script once power has been restored.
The rest of the script is pretty straight forward. Basically sets DRS to manual mode, disables HA, gracefully shuts down the VMs (any without VMware Tools installed get a hard shutdown), waits a couple of minutes and does another pass, then procedes to shutdown the hosts. Again, the complete poweroff script can be downloaded here.
Power On the VMs
So all is good, VMs got powered off, but what happens when power comes back up. Even if you do get your hosts powered back on they aren’t going to turn on your VMs automatically. Now I know there is a start and stop with host setting but when you are in a DRS cluster this doesn’t do you much good as it is a host setting and VMs move between hosts. Thus the power on script! So this script will sit on the same box as your shutdown script. It needs to be outside of your virtual infrastructure. My vCenter Server is a physical box so I have placed both the shutdown and the power on scripts there. Also, you will need to have this script be triggered on start-up for this machine. This is simply do to the fact that I don’t want to be running any scripts in the middle of the night if power is restored…I’d rather walk in to a fully functional datacenter the next morning 🙂
The full script can be downloaded here but below are a few explanations of what happens….
|
39 |
if (Test-Path $filename) |
The complete script is wrapped in this if statement. Meaning if the power off dump file doesn’t exist then the script will not execute.
|
47 48 49 50 51 52 53 54 55 |
while ((Get-Service vpxd).Status -ne "Running") { Write-Host "." -nonewline Sleep 2 Write-Host "." -nonewline Sleep 2 Write-Host "." -nonewline Sleep 2 } |
This repeating loop basically waits for the vCenter Service to be started before the script can continue. Since this script should be executed during startup we may have to spend some time here waiting for vCenter to get all its ducks in a row 🙂
|
74 75 76 77 78 79 80 |
foreach ($iVM in $myImportantVMs) { Write-Host "Powering on $iVM ..." -nonewline Start-VM $iVM Sleep 5 Write-Host "DONE" -Foregroundcolor Green } |
Before reading the dump file and getting into the thick of everything we have the option to power on some of the more important VMs first. So if you have any mail servers, dhcp servers, dns servers, it would be a good idea to put them into $myImportantVMs to have them started first. Once these VMs are on a similar type function will power on the rest of the VMs in the csv file, one by one, every 5 seconds. You can set the sleep command to whatever you want but 5 seems good for me.
|
106 |
rename-item "$fileName" "$nameOnly-$DateStamp$extOnly" |
This is nearing the end of the script and basically appends the date to the csv file. This prevents the script from running on next startup – unless you have another power outage of course.
So there you have it – a complete power off script and a complete power on script. I know I’ve rambled a little bit in the post but I’m just in that sort of mood. If you need a hand setting things up or have any questions, concerns, improvements, criticism don’t hesitate to leave them in the comments. I’ll get back to you…. And yes, my tomato plants are all dead!
This is amazing! Thanks for putting this together. This is the exact problem we are having after upgrading to vSphere 5, we have to use the network shutdown tool. Will this script run with normal powershell, or is there something I have to do special to get the network shutdown software to call it and run it?
Will this poweroff script work on ESX 5.1 ?
Hey Remco – send me an email at mwpreston [at] mwpreston dot net – i’ll give you the updated script – it’s quite simple 🙂
Thank’s, great script,
i will test it on our Cluster, but i would like to ask you if there is the possibility to exclude some vm’s from the primary shutdown proces (DC and DB server) and do that servers last in a particular order?
Hi great blog!
we are just started in virtualization and also having similar challenge. would this script require Powerchute Network Shutdown, or can it be run as well with typical APC UPS w/o UPS network Mgt card?
APC release script but requires UPS network mgt card for HA enabled.
thanks you for sharing.
You shouldn’t have any issues with it, just may need to call powershell from a cmd or bat file. I think most the APC have a ‘Run this command before shutdown’ or something along those lines!