February 25th Toronto VMUG Agenda Finalized

1362963110-TorontoVMUG1-300x79Consider this post what you will – call it self promotion, a plea to get more registrations, an advertisement for the VMUG – it really doesn’t matter.  The truth is I’m just excited about the Toronto VMUG UserCon coming up on Feb 25th and wanted to spread the word about what we have put together on the agenda for the day.  Last years UserCon was a good one – everything went off without a hitch and honestly it will be hard one to top but I think we’ve done a pretty good job…  With that said, before you even read on I’d suggest you register below – then come back!

First up let’s start with the keynote!  We are super pumped this year to bring you Nick Marshall (blog / twitter) from VMware to deliver our morning keynote – from what I can gather Nick will be talking about IT careers and how to develop and elevate your own IT profession.  Judging by past conversations at VMUG events with attendees, and listening to where a lot of the “hallway” conversations end up I think this subject will be a GREAT fit with our Toronto VMUG attendees and I for one am very excited to hear what Nick has to say.   Although Nick works at VMware he’s done a lot of community driven work in his own time, including authoring both the Mastering VMware vSphere 5.5 and Mastering VMware vSphere 6.0 books, along with blogging, helping out with the vBrownBag podcast and playing a big role in AutoLab!  We are pretty lucky to be able to lure him away from Palo Alto and into the -20C temps currently whistling around Toronto!

Last year we had somewhat of an impromptu VCDX panel at the end of the day – although we had some people who had already left there was great discussions around pretty much every “v” topic you can think of!  This year we decided to repeat this, and at the same time move it to a more structured time – so our lunchtime keynote will consist of 4 very, very bright minds!  Tim Antonowitz (VCDX #112), James Wirth (VCDX #83), Joe Silvagi (VCDX #175), and Eiad Al-Aqqad (VCDX #89) will be on hand just after the lunch hour to talk about pretty much anything we want.  And looking at their bios I really can stress the word anything – between the four of them they have a ton of experience in hyperconvergence, storage, performance optimization, SDDC, automation, NSX, Cloud, professional development… – it just goes on and on – be sure to jot your design questions down and don’t miss this!

Our Demo Zone will be in full effect again this year with Daemon Behr (twitter) doing a customer presentation in regards to risk and design!  I always love seeing the community members present and try to encourage as many to do so as we can.  Big thanks to Daemon for stepping up and doing so…

As always we also have a slew of sponsors, vendors, and partners with some awesome content lined up for you as well.  A big thanks to those people because without them we wouldn’t be able to bring you things like toques and stickers!  This year we have presentations lined up from the likes of Brocade, Veeam, EMC, VMTurbo, Cohesity, Dell, Trend Micro, VCE, Zerto, Nutanix, Tintri, Simplivity, Vertias, and Riverbed – I’m sure you can pick a few names out of that list you recognize!  Also, we have VMware slotted for 8 sessions as well – and with the recent announcements you can bet there will be some great content to talk about.

Wait did you say toques?

toqueThe “Virtualization eh” toques will be back again this year so be on the lookout to nab yourself one of those – we will also have a drawing where one lucky attendee will take home a homelab as well as many giftcards, stickers, etc – you know, the usual swag!

So with all that said I hope to see you at the end of the month in Toronto!  As always you can keep up to date with the happenings of the Toronto VMUG by following the @TorontoVMUG account on Twitter and joining in with the #TOVMUG hashtag!

Quick Fix – Making your inactive NFS datastore active again!

Sticking to my rule of “If it happens more than once I’m blogging about it” I’m bringing you this quick post around an issue I’ve seen a few times in a certain environment.  Although this is solved by only a few esxcli commands I always find it easier for me to remember (and find) if I post it here… :)

Anyways, as it is I have a couple of NFS datastores that sometimes “act up” a bit in terms of their connections.  For the most part they are fine and dandy – however every now and then they show up within the vSphere client as “inactive” and ghosted.  After checking the network (I always try and pin things on the network) it appears that all the connections are fine – Host communicates with storage, storage with host – the same datastores are even functioning fine on other hosts.  Logically my next step is to remount them on the host in question but when trying to unmount and/or remount them through the vSphere client I usually end up with a “Filesystem busy” error.

Thankfully it doesn’t take a lot to fix this issue, but could certainly become tedious if you have many NFS datastores which you need to perform these commands on…

First up, list the NFS datastores you have mounted on the host with the following

esxcli storage nfs list

You should see that the ‘inactive’ datastores are indeed showing up with false under the accessible column.  Make note of the Volume Name, Share Name and Host as we will need this information for the next couple of commands….

Before we can add our datastore back we need to first get rid of it.  The following command takes care of that…

esxcli storage nfs remove -v DATASTORE_NAME

Depending on whether or not you have any VMs registered on the datastore and host you may get an error, you may not – I’ve found it varies :)  Anyways, lastly we simply need to mount the datastore back to our host using the following command…  Be sure to use the exact same values you gathered from the ‘nfs list’ command.

esxcli storage nfs add -H HOST -s ShareName/MountPoint -v DATASTORE_NAME

There you go!  You should now have a happy healthy baby NFS datastore back into your storage pool.  On a side note I’d love to see some sort of esxcli storage nfs remount -v DATASTORE_NAME command go into the command line in order to skip some of these steps – but, hey, for now I’ll just use three commands.

VCSA 6.0 prompting for a manual fsck

One of my VCSA deployments, the only one running 6.0 experienced a switch failure and in result a network outage of roughly 5 minutes the other day.  Not a big deal, but unfortunately this was a very “cost effective” solution and the switch that hosted the production network also hosted the VLANs carrying all of the NFS traffic to the datastores the VCSA resided on as well!  In short, VCSA done got grumpy – after fixing the issues with the switch I ended up at the screen shown below…

VCSAFileSystemError

Not an overly complicated error – just stating that we need to run a file system check on the /dev/mapper/log_vg-log volume manually.  In the past, say with 5.5 I’d just drop to a bash shell and do so – however the default appliance shell in the 6.0 version of VCSA presents a few challenges in doing the same thing.  First off, if I went ahead and gave the root password to the VCSA I was presented with the default menu – the same menu you would receive if you ssh’d to the box under normal circumstances – that said, in the maintenance mode, the shell.set and shell.enable commands don’t work.  So in order to get to a point where we can actually execute fsck we need to do a couple of things…

Grub to bash

So the first thing we need to do is get our VCSA booting into a bash prompt.  To do so, hit CTRL+D at the presented screen and get the box to reboot.  When the boot loader appears we will need to hit the space bar or up/down keys to stop the auto boot process.  Once stopped we can selected ‘p’ to unlock the menu and enter the root password for the box.  We then want to select “e” to edit our boot sequence –  highlight the second line, the one that displays the kernel parameters and select “e” once again.  At the end of that line we will want to append “init=/bin/bash” as shown below – this will boot our system into a bash shell.  Once done, hit “enter” to save and “b” to boot.

grubmenu1

grubmenu2

After the system has booted you should now be sitting at a bash prompt.  On a normal day we would simply run our fsck command here however the file system we are looking to check is still not mounted at this point.  I tried numerous commands and options to try and get it mounted but came up short.  That said running the following command and rebooting our vCenter will switch the login shell for root back to the ‘normal bash’ and allow us to continue

chsh -s /bin/bash root

Once the command has been run and the server rebooted we will be brought back to the same error prompting us to enter the root password.  Go ahead and do that.  This time we will be brought directly to a bash prompt with log_vg-log being available to us!  So, without further ado go ahead and run the following command to complete the file system check.

fsck /dev/mapper/log_vg-log

More than likely you will get numerous prompts asking you whether or not to fix any errors that occur.  Use your discretion here, however I didn’t have much of a choice and needed to say ‘Yes’ to all.  After it’s done give the VCSA another reboot and everything should come back up normally (at least it did for me).  Hopefully this helps push someone in the right direction if they are experiencing similar issues :)

Manually updating the Veeam Proxy Transport and Mount services

With the release of v9 hitting the Internets on Tuesday I’ve been a very busy man upgrading various Veeam Consoles, Proxies, and Repositories.  With nearly 70 different locations to look after you can imagine the amount of proxies and repositories I have, both on and off site all requiring their respective Veeam services to be upgraded.  Mix that together with a few slower WAN connections and I can almost bet that the automated component update that Veeam ships with will naturally fail on a couple servers.

Want the tl;dr version?
Packages are in c:\Program Files\Veeam\Backup and Replication\Packages – copy them to your failed server and install :)

 

Failed to upgrade host components

When this happens to me usually I get some sort of error message like the following

TransportErrorVeeam

For the most part, re-running the automated component update will fix the issue, but there are times when it fails, again, and again, and again.  Usually by the third time I resort to manual intervention.

Manually installing the transport/mount service

First up you need to get a hold of the installation files.  These are located on your Veeam Backup and Replication server under the path C:\Program Files\Veeam\Backup and Replication\Packages – You will find there the individual packages for each service that Veeam provides (mount, transport, tape, etc).  Depending on what services your proxy is providing you may need an number of these.  Since my server was acting as a repository as well as a proxy I simply needed the transport and mount server packages (VeeamTransport.msi and VeeamMountService.msi respectively).  Also, don’t forget that Veeam relies heavily on the .NET framework so you must keep that updated as well – you can find the redistributable installation package for that within the packages folder along side the others (NDP452-KB2901907-x86-x64-AllOS-ENU.exe)

Installation is just like any other install – your typical Next->Next->Done type of scenario.  Once you have ran the required packages head back to Veeam Backup and Replication.  If you are still on the component update screen a ‘refresh’ should update the status of the packages – if not, a rescan of your server within the Backup Infrastructure section is required.

So that’s that – as it turns out the issue on why my installation was failing during the automated process was due to lack of disk space, but nonetheless this is good information to have – If you are looking for more information in regards the the new features within Veeam Backup and Replication v9 I’ve done a post here – feel free to check it out!.

 

Quickfix – Mass editing Veeam VM Attribute settings with PowerShell

Hi – I’m Mike – you may remember me from such blogs as, oh this one last year!  I know, it’s been a while since I’ve written anything, moreso, published anything.  It’s been a hectic couple of months and I’ve got a ton of drafts sitting just waiting to be touched up so stay tuned – I promise to be a bit more visible in the next coming weeks!  Anyways I called this a quick fix so I guess I should get to the point…

VeeamVMAttributeThere is a setting within the notification options inside Veeam Backup and Replication that allows you to write some details to a processed VMs annotations section.  Now I’ve never had a use for this…until now.  I was reporting on a wide variety of things in regards to specific VMs, and having the last successful backup was one of the things that I wanted to report on.  This was within an asp.net application, and dropping to PowerShell and loading up the Veeam cmdlets was something I just didn’t feel like coding within the application.  Also, accessing the Veeam REST API was out of the question seeing as these VMs were being processed by Veeam Standard lisenses – REST is only available within Veeam’s Enterprise Plus offering.  Since I was already connected the vSphere API to gather a bunch of information such as CPU and Memory usage I thought by having Veeam write the last successful backup to the Notes field within vSphere would be a good alternative, as I could just gather that info with the same API calls I was making to vSphere.

One problem presented itself – I had roughly 50 different jobs within the Veeam Backup and Replication deployment that needed to be updated in order to make it happen.  Naturally I looked to automation for the solution – and I found it with PowerShell.  In order to completely enable this feature, and turn off the ‘append’ option you have to touch two different spots within Veeam PowerShell; one, the Set-VBRJobAdvancedViOptions (to enable the actual writing to VM Attributes) and another by setting the VmNotesAppend flag within the ViSourceOptions object.

Its a simple script, and honestly you probably don’t care, or didn’t read any of the introduction stuff above – I know how things work, you came here for the script – well, here its…

$jobs = Get-VBRJob

foreach ($job in $jobs)
{
  $job | Set-VBRJobAdvancedViOptions -SetResultsToVmAttribute $True
  $joboptions = $job.GetOptions()
  $joboptions.ViSourceOptions.VmNotesAppend = $False 
  $job.SetOptions($joboptions)
}

There you have it!  A simple script that peels through all the VBR Jobs within a console, enables the writing to the VM Attribute, and disables the append flag.  An easy script but useful none the less :)

Friday Shorts – JSON, Veeam, Embedded Host Client and more…

I think you’re really beautiful and I feel really warm when I’m around you and my tongue swells up. – Buddy the Elf

JSON will be the death of me

I don’t do a lot of JSON parsing but when I do I tend to get just a little frustrated.  Honestly I almost always try and retrieve API responses as XML if the option is there, but sometimes it isn’t and sorting through JSON is the only alternative!  I’m a newbie to this stuff and still am learning quite a deal about interacting with APIs using various methods – one of the many reason I follow Scott Lowe’s blog regularly.  In a post this week he introduced us to a lightweight command line tool called jq, which looks like it could most definitely help me with my JSON woes – next time I find myself staring down at a page full of squigglies (not sure of the official term for  {}) I’ll give it a go… Thanks Scott!

Embedded Host Client is awesome x 5

I haven’t had a lot of time to spend with the Embedded Host Client fling – a replacement that will allow us to do away with the giant c# installable that we have all grown to love for individual host management.  I did get to use the EHC early on, and it was not the greatest experience – but as time went on and it was released via VMware Flings it is now a lot better, in fact people are loving it!  Here’s a great article outlining 5 reasons why the Embedded Host Client is awesome!

VVOLs 4 DBs!

VVOLs is still a little unknown when it comes to adoption – I haven’t seen a lot of blogs or information out there about widespread adoption!  I do know that there is general excitement out there and definitely benefits to be had by utilizing them, however they are still a little “ripe” in IT years!  That said there’s an interesting series on the VMware blogs going on right now outlining the benefits they can provide to databases – you can check out Part 1 and Part 2 if you like…

The Expert Guide to VMware Disaster Recovery and Data Protection

Shameless plug for myself on this one – A few months ago I began work on a guide for Veeam centering around how to provide Data Protection within a VMware environment – and the result is finally live!  Note – you are going to hit a reg-wall when trying to view this – it’s the nature of the world we live in, but if you do end up checking it out I’d love to know what you think – good or bad, but mostly good!