Monthly Archives: January 2016

VCSA 6.0 prompting for a manual fsck

One of my VCSA deployments, the only one running 6.0 experienced a switch failure and in result a network outage of roughly 5 minutes the other day.  Not a big deal, but unfortunately this was a very “cost effective” solution and the switch that hosted the production network also hosted the VLANs carrying all of the NFS traffic to the datastores the VCSA resided on as well!  In short, VCSA done got grumpy – after fixing the issues with the switch I ended up at the screen shown below…

VCSAFileSystemError

Not an overly complicated error – just stating that we need to run a file system check on the /dev/mapper/log_vg-log volume manually.  In the past, say with 5.5 I’d just drop to a bash shell and do so – however the default appliance shell in the 6.0 version of VCSA presents a few challenges in doing the same thing.  First off, if I went ahead and gave the root password to the VCSA I was presented with the default menu – the same menu you would receive if you ssh’d to the box under normal circumstances – that said, in the maintenance mode, the shell.set and shell.enable commands don’t work.  So in order to get to a point where we can actually execute fsck we need to do a couple of things…

Grub to bash

So the first thing we need to do is get our VCSA booting into a bash prompt.  To do so, hit CTRL+D at the presented screen and get the box to reboot.  When the boot loader appears we will need to hit the space bar or up/down keys to stop the auto boot process.  Once stopped we can selected ‘p’ to unlock the menu and enter the root password for the box.  We then want to select “e” to edit our boot sequence –  highlight the second line, the one that displays the kernel parameters and select “e” once again.  At the end of that line we will want to append “init=/bin/bash” as shown below – this will boot our system into a bash shell.  Once done, hit “enter” to save and “b” to boot.

grubmenu1

grubmenu2

After the system has booted you should now be sitting at a bash prompt.  On a normal day we would simply run our fsck command here however the file system we are looking to check is still not mounted at this point.  I tried numerous commands and options to try and get it mounted but came up short.  That said running the following command and rebooting our vCenter will switch the login shell for root back to the ‘normal bash’ and allow us to continue

chsh -s /bin/bash root

Once the command has been run and the server rebooted we will be brought back to the same error prompting us to enter the root password.  Go ahead and do that.  This time we will be brought directly to a bash prompt with log_vg-log being available to us!  So, without further ado go ahead and run the following command to complete the file system check.

fsck /dev/mapper/log_vg-log

More than likely you will get numerous prompts asking you whether or not to fix any errors that occur.  Use your discretion here, however I didn’t have much of a choice and needed to say ‘Yes’ to all.  After it’s done give the VCSA another reboot and everything should come back up normally (at least it did for me).  Hopefully this helps push someone in the right direction if they are experiencing similar issues 🙂

Manually updating the Veeam Proxy Transport and Mount services

With the release of v9 hitting the Internets on Tuesday I’ve been a very busy man upgrading various Veeam Consoles, Proxies, and Repositories.  With nearly 70 different locations to look after you can imagine the amount of proxies and repositories I have, both on and off site all requiring their respective Veeam services to be upgraded.  Mix that together with a few slower WAN connections and I can almost bet that the automated component update that Veeam ships with will naturally fail on a couple servers.

Want the tl;dr version?
Packages are in c:\Program Files\Veeam\Backup and Replication\Packages – copy them to your failed server and install 🙂

 

Failed to upgrade host components

When this happens to me usually I get some sort of error message like the following

TransportErrorVeeam

For the most part, re-running the automated component update will fix the issue, but there are times when it fails, again, and again, and again.  Usually by the third time I resort to manual intervention.

Manually installing the transport/mount service

First up you need to get a hold of the installation files.  These are located on your Veeam Backup and Replication server under the path C:\Program Files\Veeam\Backup and Replication\Packages – You will find there the individual packages for each service that Veeam provides (mount, transport, tape, etc).  Depending on what services your proxy is providing you may need an number of these.  Since my server was acting as a repository as well as a proxy I simply needed the transport and mount server packages (VeeamTransport.msi and VeeamMountService.msi respectively).  Also, don’t forget that Veeam relies heavily on the .NET framework so you must keep that updated as well – you can find the redistributable installation package for that within the packages folder along side the others (NDP452-KB2901907-x86-x64-AllOS-ENU.exe)

Installation is just like any other install – your typical Next->Next->Done type of scenario.  Once you have ran the required packages head back to Veeam Backup and Replication.  If you are still on the component update screen a ‘refresh’ should update the status of the packages – if not, a rescan of your server within the Backup Infrastructure section is required.

So that’s that – as it turns out the issue on why my installation was failing during the automated process was due to lack of disk space, but nonetheless this is good information to have – If you are looking for more information in regards the the new features within Veeam Backup and Replication v9 I’ve done a post here – feel free to check it out!.

 

Quickfix – Mass editing Veeam VM Attribute settings with PowerShell

Hi – I’m Mike – you may remember me from such blogs as, oh this one last year!  I know, it’s been a while since I’ve written anything, moreso, published anything.  It’s been a hectic couple of months and I’ve got a ton of drafts sitting just waiting to be touched up so stay tuned – I promise to be a bit more visible in the next coming weeks!  Anyways I called this a quick fix so I guess I should get to the point…

VeeamVMAttributeThere is a setting within the notification options inside Veeam Backup and Replication that allows you to write some details to a processed VMs annotations section.  Now I’ve never had a use for this…until now.  I was reporting on a wide variety of things in regards to specific VMs, and having the last successful backup was one of the things that I wanted to report on.  This was within an asp.net application, and dropping to PowerShell and loading up the Veeam cmdlets was something I just didn’t feel like coding within the application.  Also, accessing the Veeam REST API was out of the question seeing as these VMs were being processed by Veeam Standard lisenses – REST is only available within Veeam’s Enterprise Plus offering.  Since I was already connected the vSphere API to gather a bunch of information such as CPU and Memory usage I thought by having Veeam write the last successful backup to the Notes field within vSphere would be a good alternative, as I could just gather that info with the same API calls I was making to vSphere.

One problem presented itself – I had roughly 50 different jobs within the Veeam Backup and Replication deployment that needed to be updated in order to make it happen.  Naturally I looked to automation for the solution – and I found it with PowerShell.  In order to completely enable this feature, and turn off the ‘append’ option you have to touch two different spots within Veeam PowerShell; one, the Set-VBRJobAdvancedViOptions (to enable the actual writing to VM Attributes) and another by setting the VmNotesAppend flag within the ViSourceOptions object.

Its a simple script, and honestly you probably don’t care, or didn’t read any of the introduction stuff above – I know how things work, you came here for the script – well, here its…

$jobs = Get-VBRJob

foreach ($job in $jobs)
{
  $job | Set-VBRJobAdvancedViOptions -SetResultsToVmAttribute $True
  $joboptions = $job.GetOptions()
  $joboptions.ViSourceOptions.VmNotesAppend = $False 
  $job.SetOptions($joboptions)
}

There you have it!  A simple script that peels through all the VBR Jobs within a console, enables the writing to the VM Attribute, and disables the append flag.  An easy script but useful none the less 🙂