VCSA 6.0 prompting for a manual fsck

One of my VCSA deployments, the only one running 6.0 experienced a switch failure and in result a network outage of roughly 5 minutes the other day.  Not a big deal, but unfortunately this was a very “cost effective” solution and the switch that hosted the production network also hosted the VLANs carrying all of the NFS traffic to the datastores the VCSA resided on as well!  In short, VCSA done got grumpy – after fixing the issues with the switch I ended up at the screen shown below…


Not an overly complicated error – just stating that we need to run a file system check on the /dev/mapper/log_vg-log volume manually.  In the past, say with 5.5 I’d just drop to a bash shell and do so – however the default appliance shell in the 6.0 version of VCSA presents a few challenges in doing the same thing.  First off, if I went ahead and gave the root password to the VCSA I was presented with the default menu – the same menu you would receive if you ssh’d to the box under normal circumstances – that said, in the maintenance mode, the shell.set and shell.enable commands don’t work.  So in order to get to a point where we can actually execute fsck we need to do a couple of things…

Grub to bash

So the first thing we need to do is get our VCSA booting into a bash prompt.  To do so, hit CTRL+D at the presented screen and get the box to reboot.  When the boot loader appears we will need to hit the space bar or up/down keys to stop the auto boot process.  Once stopped we can selected ‘p’ to unlock the menu and enter the root password for the box.  We then want to select “e” to edit our boot sequence –  highlight the second line, the one that displays the kernel parameters and select “e” once again.  At the end of that line we will want to append “init=/bin/bash” as shown below – this will boot our system into a bash shell.  Once done, hit “enter” to save and “b” to boot.



After the system has booted you should now be sitting at a bash prompt.  On a normal day we would simply run our fsck command here however the file system we are looking to check is still not mounted at this point.  I tried numerous commands and options to try and get it mounted but came up short.  That said running the following command and rebooting our vCenter will switch the login shell for root back to the ‘normal bash’ and allow us to continue

chsh -s /bin/bash root

Once the command has been run and the server rebooted we will be brought back to the same error prompting us to enter the root password.  Go ahead and do that.  This time we will be brought directly to a bash prompt with log_vg-log being available to us!  So, without further ado go ahead and run the following command to complete the file system check.

fsck /dev/mapper/log_vg-log

More than likely you will get numerous prompts asking you whether or not to fix any errors that occur.  Use your discretion here, however I didn’t have much of a choice and needed to say ‘Yes’ to all.  After it’s done give the VCSA another reboot and everything should come back up normally (at least it did for me).  Hopefully this helps push someone in the right direction if they are experiencing similar issues :)

Manually updating the Veeam Proxy Transport and Mount services

With the release of v9 hitting the Internets on Tuesday I’ve been a very busy man upgrading various Veeam Consoles, Proxies, and Repositories.  With nearly 70 different locations to look after you can imagine the amount of proxies and repositories I have, both on and off site all requiring their respective Veeam services to be upgraded.  Mix that together with a few slower WAN connections and I can almost bet that the automated component update that Veeam ships with will naturally fail on a couple servers.

Want the tl;dr version?
Packages are in c:\Program Files\Veeam\Backup and Replication\Packages – copy them to your failed server and install :)


Failed to upgrade host components

When this happens to me usually I get some sort of error message like the following


For the most part, re-running the automated component update will fix the issue, but there are times when it fails, again, and again, and again.  Usually by the third time I resort to manual intervention.

Manually installing the transport/mount service

First up you need to get a hold of the installation files.  These are located on your Veeam Backup and Replication server under the path C:\Program Files\Veeam\Backup and Replication\Packages – You will find there the individual packages for each service that Veeam provides (mount, transport, tape, etc).  Depending on what services your proxy is providing you may need an number of these.  Since my server was acting as a repository as well as a proxy I simply needed the transport and mount server packages (VeeamTransport.msi and VeeamMountService.msi respectively).  Also, don’t forget that Veeam relies heavily on the .NET framework so you must keep that updated as well – you can find the redistributable installation package for that within the packages folder along side the others (NDP452-KB2901907-x86-x64-AllOS-ENU.exe)

Installation is just like any other install – your typical Next->Next->Done type of scenario.  Once you have ran the required packages head back to Veeam Backup and Replication.  If you are still on the component update screen a ‘refresh’ should update the status of the packages – if not, a rescan of your server within the Backup Infrastructure section is required.

So that’s that – as it turns out the issue on why my installation was failing during the automated process was due to lack of disk space, but nonetheless this is good information to have – If you are looking for more information in regards the the new features within Veeam Backup and Replication v9 I’ve done a post here – feel free to check it out!.


Quickfix – Mass editing Veeam VM Attribute settings with PowerShell

Hi – I’m Mike – you may remember me from such blogs as, oh this one last year!  I know, it’s been a while since I’ve written anything, moreso, published anything.  It’s been a hectic couple of months and I’ve got a ton of drafts sitting just waiting to be touched up so stay tuned – I promise to be a bit more visible in the next coming weeks!  Anyways I called this a quick fix so I guess I should get to the point…

VeeamVMAttributeThere is a setting within the notification options inside Veeam Backup and Replication that allows you to write some details to a processed VMs annotations section.  Now I’ve never had a use for this…until now.  I was reporting on a wide variety of things in regards to specific VMs, and having the last successful backup was one of the things that I wanted to report on.  This was within an application, and dropping to PowerShell and loading up the Veeam cmdlets was something I just didn’t feel like coding within the application.  Also, accessing the Veeam REST API was out of the question seeing as these VMs were being processed by Veeam Standard lisenses – REST is only available within Veeam’s Enterprise Plus offering.  Since I was already connected the vSphere API to gather a bunch of information such as CPU and Memory usage I thought by having Veeam write the last successful backup to the Notes field within vSphere would be a good alternative, as I could just gather that info with the same API calls I was making to vSphere.

One problem presented itself – I had roughly 50 different jobs within the Veeam Backup and Replication deployment that needed to be updated in order to make it happen.  Naturally I looked to automation for the solution – and I found it with PowerShell.  In order to completely enable this feature, and turn off the ‘append’ option you have to touch two different spots within Veeam PowerShell; one, the Set-VBRJobAdvancedViOptions (to enable the actual writing to VM Attributes) and another by setting the VmNotesAppend flag within the ViSourceOptions object.

Its a simple script, and honestly you probably don’t care, or didn’t read any of the introduction stuff above – I know how things work, you came here for the script – well, here its…

$jobs = Get-VBRJob

foreach ($job in $jobs)
  $job | Set-VBRJobAdvancedViOptions -SetResultsToVmAttribute $True
  $joboptions = $job.GetOptions()
  $joboptions.ViSourceOptions.VmNotesAppend = $False 

There you have it!  A simple script that peels through all the VBR Jobs within a console, enables the writing to the VM Attribute, and disables the append flag.  An easy script but useful none the less :)

Friday Shorts – JSON, Veeam, Embedded Host Client and more…

I think you’re really beautiful and I feel really warm when I’m around you and my tongue swells up. – Buddy the Elf

JSON will be the death of me

I don’t do a lot of JSON parsing but when I do I tend to get just a little frustrated.  Honestly I almost always try and retrieve API responses as XML if the option is there, but sometimes it isn’t and sorting through JSON is the only alternative!  I’m a newbie to this stuff and still am learning quite a deal about interacting with APIs using various methods – one of the many reason I follow Scott Lowe’s blog regularly.  In a post this week he introduced us to a lightweight command line tool called jq, which looks like it could most definitely help me with my JSON woes – next time I find myself staring down at a page full of squigglies (not sure of the official term for  {}) I’ll give it a go… Thanks Scott!

Embedded Host Client is awesome x 5

I haven’t had a lot of time to spend with the Embedded Host Client fling – a replacement that will allow us to do away with the giant c# installable that we have all grown to love for individual host management.  I did get to use the EHC early on, and it was not the greatest experience – but as time went on and it was released via VMware Flings it is now a lot better, in fact people are loving it!  Here’s a great article outlining 5 reasons why the Embedded Host Client is awesome!

VVOLs 4 DBs!

VVOLs is still a little unknown when it comes to adoption – I haven’t seen a lot of blogs or information out there about widespread adoption!  I do know that there is general excitement out there and definitely benefits to be had by utilizing them, however they are still a little “ripe” in IT years!  That said there’s an interesting series on the VMware blogs going on right now outlining the benefits they can provide to databases – you can check out Part 1 and Part 2 if you like…

The Expert Guide to VMware Disaster Recovery and Data Protection

Shameless plug for myself on this one – A few months ago I began work on a guide for Veeam centering around how to provide Data Protection within a VMware environment – and the result is finally live!  Note – you are going to hit a reg-wall when trying to view this – it’s the nature of the world we live in, but if you do end up checking it out I’d love to know what you think – good or bad, but mostly good!

Dumping SharePoint Integrated SSRS reports to pdf using c#

Anytime I’m working on any sort of coding project Google is really my savior – there is absolutely no way I can know how all of the different libraries and functions available inside of all the .net components, nor do I really care to.  Sometimes I’ll get lucky and find exactly what I’m looking for but in reality it’s a fury of Google searches opened up in multiple tabs that end up getting pieced together into what I call “Production Code”.    Anytime I have to piece together solutions from various sites I try and post the finished product here – mainly so I don’t have to go and find it all again the next time I do it but also I’m sure it helps the odd stranger during his Google-Fu – so with that said, and with #vDM30in30 looming over me here’s a solution to help with connecting to a SharePoint integrated reporting services instance, passing parameters to a report, and exporting the report to pdf through code-behind.

I chose to do this as a c# Windows Forms app but you could technically try and reproduce the code with some slight changes if you were to use asp.  Either way we only need two controls on the form to make it work; a ReportViewer and a Button – go ahead and drag those onto the form.


So for the setup of the ReportViewer control there are only a few properties we need to setup in order to get our connection to the SharePoint SSRS instance which are listed below

  • ProcessingMode – Set this to Remote.  This simply states that the report we will be running is located remotely
  • Then under ServerReport we need to specify a couple of options
    • ReportServerUrl – This is the URL to the SSRS Report Server.  Usually, in SharePoint integrated mode it will be the same as your site URL, with ‘_vti_bin/reportserver/’ tagged on the end.  IE, https://mysharepoint.url/sitename/_vti_bin/reportserver/
    • ReportPath – This is the path to the report you would like to process, relative to your ReportServerUrl.  So if you browse to your Report Server and keep track of the directories you click on to get to the desired rdl – this is what goes into this property – IE Reports/MyReports/ReportName.rdl

And some code…

First up we are going to be accessing a few methods and objects outside of the normal c# includes.  In order to gain access to these you will need to import the following references into your project and declare them at the top of your cs file – Use nuget it’s much easier :)

using Microsoft.Reporting.WinForms.Internal.Soap.ReportingServices2005.Execution;
using Microsoft.Reporting.WinForms;
using System.Web.Services.Protocols;
using System.IO;

And the code for the main click event on our button….

private void Button1_Click(object sender, EventArgs e)
  string FilePath = "C:\\Desired\\Path\\";
  // define and add parameteres
  Microsoft.Reporting.WinForms.ReportParameter[] reportParameterCollection = new Microsoft.Reporting.WinForms.ReportParameter[3];
  reportParameterCollection[0] = new Microsoft.Reporting.WinForms.ReportParameter();
  reportParameterCollection[0].Name = "Parameter1";
  reportParameterCollection[1] = new Microsoft.Reporting.WinForms.ReportParameter();
  reportParameterCollection[1].Name = "Parameter2";
  reportParameterCollection[2] = new Microsoft.Reporting.WinForms.ReportParameter();
  reportParameterCollection[2].Name = "Parameter3";
  //render report
  string[] streamids;
  string mimeType, encoding, extension;
  string deviceInf = "8.5in11in";
  Microsoft.Reporting.WinForms.Warning[] warnings;
  byte[] bytes = reportViewer1.ServerReport.Render("PDF", deviceInf, out mimeType, out encoding, out extension, out streamids, out warnings);
  //write file and refresh reportviewer control
  string completefilename = FilePath +  "Report.pdf";
  FileStream fs = new FileStream(completefilename, FileMode.Create, FileAccess.Write);
  fs.Write(bytes, 0, bytes.Length);

So lets break this down into manageable chunks of code…

Line 3 simply defines the path to the folder where we would like to export the report

Lines 5 through 15 define and set three parameters for our report.  Obviously if your has more/less you can adjust accordingly.  Line 16 simply calls the setParameters function of our ReportViewer component and passes our newly generated parameters.

Lines 18 through 22 define and set a few variables that we need setup in order to render our report but the real magic happens on line 23 where we call the render function of our ReportViewer and store the PDF result in an array of bytes.

Lines 26 through 29 take our bytes array and utilize a FileStream object in order to write the information to a pdf file.

I’ve included a refresh command on Line 30 even though it isn’t needed for just rendering one report.  I wanted to place this in here just in case someone may be copying the code to achieve the same thing I was trying to achieve.  Basically I had all of this code within a loop – I was looping through various strings and running my target report multiple times, each time with different parameters – in order to do this you need to call the RefreshReport() method at the end of every run in order to allow the object to completely initialize.  I even went as far as to set the visibility of the ReportViewer object false as I simply just wanted to click a button and be left with a folder full of pdfs.  So, not needed for 1 run, but if you place the code into a loop its needed.

So in essence this is the code I’ve used.  It certainly is a time saver – we used to have someone manually changing parameters and dumping these reports to PDF which as you can imagine can take quite a long time if you need to do it a few hundred times – now, it’s simply an application that they run…

Happy coding.

Ravello on my wrist – Pebble in the cloud

You can probably get the gist as to what this post might be about by the title but it does leave a little to the imagination.  For those who hate waiting for the point go ahead an watch this small video…

Before I get right into what I’ve done let me first provide a little background information as to why I’ve done this aside from just looking for “something geeky to do”.

Ravello-Systems-LogoFirst up I’ve pretty much let everyone know how much I heart Ravello Systems .  Not to go too deep but as I build up labs and environments for this blog and for other interests I really like to break things.  Why?  That’s how I learn best, breaking things, fixing them, then writing them down.  The problem is I seem to always be rebuilding or fixing before I can move onto my next project.  Ravello solves that issue for me – with Ravello I’m able to keep multiple blueprints of completely configured vSphere labs (different versions, different hardware configs) in the cloud.  When I’m feeling geeky I can simply deploy one of these as an application to either Google or Amazon and away I go.  If I break it to the point of no return it’s no biggie, I can simply redeploy!  Point and case it’s a time-saver for me!

Secondly I love to write code – it’s an odd passion of mine but it’s something I actually went to school for and never 100% pursued.  Meaning I love to write code….casually!  I couldn’t imagine dedicating my whole career to it, but having the knowledge of how to do it casually sure has helped me with almost every position I’ve held.

pebbleThirdly a little while I ago I purchased a Pebble watch.  I’m still not sure why I wanted a smartwatch but I knew if I had one I’d want it to be somewhat “open” and Pebble met those needs.  Using a service called CloudPebble and by turning on the development mode on the iPhone app I’m able to deploy custom applications to my Pebble – so that was a big seller when I was looking at watches – oh, and the fact that it’s only like $100 helps as well…

So on to the problem – I mentioned I love Ravello and have multiple applications setup within the service.  The applications are great, however it takes a good amount of time after powering one on before you are able to start using it.  Those vSphere services need time to initialize and boot.  My usual routine involves me logging into Ravello and powering on what I might need for the night before I leave work.  That way the initialization can happen during my commute, supper with my family, and bedtime routines and is ready to go when I am.  There are times though when I get half way home and realize I forgot to power on my labs, or I’m not near a computer and can’t be bothered to use the small iPhone screen.

There’s an app for that!

PowerOnRavelloFor these reasons I decided to try and figure out the Ravello APIs and the Pebble SDK and see if it was possible to create a small application to simply login into Ravello, select an existing application, and power it on!  It sounds simple enough but took a lot of trial and error – I had no clue what I was doing but in the end I was left with the solution below – and it works so I guess you could call it a success.


There’s a few pieces that need to fall into place before any of this will work.  First up you wiill need a CloudPebble account.  CloudPebble is a development environment that allows us to write applications for use on the Pebble watch in either JavaScript or C.  You can use an existing Pebble account to log into CloudPebble or simply setup a new account – either way you need one and it’s free!

devmodeSecondly you will need to enable developer connections within the Pebble app on your phone.  This is easlily done by selecting ‘Developer’ within the main menu and sliding the switcher over.  Honestly, it’s a phone app I’m sure you can figure it out.

Thirdly lets go ahead and setup a project within CloudPebble You can do this by simply importing mine, or manually by giving your new project a name and select PebbleJS as your Project Type.  Once created you should be at a screen similar to that shown below…


As you can see we have one source file (app.js).  This is the only source file we will need for this project.  If you imported my project you are done for now, but if you created a new project manually this file will be full of a bunch of example code on how to perform various functions and respond to different events within the Pebble interface – we won’t need any of this so go ahead and delete all the code within the file, but not the file itself.  We will replace it with all of this syntax –  explained in the next section.

The code

If you simply just want all the code to go through on your own go head and get that here.  For the rest of us I’ll try and explain the different blocks of code below…

// import required libraries
var UI = require('ui');
var ajax = require('ajax');
var Vector2 = require('vector2');
var Vibe = require('ui/vibe');

Lines 1 through 5 simply deal with importing the libraries we will be working with – UI will give us access to the Pebble UI, Ajax is what we will use for the Ravello API calls, Vector2 for use with positioning on items on the watch, and Vibe is simply so we can access the vibration features of the watch.

// setup authentication information
var encodedLogin = "mybiglongencodedstring";
var expirationTimeInSeconds = 600; // timeout for app

Lines 8 and 9 set up a couple of variables for the application.  First up, encodedLogin represents a base64 encoded string of the username and password you use to login to Ravello, with a “:” between them.  You can grab this by heading to and grabbing the encoded string using UTF-8 as the output – just don’t forget to place the : between (ie. I encoded “”).  Copy the result and place assign it to the encodedLogin variable on Line 8

Line 9 deals with our expiration time – When we power on an application within Ravello we need to specify an auto power off parameter which states how long we want before the application powers itself down.  You don’t want to use up all those valuable CPU hours right?  The variable defined on line 9 is matched to that, however in seconds so get your calculator out and come up with a number.

// main window
var splashWindow = new UI.Window();
var text = new UI.Text({
position: new Vector2(0,0),
size: new Vector2(144,168),
text: 'Logging into Ravello Sytems, please wait...',
// Add to splashWindow and show

Lines 11 through 25 simply define the first splash window we will see in the application.  Kind of a message to show the user as we are making the API calls and gathering the application lists.  You can start to see some of the Pebble object functions and parameters here…

As we move into ajax calls starting on Line 28 we can start to see the URLs and API calls to Ravello and how they are formatted when using PebbleJS.  From here each API call that is sent to Ravello is nested within the previous – this was the only way I could get this to work.   You can go ahead and read the docs on the ajax function here – I still don’t really completely understand the values being returned but hey, it works!

Anyways, back to the task at hand – As shown below lines 28-30 makes are login request, passing basic authorization and our encodedLogin variable within the header.  After parsing the response on Line 34 we display yet another splash screen (Lines 35-44) with a success.

//login to Ravello
ajax({ url: '',method: 'post',
headers: {Authorization: "Basic " + encodedlogin,
Accept: "application/json"}
function(data,status,obj) {
// success into Ravello
var contents = JSON.parse(data);
var text2 = new UI.Text({
position: new Vector2(0,0),
size: new Vector2(144,168),
text: 'Hello ' + + ', you are now logged in! - Fetching applications, please wait...',

Another API call, this one to gather our application lists takes place on lines 46 and 47.  From there lines 51 through 74 build a menu to display the application listing, hide our previous screen, and display our newly formed menu.

ajax({ url: '',method: 'get',
headers: {Accept: "application/json"}
// success application list
var apps = JSON.parse(data); 
var count = Object.keys(apps).length;
var menuItems = [];
var appname;
var appid;
for(var i = 0; i < count; i++) {
appname = apps[i].name;
appid = apps[i].id;
// Construct Application menu to show to user
var resultsMenu = new UI.Menu({
sections: [{
title: 'My Applications',
items: menuItems
// Show the Menu, hide the splash;

At this point we are waiting on user interaction – the user needs to select the application they want powered on.  Line 76 defines that exact event listener, triggered once the user hits the select button.

// Add an action for SELECT
resultsMenu.on('select', function(e) {
console.log('Item number ' + e.itemIndex + ' was pressed!');
// this is where magic happens and we translate which item was pressed into turning on applications
var detailCard = new UI.Card
{title: "Starting Lab", subtitle: e.item.title }
detailCard.body('Setting lab power off time to ' + expirationTimeInSeconds.toString() + ' seconds...');
var ExpURL = ''+e.item.subtitle+'/setExpiration';
// set expiration time for selected application
var expbody = { "expirationFromNowSeconds": + expirationTimeInSeconds };
url: ExpURL,type: "json",method: "post",headers: { Accept: "application/json" }, data: expbody
// success setting expiration time
detailCard.body('Setting lab power off time to ' + expirationTimeInSeconds.toString() + ' seconds...'+ 'DONE!\nPowering on lab...');
var StartURL = ''+e.item.subtitle+'/start';
url: StartURL,type: "json",method:"post",headers: {Accept: "application/json"}

Once an application is selected lines 78 through 84 display some status messages as to what is happening, and beginning on line 89 we start the API calls to power on the application.  First (Line 92) sets the expiration time for the selected application.  Then, line 102 sends the actual Power Up command to Ravello.

// success starting application
console.log("Success on start:" + status);
detailCard.body('Setting lab power off time to ' + expirationTimeInSeconds.toString() + ' seconds...'+ 'DONE!\nPowering on lab...' + 'DONE!\nLab Powered On' );

Lines 108 and 109 simply display some success messages to the user and send a short vibrate command to the Pebble watch.

I’ve done my best to explain the code – it’s probably not the cleanest or best way to do all this but guess what?  I can power on a Ravello application from my watch so that’s all that matters…  Please feel free to steal all the code if you want it – or here is the complete CloudPebble project if you are in a hurry and just want to skip the copy/paste.  I’d love any feedback anyone may have on this.  For now this is where the project sits but I’d love to expand it further and integrate with more of the Ravello APIs available.  At the moment I’m happy with powering on my Ravello labs from my wrist!