Tag Archives: vSphere 5.1

VMware launches new delta version of VCP5-DCV exam

VMware LogoToday I received a message from VMware Education Services introducing a new way for current VCP holders to refresh or re-certify before their VCP expires.  Currently as it stands, anyone holding a VCP certification prior to March 10, 2013 has only until March 10, 2015 to re-certify using one of the following methods.

  • Take the most current VCP exam in any of the available tracks (Datacenter Virtualization, Cloud and Desktop – not sure if Network Virtualization qualifies for this or not).  No matter which track you held your VCP in, all will be refreshed with another two years.
  • Take an advanced level exam, meaning the VCAP DCA or VCAP DCD.  Not only will you advance to the next level, you will refresh your VCP expiration as well.

Prior to today, these were your options.  Now however all you VCP holders have a third option, so long as you are currently hold the VCP5-DCV status.

What is a delta exam?

This is something new to VMware certifications.  Basically, this exam is based only on the differences between vSphere 5.0/5.1 and the vSphere 5.5 exams.  Also, instead of your normal 135 questions the delta exam will only have 65.  The biggest difference is how the exam is delivered – you won’t need to drive to a testing center for this one, it is being offered online through Pearson Vue – and I’m assuming this will be a similar fashion to that of the VCA delivery.  Another noticeable difference is price – this one, coming in at $120 USD instead of the normal $220 USD.

Is it worth it?

ScreamingMan-300x225This is something I can’t answer for you – you will have to go through the scenarios in your head.  Currently I have an expiry date of January 2016 for my VCP5 and honestly I’d rather sit a new version of the VCAP then do the VCP again.  That said, can I expect a VCAP6-DCA to be available by Jan 2016?  I have no idea!  Do I want to risk the chance of losing my VCP due to no new VCAP exam coming out or possibly failing the VCAP when it does come out?  It’s all a giant kerfuffle in my head right now!  One note, the email I received said it was only available to those who need to renew their VCP before March 10, 2015.  As noted above, mine was extended to Jan 2016 due the completion of my VCAP in January of this year.  That said, I went through the process of being authorized for this delta exam and had no issues getting into the portion of the Pearson Vue site which allows me to schedule it.  So, try for yourself I guess!

Time’s a wastin!

Oh yah, better hurry and make your mind up.  This delta exam will only be available until November 30th, 2014!  So you have just less than a couple of months to figure out what you are going to do!  Honestly, this whole re-certification process just confuses and puts me in a bad mood Smile  Nonetheless, though I’d share the news!  Oh, I tried to use the VMUG Advantage VCP discount code – didn’t work!

Troubleshooting vSphere Storage eBook giveaway!

2062EN_mockupcover_normalA few weeks ago I released a post in regards to my finishing of the Troubleshooting vSphere Storage book.  This has been a lot of work and I’ve had a lot of help from the community in getting this book completed.

Writing a book is not a simple task.  It involves research, lab time, and focus.  And then there is the editing, both grammatically and technically – which can be even more work than the writing itself! 

I tried to gear this book towards the vSphere Admin.  The “jack of all trades” system administrator.  Hopefully readers will find the knowledge and how to within the book to solve common storage issues that tend to spring up within a vSphere environment.  The book is broken into three main subjects of focus; troubleshooting connectivity, troubleshooting contention and troubleshooting capacity.  If you’d like to purchase the book you can do so by visiting the landing page on Packt’s website and following the various channels.

Tis the season for winning

That said – Tis the season right?  The season for giving!  That’s why I’m happy to announce that I have three eBook versions of Troubleshooting vSphere Storage to giveaway over the next week or so.  A little light reading to enjoy over the xmas holidays!

As you can see by the little widget below, I’ve opted to use PunchTab to gather the entries for this contest.  It’s hectic trying to follow Twitter hashtags and what not so I thought “Hey, why not use a CaaS (contest as a service) solution.  Sorry for all the PunchTab branding but this is what you get when you use a free service 🙂

So how do you enter?  It’s easy, leave a comment, send a tweet and follow me!  That will get you three entries!  Just make sure you do it through the widget below.  I’ll have PunchTab pick three random winners at the end of the contest (end of Sunday, December 15th) and announce the winners shortly thereafter.  Good luck and let me know what you think of the book!

Tuning Linux (Debian) in a vSphere VM – Part 3 – /dev/random

tuxcloudSo here we are – Part 3 and the final part of the Linux series.  The title of this part is /usr/bin/random because well the content will be indeed quite random.  I couldn't think of a way to classify the content of this post into a single category!  So get ready for a hodge podge of fixes, modifications and configurations that have nothing in common and no similarities whatsoever.  I apologize up front for the flow (or lack thereof) of this post but hopefully someone will find something useful in it.

What time is it?

Almost every OS is very dependent in having accurate time and use different hardware and software techniques to do so.  When something is virtualized this adds yet another layer in between the OS and the hardware and creates some challenges as it pertains to timekeeping.  I’m not going to go through a time keeping lesson – there’s a great VMware whitepaper here that goes very deep into the topic.  Instead I'll just try to summarize what I've learned over the years as it relates to Linux and timekeeping.  Depending on the Linux distribution and kernel version you are running you may need to add some boot parameters to either the grub or LILO menu in order to disable the High Precision Event Timer (HPET) and set the proper clock managers.  Most new releases of Linux distributions (within the last couple of years) don’t require any changes, but for instance, if you are running Debian 4.x you would need to append “divider=10 clocksource=acpi_pm” to your kernel boot line.  For a full list of available options have a look at KB1006427 from VMware.

The I/O Schedule

ESXi has its' own special sauce as it pertains to scheduling disk I/O – and so does Linux.  Linux has some different I/O schedulers built into the OS, such as NOOP (noop), Completely Fair Queuing (cfq), Anticipatory (anticipatory), and Deadline (deadline).  By default, as of kernel 2.6, most Linux distributions use CFQ as their I/O scheduler of choice – not really a problem in the physical world but as a guest OS can cause some performance degradation.  As stated earlier, ESXi has its' own I/O scheduling, so does it really make sense to schedule I/O at the guest OS level, and then at the hypervisor level?  Probably not!  That's why there is a VMware KB article that states to switch your I/O scheduler to noop or deadline.  Honesty I would switch to noop as it does nothing to optimize and disk I/O, which would allow the hypervisor to do its thing.  Here's how!

You can change the scheduler during runtime by echoing to the proper disk, IE for sda we would use

echo noop > /sys/block/sda/queue/scheduler

However, to permanently switch you need to add an elevator switch to your grub kernel entry in grub or your Linux entry in grub2.  The above will reset back to cfq on reboot.  To permanently do this your kernel entry in the grub menu should look similar to the following (menu.lst for Grub and grub.cfg for Grub2).  Below is Grub2.

linux   /boot/vmlinuz-2.6.32-5-686 root=UUID=b62a38cf-8917-484a-9a96-d5a74beb8d59 ro  quiet elevator=noop

Copy the floppy!

In Part 2 we went over how to completely get rid of the floppy drive.  Part of those instructions included blacklisting the floppy module inside of /etc/modprobe.d/ – well guess what?  There are a slew of other modules that are loaded by default that you will probably never use when running a virtualized instance of Linux.  Below is a list of modules that I often blacklist – Sure, there are the one off cases where you will need one or two of these modules loaded so just pick and chose for yourself…

floppy – Take a guess, yup you got it, the floppy drive 🙂

mptctl – This monitors your RAID configuration.  I don't normally RAID inside of my Linux guests so this is really not needed – also this will spam up your messages log quite a bit as well.

pcspkr, snd_pcm, snd_page_alloc, snd_timer, snd, soundcore – These all have to do with sound which I'm not even sure is possible.  Disable them!

coretemp – if you don't care how hot your vCPU's are running you are safe to disable this.  If you do care, then, well, I'm not sure what to tell you 🙂

parport, parport_pc – these have to do with your parrallel ports.  I've never used these and always blacklist them.

Virtual Consoles – Do you even use them?

If you ware wondering how can I even use them inside of vSphere – check out my post here.  If you don't use them, why leave them enabled?  Disabling them is pretty easy, just comment out the tty# lines in /etc/inittab – I always tend to leave the lines in the file and just place a '#' in front of the ones I don't need – below you can see an example of my initttab files.  As you can see I left one console activated.

1:2345:respawn:/sbin/getty 38400 tty1
2:23:respawn:/sbin/getty 38400 tty2
#3:23:respawn:/sbin/getty 38400 tty3
#4:23:respawn:/sbin/getty 38400 tty4
#5:23:respawn:/sbin/getty 38400 tty5
#6:23:respawn:/sbin/getty 38400 tty6

So that's really all I can think of that in terms of tips and tricks that I do when deploying a Linux guest on vSphere.  I'm sure there are many others.  Linux and vSphere both have a ton of advanced settings and are very flexible when it comes to tuning for performance so if you know of any that I've missed I'd love to hear about them, just leave them in the comments box below.  By no means am I a Linux expert and I'm always eager to learn 🙂  Thanks for reading!

Part 1 – Installation

Part 2 – Virtual Hardware

Part 3 – /usr/bin/random

Tuning Linux (Debian) in a vSphere VM – Part 2 – Virtual Hardware

tuxcloudAlright – so in Part 1 we covered LVM, partitioning and the basic installation of our VM.  Now its time to tackle a few key points as it pertains to virtual hardware.  Now there isn't a whole lot to cover here and honesty I'm just going to graze the surface on some of these topics.  You will find a bunch of links to some KB articles and white papers where you can go pretty deep on some of these topics so feel free to read them as well.  Again, leave all of your feedback and comments in the sections outlined below – it's a learning process for us all 🙂  So, let's get into it!

Use the virtualized hardware!

VMXNET3 right!  Hopefully by now you know of the advantages of using the VMXNET 3 adapter – most of those same advantages apply to Linux as well.  I’d definitely recommend using this.  Since one of the benefits of VMXNET is the RSS or Multiqueuing it makes sense to use it, however there are a few things to keep in mind when utilizing this with the VMXNET3 driver; Any modern Linux kernel will now have it’s own built in VMXNET3 driver/module and for the most part (I think by default) your VM will use it, even if you have VMware Tools installed. – and this is fine.  There are times however where you might want to use the driver which is included with VMware Tools.  If you are running an older Linux kernel then the version of VMXNET3 you have might not support RSS or Multiqueueing, therefore you might want to use the VMware version which does.  To do so you can run the following command to replace the kernel driver with the VMware provided driver from the VMware Tools install package.

./vmware-install.pl –clobber-kernel-modules=vmxnet3

Go head and check out KB 2020567 for a bit more information on this.   Just remember, newer kernel = do nothing, older kernel = run the command above.

Who wouldn't want to be paravirtual?

Hey sounds cool right!  I know I'd want to be paravirtual!  Well your virtual SCSI controller does too!  PVSCSI has been the recommended adapter of choice for a little while now and can provide you with a multitude of benefits such as higher throughput at a lower CPU cost!  That being said, you need to be sure you are on a supported OS list!  When it boils down to it, so long as you are running at least 2.6.33 of the Linux kernel and have the vmw_pvscsi driver you should be ok, but, for those that like to be officially supported, the list is here.

Manage that memory!

memoryIts always best practice to right-size your VMs.  Keeping an eye on what CPU and memory resources are being utilized, knowing your workloads and sizing it appropriately – a Linux VM is no different (Hey, we are VMs too you know).  There is one setting from within a Linux VM that can be changed and tweaked to help though – swappiness!  Yup I said it, swappiness!  As always, open source and Linux never fail to amuse me with their terms and names for things!  So swappiness is essentially an integer value that determines when to move memory pages from physical memory( which is really virtual now, just mapped to physical) to your local swap.  Sounds cool eh but there is a catch!  Linux will move pages to swap even if you have physical memory available and this is completely separated from the fact that vSphere will also do its' own memory swapping if need be.  To cut to the chase a high swappiness value indicates that processes are more likely to to be swapped, whereas a low value indicates the kernel will leave them be.  The default value I've seen the most is 60 – normally I like to change this to somewhere around 15-20.  This is done by executing the following command

Echo 15 > /proc/sys/vm/swappiness

This works great for the current running state but to persist across reboots you will need to add the "Vm.swappiness=15" somewhere inside of /etc/sysctl.conf

Useless Hardware – Get rid of it!

LinuxPerf-Part2-fd0errorDo you honestly ever plan on using that floppy disk?  Didn't think so – get rid of it!  This is a process I try and apply to all my VMs not just the Linux ones.  Unneeded hardware can tie up not only vSphere resources but your OS resources as well.  So in terms of floppy it's not just as simple as just removing it from the VMs settings – you will need to also disable it from the BIOS of the VM.  If not you will always see that crazy floppy error that flashes up when grub is booting (the one on the right).  And even further, remove and blacklist the floppy module and prevent it from even loading into the Linux OS.  If you perform an 'lsmod | grep floppy' you will see that even though you have removed it from the VM and disabled it in the BIOS, the module still loads.  To completely blacklist a module from loading simply add it to a blacklist file in /etc/modprobe.d/ – In the case of the floppy we can execute the following list of commands to remove all traces of that pesky disk!

Echo "blacklist floppy" | tee /etc/modprobe.d/blacklist-floppy.conf
Rmmod floppy
Update-initramfs -u

So, that's it for now!  As with anything you do in production be sure to test it first!  Try different swappiness values, keep an eye on your resource usage and adjust as needed!  If you have any other tips/tricks for a Linux VM please leave them in the comments section below!  I'd love to hear them!  Next up in this series we will look at a bunch of random things like timekeeping, the i/o scheduler, as well as some other modules that we probably don't require when running virtualized!  Stay tuned and thanks for reading!

Part 1 – Installation

Part 2 – Virtual Hardware

Part 3 – /usr/bin/random

VMware vCenter Log Insight – Make your logs make sense!

VMware LogoToday VMware has introduced the world to VMware vCenter Log Insight, labeling it as a "new automated log management and analytics product for the cloud era".  In my opinion this is a great next step for VMware's management portfolio and if integrated correctly, could really compliment the analytics and performance data crunched by VMware vCenter Operations.  

More than just syslog?

From what I have seen, YES!  Although the underlying technology utilizes syslog collectors/receivers to receive the data, the visualizations and dashboards by which that data is presented to the end user is really where the value resides.  On average an ESXi host will dump roughly 250MB of data per day.  That's 250MB of data, that you, the end-user will need to parse and correlate line by line to try and make some sense out it.  I know I only understand about 25% (if that) of what is spit out in some of those logs.  vCenter Log Insight takes this data and with what they call 'content packs', presents the user with a bunch of predefined dashboards of some of the most relevant data that you may be looking for, along with common links to KB articles if any.


Easy transition from monitoring to troubleshooting

Hopefully we have all seen the power of vCenter Operations; How it correlates and analyzes all that data to really help us drill down and find out where any current (or future) problems exist.  If the issues are not evident, or if we are still unure of what the problem still is, the next viable step would be to jump into our logs to see what information we can find there.  With integration between vCOPs and vCenter Log Insight hopefully this will make that transition from our monitoring solutions into our log analyzing solutions a whole lot easier.  Again, saving us time and helping us discover root causes that much quicker.


Even more for advanced users

For those that love to look at the raw log data (huh?!?!?!) you can do that as well.  A search type functionality, similar to that of Splunk is available as well.  Use this to parse and filter through all of your logs that vCenter Log Insight collects.  The main difference here is there is no need to learn any new "languages" to drill around in and query your data.  VMware seems to have really made a big effort to keep this product simple and easy to use, but powerful and extendable at the same time.  Also, the ability to generate alerts and send email notifications on a custom query is a very nice functionality to have.


More than just ESXi and vCenter

As mentioned above visualizations and presentations are provided by content packs.  These are easily exported and imported to and from vCenter Log Insight, in turn allowing third parties (including YOU) to easily develop, distribute and share.  So, hopefully, within time, we will see more than just ESXi and vCenter logs getting pumped into this.  On that note, we will probably see more than just VMware products being analyzed.  In my opinion the community will really need to take the lead on this one, and looking at past performance that the VMware community has, I'm sure they will!

So VMware says to expect to see some sort of GA in Q3 of this year, I'll let you guess the timeframe!  I hope to get a few more posts out about vCenter Log Insight as I delve more into the product but for now you can find some here, here and here.  Have a look for yourself and let me know if you think!

My First vCenter Orchestrator Workflow – Part 6 – Resources

booksOK!  Finally the end of this series!  Honestly without the existence of the following resources there is no chance in hell that I would have been able to even develop a simple workflow, let along start scripting and what not.  I was a couple weeks into vCO before I even learned that you can switch to ‘design’ mode 😉

So, if you are looking for some awesomeness on vCO, have a look at the following…

Official vCenter Orchestrator Documentation – Should really be your first go to reference for all that is vCO

Automating vSphere with VMware vCenter Orchestrator – Although official docs SHOULD be your first go to item – I find that this book written by Cody Bunch actually IS my first go to reference.

www.vcoteam.info – Great blog with a ton of script examples and whatnot!  Bookmark this…

www.vcoportal.de – Ditto to the above line, might better bookmark this on while you in your bookmarks…This is a fabulous blog!

VMware vCenter Orchestrator blog – the official blog from VMware centered around vCO.

VMware API and SDK documentation – this always helps when trying to determine what type of objects or properties any given function requires.

Good Ol’ Twitter – Follow @cody_bunch, @josh_atwell, @vCOteam, @technicalvalues – there are a ton more but these are the ones that I can think of off the top of my head – just search for #vCO and find someone to ask your question to – community seems to be always willing to help.

Thanks for reading and hopefully you can find some usefulness out of vCO as I did!  I encourage everyone to explore what it has to offer, which the same results, everytime!  Check out the full series below…

My first vCenter Orchestrator Workflow

My First vCenter Orchestrator Workflow – Part 5 – A little bit of SQL

orchestratepart5Thus far in this series of posts we have installed and configured vCenter Orchestrator as well as setup and utilized a couple of plugins; the vCenter plug-in and the PowerShell plug-in.  Before the series ends I wanted to go through one more plug-in.  The SQL plug-in.  The SQL plug-in is used for, well, you guessed it; running SQL queries against a database.  If you remember in Part 4 we setup a workflow that took two input parameters; a host and a location code.  The script then went on to create a new port group on a vSwitch named locationcode-VM_Network.  The logic in itself works fine and the workflow validates, the only problem I see with it is that the user needs to enter the ‘location code’ manually as an input.  Now I know you are all very careful and take your time when doing things, but I’m not, and more than likely after doing this 40 or 50 times you can count on me keying in that location code wrong 🙂 – Enter the SQL plugin.

So the setup is as follows; I have a database containing a table with the following columns; locationcode and deployed (boolean).  What I would like to do is have an input parameter that would allow me to select the location code from a list of non-deployed locations in a drop-down box (rather than manually enter this information in), and in turn pass the location code to my PowerShell script.  Once the script has ran I’d like to update the deployed column for that specific location, ensuring it isn’t displayed in the list again in turn ensuring I can’t deploy the same location twice.  Make sense?  I know it’s a lot to follow but the concepts of setting it up are pretty much the same no matter what you are looking to do.

Alright – enough background – Let’s get into it.  I’m not going to go through the installation of the SQL Plug-in – it’s exactly the same installation process as the Powershell plug-in which we covered in Part 4.  Similar to how we setup our Powershell host we need to setup our SQL server database connection inside of vCO.  To do this fire up your vCO client and navigate through the workflow tree to Library->SQL->Configuration and select the ‘Add a database’ workflow, right click and Start Workflow.  There are a few parameters as you can see that we need to pass to the workflow in order for it to successfully run.  First, give this connection a name and select your desired Database Type – in my case MS SQL.  Also here we need to pass a Connection URL.  Now if you don’t know anything about jdbc connection urls no fear, it’s not that difficult.  Simply enter it in the the following format…


So, for a SQL Server with named DC.lab.local running on the default port 1433 and  database named ServerDeploy you would use the following…



After clicking Next we are once again presented with our Shared/Per User session mode – again, I chose shared to use one simple set of credentials rather than a per user authentication.  When you are ready click ‘Submit’ to add your new database to vCO’s inventory.  One thing to note here is that this step is not necessary   If we wanted we could perform all of this at run time inside code, however for tutorial purposes and learning purposes it’s sometimes easier to do it this way.

Alright, nuff config!  It’s time now to get started on our task at hand; Querying the database for non-deployed servers and pushing the result as a drop-down box as an input parameter to our workflow that we created in Part 4.  First off there is a simple change we need to make to our existing workflow.  Here’s a tip – don’t feel like buffalo-ing your original workflow, simply right click on it and  select ‘Duplicate Workflow’ to copy it.  OK, first off we need a new attribute.  We originally have locationcode setup an input parameter of type string – and we still need this, however the result we get back from our database will be an array of strings.  So, on the General tab of your workflow create a new attribute called  databaseParameter of type SQL:Database and assign it the value of the Database we created earlier (this should allow you to browse the inventory to do so).  Once you are done that simply Save & Close and continue on with any validation errors.


So here comes the real magic!   We need to take that database attribute and pass it to a newly created action which will in turn spit out an array of string attributes (our locations in our database).   Again, you could do the following all in script embedded within your workflow, but you never know when you are going to need to reuse something so I’ve decided to create a new action to do so.    To create an new action be sure you are on ‘Design’ view from within your client and click on the actions tab in the left hand side menu.  Where you place your action doesn’t really matter, I chose to right click com.vmware.library.sql and create my action inside that module.  Just remember where you put it and what you named it:).


OK, you should now be in the Action editor.  This is where we are going to place the code that does all the querying of the database.  As I said earlier we need to pass this Action a database parameter and it will return an array of string.   The setup of input parameters and return types, along with all the other work we are going to do is all done on the scripting tab.  First off define your return type of an Array/string.  Secondly add an input parameter of type SQL:Database.  You can see all of this outlined in the capture below…


Hey!  That was easy!  Now it’s time for a little scripting.  vCO script is nothing but Javascript which calls the rich set of API’s that vSphere provides, along with the functions and properties of all the plug-ins provided in vCO.  The script I used is displayed below…

var resultingArray = new Array();
var query = "SELECT locationcode FROM Locations WHERE deployed = 0";
var resultingActiveRecords = databaseParameter.readCustomQuery(query);
for each (var resultRecord in resultingActiveRecords) {
var locationcode = resultRecord.getProperty("locationcode");
return resultingArray;

Simply paste this into the script pane of your action.  As you can see it’s a pretty simple script.  Creates and array, queries our database, pushes the locationcode column value of each record returned into that array and finally returns the array.  So – time to Save and Close and head back to our workflow.

So at this point we are left with 2 tasks.  The first being the creation of the drop-down box as an input.  To do this we will need to change the way our original input parameter, locationcode, is displayed.  This is done on the Presentation tab of our script editor.  Be sure you have selected locationcode in the Presentation tree that the top of the screen and select the icon to add a new property to it.  There are many several different properties listed but the one we are after is called Predefined list of elements.  Under the value field of our new property select the icon to assign an action.  In the popup find/filter for our action we created earlier, assign the actions parameter to our database parameter and click Apply.


There done right…well, kinda, you could run the workflow now and it would populate our input and it would go ahead and run the PowerCLI script and the VM Network would be created, however if you remember it was my intent to go back at the end of the workflow and update our database to ensure that the same locationcode could not be selected twice.  To do this we will need to drag a new Scriptable task element to run after we invoke our PowerCLI script.  Inside this scriptable task we will need to import a couple of local attributes in order to accomplish the sql we need to do, the locationcode, and the databaseParameter.


As for the script it’s going to look very similar to the syntax that we placed inside of our action with the exception of and executeCustomQuery function in place of the readCustomQuery and the actual query itself is different.  The following is what I used…

var query = "UPDATE locations SET deployed= 1 WHERE locationcode= '" + locationcode+ "'";

And now at last we are done!!  Go ahead and run your workflow, either from within the vCO client or the Web Client and you should now see a drop-down selection for the locationcode.  Once it’s selected once the script will run, create the VM Network, then update our database to ensure that the selected locationcode is not shown again in the drop-down.


So that my friends is the end of the line for me on creating my first vCenter Orchestrator Workflow but it is most definitely not the end of the line with vCO.  With access to a huge set of vSphere API’s along with all of the functionality and properties provided by its’ plugins, throw in some sweet integration with the vSphere Web Client I’m beginning to see my usage of vCO ramp up within my current role.  This series has hardly even grazed the surface in terms of vCO’s functionality so I urge you all to go out there and learn as much as you can about this product.  I’ll do one more post in the series and outline some of the resources that I’ve found along the creation of this workflow so expect to see that soon.

My first vCenter Orchestrator Workflow

Backing up your vCenter DB – all three of’em

three-fingersWait!  What!  3!?!?!  Yes, you read correctly!  While in the days of vCenter 5.0 and below we only had to worry about 1 database, the release of 5.1 has tripled that!   This is something I hadn’t even thought about until recently attending a vBrownbag put on by Justin King (@vcenterGuy).

So there’s the SQL database from vCenter right – ok – no big, I know about that one and am backing it up – KB article on how to do that.

Then there is the SSO database – no problem, I knew about that one as well since I had to create it when I first upgraded to 5.1.  Again it’s a MS SQL DB, doesn’t change that much –  which is easy enough to backup…

But then Justin started talking about the Inventory service – remember, that’s the third requirement you had to install when upgrading.  Well guess what?  It has a database too!  It’s not SQL at all – it’s sitting on your vCenter Server in xDB format.  My first thought was what is even in this database – I can’t browse it like I can the SQL databases (or I just don’t know how to).  What I can gather from the What’s New docs and VMworld presentations the Inventory database holds things such as a read cache of all the objects that are accessed within the vSphere Web Client and all of your tags and categories that are setup from vCenter.  I’m sure there’s more but this is all I can find.

However back to my main objective, how do I back this thing up?  A little digging around and I found this KB article on how to backup/restore your vCenter Inventory database.  Basically it’s as follows.

Backing up the inventory database (WINDOWS)

  1. Navigate to the Inventory scripts folder (c:\Program Files\VMware\Infrastructure\Inventory Service\scripts)
  2. Run the following
    • backup.bat -file backup_filename

Restoring the inventory database (WINDOWS)

  1. Navigate to the Inventory scripts folder (c:\Program Files\VMware\Infrastructure\Inventory Service\scripts)
  2. Run the following
    • restore -backup backup_filename

So utterly simple yet so not talked about 🙂  Wait – but what if I’m using the vCSA?  Am I out of luck?  Absolutely not!  Use the following…

Backing up the inventory database (Linux)

  1. Navigate to the Inventory scripts folder (/usr/lib/vmware-vpx/inventoryservice/scripts/)
  2. Run the following
    • ./backup.sh -file backup_filename

Restoring the inventory database (Linux)

  1. Navigate to the Inventory scripts folder (/usr/lib/vmware-vpx/inventoryservice/scripts/)
  2. Run the following
    • ./restore.sh -backup backup_filename

So there you go!  You can now sleep at night knowing you aren’t going to lose all of your hard work setting up those tags!  Moral of the story – Pay attention and participate in the vBrownBags – there is always some great information and learning to be had.

My First vCenter Orchestrator Workflow – Part 4 – A look at the Powershell Plug-in

Orchestrate-FacilitateAlright!  So far throughout this series of posts we have installed, configured and setup vCenter Orchestrator; as well as created and tested the integration between both vCO and the vSphere Web Client.  So what’s next in my venture?  Well if you can remember back to Part 1 of the series I set out on a quest to develop a workflow that would take an un-configured host and go through a series of tasks to create datastores, rename networks, setup hostname, etc…  Basically configure it the way I wanted.  Now there are built-in workflows in vCO to do this, however I had already done this work inside of a PowerShell script – so my main goal was to get vCO to execute that script.  And this leads us to part 4!

So first off as I said above there are a ton of great built in workflows inside of vCO that can do pretty much anything.  Some things however, such as the ability to run a Powershell script, are missing!  Thankfully there is a thriving developers community built around vCO and it’s SDKs allowing third party vendors to release their own sets of workflows in the form of a plug-in.  So as you probably guessed by now we will need to install a plugin to handle our Powershell script execution.

This plugin is called the vCenter Orchestrator Plug-in for Microsoft Windows Powershell and can be found over on VMware’s site.  You will need to download this file.  Once you have your hands on the vmoapp file we need to go into the vCenter Orchestrator Configuration page (https://IP_OF_vCO:8283/) to install it.  Select Plug-ins from the left hand menu, then browse down to the Install new plug-in section.  Upload your file and click ‘Upload and install’, accept the EULA and away you go… If you browse back to the main Plug-ins page you will notice that it states the plugin will be installed at the next server startup; so if you wish you can do this task manually by going to Startup Options -> Restart Service.


At this point we can fire up our vCO client.  Having a look in the Library of workflows we can now see a PowerShell folder.  Expanding this out further let’s us see some of the workflows that come packaged with the plug-in.  So we now have a few configuration steps we need to go through before executing scripts with this plug-in.

First off we need setup a PowerShell host.  A PowerShell host is simply a Windows machine with Powershell installed located somewhere in your environment that will handle the storing and executing of the scripts.  There is a great article on all of the steps here but I will go through explaining them all as well.   Keep in mind this might not be the most secure of ways to go about everything – but this is simply a lab so I’m not too worried – for those concerned w/ security implications check out the full lengthy thrilling winrm read on the msdn 🙂   

So, once you have chosen which machine you would like to be your Powershell host go ahead and drop to a command prompt on that machine and run the following list of commands.

  • To create a winrm listener and open any required firewall ports
  • winrm quickconfig
  • To enable kerberos authentication
  • winrm set winrm/config/service/auth @{Kerberos=”true”}
  • Allow transfer of unencrypted data – told you wasn’t the most secure 🙂
  • winrm set winrm/config/service @{AllowUnencrypted=”true”}
  • Up the max memory per shell – I needed to do this to get things working
  • winrm set winrm/config/winrs @{MaxMemoryPerShellMB=”2048″}

And there we are!  Things should be setup fine now on our host end to allow the connections from our vCO server.  Now we need to configure kerberos on the vCO end of things.  To do this you will need to console into your vCO server and get on the command line there – hope your comfortable with vi 🙂

Basically we need to create a file under /opt/vmo/jre/lib/security called krb5.conf – to do so simply

cd /opt/vmo/jre/lib/security
vi krb5.conf

This file will need to look as follows – obviously change to fit your environment – if you happen to be using an default configuration of the autolab then you are good just to copy the following…

default_realm = LAB.LOCAL
udp_preferences_limit = 1
kdc = dc.lab.local
default_domain = LAB.LOCAL

Once you are done simply hit :wq to save and exit.  In order to grant vCO permission to access our newly created krb5 file you will also need to change ownership of the file using the following command

chown vco.vco krb5.conf

And, in order for changes to take affect, restart the vCO service – you can do this from the configuration page->Startup Options or simply

service vcod restart

Whew!  Ok!  Getting somewhere….Now we need to add the host into our configuration of the Powershell plug-in.   I find the easiest way to do this is to simply run the ‘Add a Powershell Host’ workflow however you can develop the script/code on the fly to add the host during your own workflow if you want.   The way I have documented however is as follows…

Browse to the PowerShell plug-in configuration by expanding Library->PowerShell->Configuration in the workflows view of your vCO Client.  Right click ‘Add a PowerShell host’ and select Start Workflow.


Here we need to add a few key pieces of information.  Enter a name for your PowerShell host – this can be whatever you wish, then enter the HOSTNAME of the host you setup (kerberos will not work with IP)   and click Next.


The only item that needs to be changed on the Host Type screen is Authentication; be sure Kerberos is selected and click ‘Next’.


In the User Credentials screen you have a couple of options in regards to Session mode.  Again since this is a lab environment I selected Shared Session, however if you were looking at a more production type deployment you might want to explore the Per User settings which will execute commands under the credentials of the current user connected to vCO.  Enter in some credentials with access to vCO and click ‘Submit’.  The workflow should kick off and you can watch it flow through the different elements.  When done, hopefully you have a nice green check mark beside it!

Now that we have successfully installed the Powershell plug-in and added a Powershell host lets go ahead and create a script on our Powershell host to run in order to make sure everything works.  Although my end goal is to completely configure a host –  for simplicity sake I’ll use a small snippet of the complete script.  So I have placed the following script on my PowerShell host to simply add a new VM Network to vSwith0.  As you can see the script is pretty simple, takes two parameters, a host and a location code.  The location code is used to represent uniquness in my environment and is actually used multiple times in the complete script for the hostname, datastores, etc….

param ($hosttoconfigure, $locationcode)
add-pssnapin VMware.VimAutomation.Core
$username = "Host Username"
$password = "Host Password"
$success = Connect-VIServer $hosttoconfigure -username $username -Password $password
Get-VirtualSwitch -Name vSwitch0 | New-VirtualPortGroup -Name "$locationcode-VM_Network" -confirm:$false

So let’s get started on our workflow to call this script.  Again, create a new workflow and call it what you will.   Once you are in the workflow editor move to the Inputs tab.  Here we will get the information from the user that we require in the script.  So go head and add a couple of inputs to the workflow, one being hosttoconfigure (Type VC.HostSystem) and locationcode (Type String).


By selecting our host parameter and clicking the eye icon we can set the ‘Show in Inventory’ presentation property on the host input.  This will basically allow the user to browse through their vCenter to select the host when running the workflows, so might as well go ahead and do that as well.


The Powershell workflow that we will be using is called ‘Invoke and external script’ and requires 3 parameters; One, a Powershell Host which we have setup previously, the second is the script to execute, and the third is the arguments to pass to the script.


Once on the Schema tab go head and drag the ‘Invoke an external script’ workflow into yours.   When you see the parameter setup black bar pop up click ‘Setup’ and we can have a look at the parameters.


Through this screen we can basically promote or link the parameters from the Script workflow to our main workflow.  You can see that I’ve done a few things.  One, I’ve set the host parameter to a value (same as creating an attribute) and browsed through my inventory to select my Powershell host that we created previously.  Secondly I manually inputted the path to my Powershell script in the externalScript parameter.  Thirdly I selected to skip the arguments parameter at this time.  I’ve done this because I need to pass the hostname as well as the locationcode parameter from my main workflow to this parameter which we will do with a separate element called a Scriptable task.  Go ahead and click ‘Promote’.


So now we have our main input parameters setup (locationcode and hosttoconfigure) as well as our script setup and ready to go.  We just need to tackle that arguments parameter.  As I said we will do this with a separate Scriptable Task workflow so go ahead and drag that into your schema, just before your Invoke External Script.   A scriptable task is just that, kind of a blank canvas to do any sort of scripting (javascript) that you desire.


So let’s go ahead and click the pencil icon to edit our scriptable task.  There are few IN parameters that we are going to need inside of our script; The first is hosttoconfigure and the second is locationcode.


Now move on to the OUT parameters.  We will need to create a new out parameter to store our arguments, so click the link for a new parameter and create a new workflow attribute called scriptArgs of type string.

Now we need to move to the Scripting tab.  This is where we can concatenate a property of the hosttoconfigure parameter and the locationcode to populate our newly created scriptArgs variable.  Honestly it’s a one-liner.  I’m sure there is a way to create this workflow without even requiring this whole Scriptable task but this is the way I got to to work (again, my FIRST vCO workflow).  Copy/Paste the following into your scripting tab.

scriptArgs = hosttoconfigure.Name + " " + locationcode;

Ok, you can close out of your Scriptable task workflow now.  We are almost there.  All that we have left to do is map the External Script workflow paramater arguments to our Scriptable task elements parameter scriptArgs.  Thankfully this is an easy thing to do.


Once again go into your External Script element and click on the Visual Binding tab.  Here, as shown above, we simply need to drag the scriptArgs in attribute over to the arguments in parameter.


Guess what!  Your done!  If everything has been setup properly and I’ve done a good job at explaining what to do then you should be able to Save and Close your workflow and actually run it on a host (a test host of course).  The result – you should get a VM Network setup on vSwitch0 – oh and a pretty green checkmark beside your workflow 🙂

At this point I’m going to cut off this post since it’s getting pretty long.  That being said I’m not going to end the series!  I’m always looking at ways to better automate things and you know, having users type in that location code just isn’t good enough for me, a lot of room for error there; they could type it in wrong, do the same location twice, on and on and on…  And hey, I have a database already setup with my location codes in it and whether or not they have already been deployed.  So, stay tuned for Part 5 of my series where we will go through configuring the SQL plug-in and setting up our location input parameter to be populated by an SQL table – as well as update that same table on completion of our script.

My first vCenter Orchestrator Workflow

My First vCenter Orchestrator Workflow – Part 3 – Create a test workflow and see the awesomesauce

imgOrchestrateOk we are finally getting somewhere now with this series! So far we have downloaded, installed, and configured our vCO appliance. As well, we have registered our vCO instance with both vCenter and our SSO instance. This post will take us through verifying and testing our installation by creating and setting up a small workflow that we can then assign to some inventory objects inside of vCenter. This will allow us to perform that right-click magic in the Web Client and see for ourselves some of the awesome integration vCO and vCenter have.


With that said let’s get started on our test workflow. To do so you will need one of clients available to vCO. As you can see in the screenshot vCO provides you with a couple of options to start the client; one simply runs a java applet and the other let’s you install a client directly on your machine – whichever method you chose you end up sitting at the exact same login screen. Once you get there you should be able to simply input your vCO IP as the Host name and your proper credentials.

OK! So here we are, the vCO workflow development interface. Look a little foreign to you? No worries, it did to me as well. Actually, it still kind of does, this is just “My First” vCO workflow remember! So for the most part I’ve only really explored the workflow icon (shown below).  The others are simply Home (home), Scheduler (Allows scheduling of workflows), Policies ( Allows other plugins to trigger the kick off of workflows), and Inventory (You will see your vCenter inventory objects here). For now, we need the Workflows section active.vCOWorkflowWorkspace


So after you expand the Library tree you should see several folders by default. These folders contain all of the workflows that vCO is aware of. Obviously we can add more by creating our own or installing plug-ins (covered in Part 4). For now we will just create one of our own workflows. Before going to crazy I always like to create my own folder for things like this. I get a bit nit-picky about file management sometimes 🙂 To do so is pretty simple – Right Click Library -> Add New Folder and give it a name!



So now it’s time to create our workflow – Again Right-Click the folder you want to store it in and Select New Workflow and name it. At this point a new window, the workflow editor will open up. There is a lot of information and tabs here and to be honest I’m not going to explain what they all are. Partly because it would make this post much too long, but mostly because I don’t have a grasp on what they all are :). The main tab that we will be working in is called Schema. This is where we setup all of the flow elements inside our workflow. Once there you will see our main window is displaying the flow of everything, and the left hand side is showing all of the workflows that we can add to our workflow. The one I am interested in is ‘Reboot host’ As you can see you can browse through the tree of All Workflows to get to it or you can simply use the filter box at the top to search for specific workflows. Once you find it you simply add it to your workflow by dragging it into your flow window.


vCOPromoteInputsAfter doing so you will see a black bar along the top of the screen asking about the activities parameters. Something to know – workflows have inputs and outputs – these are pretty self explanatory. When we created the workflow initially it had inputs and output, and then when we dragged the Reboot Host workflow into ours it also had inputs and outputs. So what this is asking is if we would like to move the Reboot Hosts inputs into our workflows inputs. In this case this is something we want to do. So go ahead and click ‘Setup’. Now we see our ‘Promote Workflow Input Parameters’ window. This will allow us to do a few different things; First of all, we can promote or move our inputs from the reboot host workflow to our workflow, and secondly we can decide to assign default values to them at the same time. As you can see, the ‘Reboot Host’ workflow takes two inputs; one being the host and the other being the force switch. Leave host as it is but if you want you can set a default value of ‘Yes’ to force by selecting the value option and then the yes option. Then click ‘Promote’.

If you look at your Inputs section now you will see that only host is listed. This is because we require the user to tell us or select the host we wish to reboot. But wait, what about force?!?! Well, by assigning a default value to this input parameter we have actually converted this to an attributes. Attributes are available on the General tab. Anyways – this workflow is now complete! Go ahead and select ‘Save and Close’. You will notice that vCO will also handle a bit of version history for us! Awesome stuff!

Alright! We are done with vCO for now. Let’s flip back to our web client and assign this workflow to be executed on certain objects. Navigate through the web client to the vCenter Orchestrator section, then Select vCO Home and look in the Manage tab. This is where we will add our newly created workflow to our right-click context menu when associated with host. Click the + sign to add a new workflow. In the Available workflows browse through the tree until you find your workflow and click ‘Add’, then select the inventory objects to run it on – in our case, just a host and click ‘OK’


So, after all of this we are now ready for the awesomesauce! Get back into your Hosts and Clusters view and right-click on a host. Once the ‘All vCenter Orchestrator Actions’ menu loads have a look at what you see! Reboot Craziness!!! Your workflow!! If you see fit, go ahead and select it to kick off the workflow. You may be prompted for a token delegation – if so go ahead and click ‘Approve’ – I have no idea what this is 🙂 When your workflow window opens you can see that your host has already been pre-populated with our right-click. You can click Finish to start the workflow immediately or Next to setup a schedule of sorts. ***Warning – this will reboot your host *** – don’t know if that’s needed but whatever 🙂


So there you go! We’ve successfully created a test workflow in which we integrated into and executed from the vSphere Web Client. Pretty awesome stuff! I hope you are starting to see some of the benefits of vCO now! Next we will get back on track with our workflow and dig a bit deeper into creating workflows by installing the Powershell plugin and getting vCO to execute a script for us!

My first vCenter Orchestrator Workflow

My first vCenter Orchestrator Workflow – Part 2 – Installing vCenter Orchestrator

orchestrate2Welcome to the second part of My first vCenter Orchestrator Workflow series.  Now before we can really get into the functionality of vCenter Orchestrator we obviously need to go through the task of getting vCO installed and configured. VMware has provided a couple of different ways to do this; one being installing on a Windows box, normally on the same server as your vCenter Server. And the second being through the download of a virtual appliance. For the sake of this series I will go over the installation and configuration of the appliance as it seems to be the trickier of the two.  If you go with the windows installation it is simply included as an installable package on the vCenter Server media.

So first off you need to go and download the vCO appliance through your myvmware account.   vCO is bundled in with vCenter Server so if you have a copy of vCenter you are good to go with vCO (I think foundation only comes with the ability to run workflows and not edit)   From here you simply need to import the ovf appliance into your environment using the typical  File -> Deploy OVF Template and power it on.

Once the appliance has fully booted you will notice that it displays a slew of URLs and information.  Don’t worry!  By simply pointing your browser to the IP address of your vCO server you will see the same list – you don’t need to remember all of the ports.   There are a couple we need to pay close attention to here; Orchestrator Configuration (used to perform Orchestrater setup, plugins, ssl certs, etc – this is the main ui you will work in) and Appliance Configuration (the setup pages to manage your actual appliance settings, such as networking, ntp, admin passwords, etc.. )

For your documentation (more so mine) I’m going to list the default usernames and passwords for vCO below as it can be tricky trying to find some of them in the documentation.

Orchestrator Configuration – vmware/vmware – will prompt for new password when you first login
Appliance configuration – root/vmware – change password manually in the ‘Admin’ section.

So to get started let’s browse to Appliance Configuration – As described above here is where you can perform appliance related tasks such as setting up ntp, networking and change your root password. If you have used any VMware virtual appliances before you should be somewhat familiar with this interface.


Same look and feel as most VMware Virtual Appliances

Once you have your network settings configured we can continue on to the Orchestrator Configuration. This is the main interface for configuring all things Orchestrator. We have quite a bit of work to do here so lets get started. After logging in with the default credentials (vmware/vmware) you will be prompted to change the password so go ahead and do so.  Now we need to setup our Orchestrator appliance to listen on the proper IP address which set in the appliance configuration as well as work with the new SSO service that was shipped with vSphere 5.1. To do this we will first stop the Orchestrator server service. This can be done through the ‘Startup Options’ section on the left hand menu, then clicking ‘Stop Service’.



For the network listener, simply select the ‘Network’ menu on the left hand side, then select your IP Address in the associated dropdown box, then click ‘Apply Changes’ Easy peasy so far – well, now we are about to explore SSO and SSL Certificates – yikes!

In the same section (network) select the SSL Trust Manager tab. Here is where we list all of our imported ssl certificates. In order for SSO to work we need import a couple certs; one from our vCenter Server and the other from our SSO service. To do this there are a couple of URLs we need to know.

First enter in the following URL (https://IP_of_vCenter:443/ and click the import button. The certificate should be displayed in your browser. Simply click ‘Import’ once again to pull it into vCO.

Importing SSL Certificates

Importing SSL Certificates

Next we need to repeat the exact same steps excepting using https://IP_of_vCenter:7444/ as the URL.

Now that we have the required certificates we need to setup Orchestrator to point to SSO for authentication purposes which is done by, you guessed it, the Authentication section. Switch your authentication mode dropdown from LDAP Authentication (default) to SSO Authentication. Then you need to input your SSO Host – This will be the same as your vCenter unless you have explicitly installed SSO elsewhere in your environment. Also we need an admin username and password on your SSO host. Remember a way back when when you installed SSO and it was prompting you for admin usernames/passwords – I hope you remember these because we need them now. By default if you didn’t make to many changes your username might be [email protected] – unfortunately I can’t help you with your password 🙂 Once done select ‘Register Orchestrator’.

vCO will authenticate users using the SSO installation included in vSphere 5.1

vCO will authenticate users using the SSO installation included in vSphere 5.1

On the next page you can actually put more restrictions on who can and can’t be a vCO administrator. I left all defaults here and simply clicked ‘Accept Orchestrator Configuration’.  Basically you are configuring what users or groups can be vCO admins.

Alright, almost there – Now let’s go ahead and get the vCenter Server plugin installed and activated – we will go a little more in depth with plugins later so I’ll leave most of the details out that this point.  Simply select your Plug-ins menu and enter in a username and password in the boxes provided (SSO should be working now), select the checkbox next to the vCenter Plug-in and click ‘Apply Changes’ .

Installing the vCenter Plug-in

Installing the vCenter Plug-in

This might be a good time to give your vCO server a nice clean reboot or at least restart the services under Startup Options.  As you can see there are a few plug-ins stating they will perform the installation at the next server setup.  Either way you should see a vCenter Server menu option appear, once it’s there select it, then go to the ‘New vCenter Server Host’ tab. Fill in all of the information in regards to your vCenter Server

Adding a vCenter Server

Adding a vCenter Server

As you can see most the options here are pretty basic with the exception of the user strategies.  Basically ‘Share a unique session’ will result in orchestrator creating only one connection back to your vCenter Server.  This will certainly use less resources and may be ‘secure-enough’ for some small deployments.  The ‘Session per user’ option will actually execute the workflows under the user credentials of the user that is logged into Orchestrator.  This does have the ability to use a few more resources however provides a bit more secure and audit-able environment.  I used the ‘Share a unique’ session as I’m just running through my lab at this point.

OK, i told you we had a lot of work to do in here – but for the most part we have things configured now. If you want to double check you can log in to your vSphere Web Client, Select the vCenter Orchestrator item on the left hand navigation menu and you should be able to drill down and see your new vCO server registered.  If not and you have followed all these instructions to a tee with no break, you may want to give your vCO server a reboot and have a look. It should be there  🙂

vCO registered within the vSphere Web Client

vCO registered within the vSphere Web Client

At this point you can give yourself a huge pat on the back.  You’ve now succesfully setup and configured the vCO virtual appliance and have registered it within your vCenter Server.  In the next portion of this series I will discuss how to create a small test/sample workflow in order to test our integration between vCenter Orchestrator and the vSphere Web Client.

My first vCenter Orchestrator Workflow