Monthly Archives: September 2011

Giving vCenter a kick in the rearend!! – Defragmenting your vCenter Database

I've seen this happen numerous times on a number of our VMware vCenter installations.  For the first few months(sometimes days)  vCenter Server is very snappy and responsive, when moving from page to page things come up instantly and moving between different statistics and metrics on performance graphs is veryquick.  Then, as time goes by, things begin to drag down a bit, tabs start to take a little bit longer to load when moving between them, performance graphs can throw exceptions and things just generally slow down.

Of course there are many issues that could be causing this to happen, but the most common that I have found is a fairly simple one, a neglected VMware vCenter DB that is full of fragmentation and in need of a little TLC.  I don't want to go into what exactly database/index fragmentation is, but a good read if you have the time (and interest) is here.  Also, VMware has released KB Article 1003990 as well, which covers off fragmentation within the vCenter DB.

And then there is also my explanation….

When I'm looking at tuning my vCenter DB, the tables that I find myself always defragmenting are as follows..


As you may guess, these tables hold historical and rolled up performance statistics and data.  Since vCenter is always  collecting data (depending of course on your Intervals and durations) these tables are constantly being updated (New stats coming in, old stats going out).  Just as in file level defragmentation, the large number of writes, updates, and deletes causes some tables to become heavily fragmented.

I'm not going to go through defragmenting all of these as it is the same steps for each table/index.  For this purpose I'll just go through VPX_HIST_STAT3.  First off, to see the fragmentation inside a table just run the following command in SQL Server Management Studio

USE [vcenter database name] GO
dbcc showcontig([tablename],[indexname])

You can retrieve the names of the indexes either from  KB Article 1003990 or by expanding the Indexes folder in SSMS.  Essentially this command translates to..

dbcc showcontig ('VPX_HIST_STAT3','PK_VPX_HIST_STAT3')

In my example this returns the following stats.

To summarize these results for a VI Admin, the lines that you should really be looking at is 'Scan Density' and 'Logical Scan Fragmentation'.  In short, you want Scan Density to be as close to 100% as possible, and Logical Scan Fragmentation to be as close to 0% as possible.  To defrag the indexes in this table we use the following command….

dbcc indexdefrag('[databasename]’,'[tablename]’,'[indexname]')

so, after filling in the values we get…

dbcc indexdefrag ('VIM_VCDB','VPX_HIST_STAT3','PK_VPX_HIST_STAT3')
which then returns….

As you can see we now have a lower Logical Scan Fragmentation and a higher Scan Density.  This is the expected result we want from the defragmentation.  Just repeat this step on all of the indexes and tables you want to defragment and you should be enjoying a much snappier, more responsive vCenter Server in no time.  Keep in mind, some smaller indexes and tables will always be fragmented and not much can be done to correct that issue.  Personally, I concentrate on the 8 tables listed above.  Keep these tables/indexes happy and you are well on your way….

Committing A Veeam Replica Failover

Over the last few weeks I’ve really had the opportunity to put Veeam Backup and Replication to the test.  I’ve completed multiple restores (using Instant VM recovery, Entire VM and by picking just single vmdks) and it has saved us like a champ many times.  Today we ran into a situation where another restore was needed, however this time, since some of our replicas were now on some decent storage I decided to give the failover a shot.

Failing over to a replica is pretty straight-forward and in most cases shouldn’t even need any supporting documentation.   Where I ran into some questions was when the failover was complete.  In essence, I had my VM up and running within minutes, end users were happy as they were able to access the application, but the VM in question was now running on some lower tier storage and was experiencing a little bit more latency than it normally would running on our production storage.  I found documentation in the B&R user guide on how to ‘Undo’ a failover, but didn’t see any on how to commit a failover.

So, below is the process I followed to commit the failover and get this VM back onto production storage and get my replications jobs functioning again on the ‘new’ replica VM.

1.  Remove the Veeam Backup Running Snapshot – Veeam creates a snapshot before powering on the replicated VM.  By reverting to this snapshot you are able to get the VM back to its state before you performed the failover, thus giving you the option to undo.  In this case, I had no plans of undoing this failover as I needed this replica for production use so I deleted it using the VI Client.

2.  Storage vMotion your replicated VM to your production storage.  – This does a couple of things.  First off, if you are like me and have some lowered tier storage to host your replicas, it gets the VM to your faster spindles.  Secondly, your replicated VM files are stored inside their own directory in a folder called VeeamBackup on your datastore.  The storage vMotion will also do the ‘cleanup’ so to speak of your folder/file names, aligning them with your VM name.

3.  Remove original VM out of the replica job – Simple task really, if you only have one vm / job, just delete your replica job.  If you have multiple VMs / job, open up the properties of your job and remove the respective VM from the list.

4.  Undo the failover – Don’t worry, this won’t power down your replicated VM as Veeam now has no idea where it went (since the svMotion).  One note however, be sure to check the ‘Force Lock Removal’ box as shown below in order to release the locks on the replicated VM.

The following steps are completely optional…

5.  Remove original VM replicas. – Find your VM under the replicas section, right click and delete from disk.  This will clean up all of the replica restore points from the original VM.  Depending on storage space available, I sometimes do not perform this step.  By leaving the restore points there you it does give you the option to do more restores from the original VM if need be.  Your choice really…

6.  Add the newly recovered VM back into your replica job. – Easy one right!  Add the new VM that you have just failed over to back into the replication job in order to protect it from whatever caused you to fail over in the first place.

These are just the steps that I take to commit my replicas.  As always, I’m sure there are many different ways to do everything but this is the easiest way I’ve found to accomplish this task.  Certainly, if I’m missing something or going way off the hook on something please leave feedback and let me know…  Also, from what I’ve heard Veeam Backup & Replication v6 will have the options to permanently failover to a replica built right in!

Tech Support & Social Media

In the olden days 🙂  if you wanted to voice your opinion on a product or service, or if you had issues the process usually involved picking up a phone and calling the company and sitting on hold for a while.  Those days have most certainly changed.  With the introduction of social media outlets such as facebook and twitter people are posting their views, opinions, and experiences with products and services for the world to see.  And when that world includes users, customers, prospective customers and partners the companies and corporations manufacturing these products and services cannot afford to not participate.  And this stands true with VMware and its ecosystem partners as well.  Day after day I see the presence of the virtualization community growing on Twitter.  I just want to take you through a scenario that I’ve been through recently and outline how the participation of some companies and community members in social media outlets took an absolute horrible event and turned it into a successful and great learning experience for myself

I don’t want to go to into detail about everything so I will just start off by saying we had two of our production storage controllers  fail.  Not a big deal, unless of course you only have two production storage controllers 🙂  Needless to say I initiated a support call with our storage vendor to get the ball rolling.  As we were waiting for the dispatched field tech to arrive I decided I would send a tweet out to see if anyone has been in this situation before and had a few recommendations on how to approach the errors we were getting.  Within minutes I had a response from @hpstorageguy Calvin Zito.  After a few tweets back and forth with Calvin I started to notice some emails coming into my inbox from him.  Roughly 45 minutes after our storage went down, I had an inbox full of information from Calvin and some of his field techs outlining what had happened, why it happened and possible resolutions.  Although I wasn’t going to try and tackle this issue myself it was nice to better understand what happened and have somewhat of an idea of what our field tech was saying when they arrived.  I was lucky enough to meet Calvin at VMworld this year and thank him in person.  If you don’t follow @hpstorageguy on twitter I would definitely recommend doing so.  He’s and active blogger, runs a podcast, and constantly contributes to the community.

Since the storage failure, we noticed that quite a few of our vms’ disks had become corrupt.  I repaired what I could but some disks just proved to be non-repairable.  So I started the restore process on a handful of vms using our backup solution @veeam.  I tweeted a few times about what I was doing, and again, immediately began receiving responses from @veeam and @rickvanover.  Although I didn’t need any support from veeam (mainly due to the fact that their products are pretty intuitive and easy to use), they certainly offered up their help in the case that I needed it and that in itself was reassuring.  Whether you are a Veeam customer or not I would most definitely recommend following @veeam and @rickvanover.  Again, Rick blogs, produces quality podcasts, and is very active on twitter.

So, with our production environment once again running as it should, I continued on with the process of working on the project I was in the middle of before all this disaster hit, which ironically enough was an offsite backup and replica solution.  I was in the middle of deploying an EMC VNXe that we had purchased earlier as the storage for this solution.  I found the VNXe very easy to install and configure with the exception of some of the networking components.   I had been following @henriwithani for a month or so now and had seen that he had been putting together a very informative blog series based on the VNXe.  His post however had been based off the 3300 and we had the 3100 and there were a few differences that I had noticed.  I decided to send him a quick tweet just to see if he knew what my issue was.  The result, about 15 tweets later was a fully functional redundant EMC VNXe.  Big thanks to @henriwithani for all his help and be sure to check out his blog here.

I felt like sharing this experience as I found it amazing to get the support that I did through social media.  With the exception of the first support call regarding the storage controllers (which was pretty serious) all the issues were solved without picking up a telephone and at the comfort of my desk.  I found out through the last few years that there is a plethora of information out there on twitter, you just need to learn how to follow it.

VMworld 2011 and the community that surrounds it!

When I first learned that I was going to be attending VMworld this year,  I was very excited.  Excited to be at the centre of all the new technology, excited to participate in the technical deep-dive sessions, excited to get my hands dirty with some of the new features of vSphere 5 in the hands on labs, excited to experience all that is Vegas for the first time.  Just excited as a whole to be part of this large conference.  However, upon my return from the fabulous Las Vegas and the even more fabulous VMworld I’ve realized that my initial expectations and views on the conference had somewhat changed.

Sure, I’m grateful to have experienced everything that I have listed above, but to me, as with most other conferences that I’ve attended, the most memorable, beneficial experiences have came from the conversations in the hallways.   At VMworld, this point was elevated to new heights.  After signing up for the conference I began to pay a little more attention to my twitter feeds, my rss feeds, and just began listening to the many podcasts available centred around VMware.  This proved to be the best preparation that I could have possibly done for VMworld.  I truly found out why VMware and its’ partners have always bragged and boasted about the community surrounding them.

What made my experience at my first VMworld memorable all began and ended with the community.  The community that has assembled itself around virtualization, and more specifically VMware virtualization is one of the strongest and tightest that I’ve ever seen.  It truly is a welcoming, helpful, and educational experience being involved with this community.  I can go on all day about how much I appreciate the support of all the VMware community members, but I’d rather just stick to a few specific examples of how they helped me get the most of my first (and hopefully many) appearances at VMworld.

Example 1 – The Social Aspect

We all know that there are many parties hosted by many different community members and vendors at VMworld.  When I first started to find out through social media and blog posts how to get into some of these parties I thought there was no way I’d get invites, being just a small customer and virtually unknown to the hosts.  That was until I read a post by Christopher Kusek (@cxi).  Christopher basically mentioned that if you haven’t received a ticket, whether the were tickets left or not, to just send an email….you never know.  So, I sent an email off to Christopher regarding his CXIparty and immediately got a response with an invite.  I was shocked, even more shocked when our email conversation went back and forth many times in the weeks leading up to VMworld.  To me, Christopher is the perfect example of a truly dedicated community member.   I barely know him, have only met him once, and he has gone out of his way to ensure that my experience (and even my wifes) at VMworld was truly epic.

Example 2 – Those humble rock-stars

No, I’m not talking about The Killers, I’m talking about all of those famous bloggers and tweeps out there that are consistently day after day rolling out blog posts and dedicating their personal time to speak at various events and VMUGs.   After following some of these people on twitter and reading their blogs for the last few years I made it a point to go up and introduce myself and thank them for their commitment, time, and great content that they have been producing.  Each and everyone one of them (and there are way too many to name) seemed to be somewhat humbled and appreciative to accept complements from me.  They thanked me for saying hello, and it was really nice to put a real face to all of those twitter handles!  I felt a bit awkward in a way, but quickly realized that I was not alone, as there were many other attendees there doing the same thing.  Most of the ideas and conversations throughout the community are started by one of the many virtualization bloggers and that’s what makes them the rockstars!

Example 3 – John Troyer

Does this guy ever quit?  It seemed that everywhere that I went the VMware social media evangelist was there…and with more energy than the rest of the room combined.  The man arranged many vMeetups in the community lounge (including the one that caused me to start this blog), delivers a content rich podcast every Wednesday and really knows how to make  splash at a party 🙂  Not to mention I’m sure he had many, many hours of planning leading up to VMworld.   Kudos to John (@jtroyer)  for all his hard work over the years.

These are only three examples of many that make the VMware community what it is today.  I urge anyone who hasn’t invested some time in the community to do so, as it seems to be growing stronger and stronger every day.  I would also like to thank the community for being so welcoming, it really did make my first VMworld experience one to remember.

Hey, Who’s the new guy?

Okay, here we are.  A blog!?!  I’ve been so busy reading everyone elses’ blogs that I never even thought of writing my own.  That is until now!

So, a bit about @mwpreston, well, I’m from the great white north!  A fairly rural city in Ontario, Canada eh?  I’m a husband, father, and a huge Montreal Canadiens fan.  I’ve been employeed in and around the I.T. industry for the last 12 years or so, with the last 6 focusing in public education.  Throughout those 12 years I’ve mainly focussed on Applications development, Database administration, and as of late, virtualization.  I have a passion for all things surrounding I.T (especially when they work) and I hope that will come through in the content you find here.

Speaking of content, I’m really not sure what you will find on this space yet!  I hope to fill it up with my trials and tribulations of anything I have to deal with and resolve that I think may help others.  More specifically I’m sure you will see most posts with a strong focus on virtualization with a little bit of application and database development thrown in there.  I will try to keep my personal life out this space (mainly because I don’t think you would care to know anything about it) but I can’t help myself sometimes, and tend to go on the odd rant!

So you may find a hodge podge of almost anything here but my main goal will be to at the very least provide consistency.  It may be hard during the long cold months that we here in Canada call hockey season, but I’m sure I could squeeze some sort of update in during the 2nd intermission.  I wouldn’t count on anything valid being in the those 2nd intermission posts as they may turn out to be negative and ranty in nature, depending on the score of the Habs game.

I don’t claim to be an expert in anything, so if you find any error in what I am saying, or for that matter if you just plain feel like leaving a comment please do.  They are always welcome.  Also, I know for certain that I’m not the greatest writer in the world so hopefully people can bear with me as I produce content without proper punctuation and grammar.  You are just going to have to deal with it…hey, if you want, comment on how horrible it is…I don’t care!

So, in the great words of David Davis (@davidmdavis) “With that said, let’s get started…”