Monthly Archives: September 2011
I've seen this happen numerous times on a number of our VMware vCenter installations. For the first few months(sometimes days) vCenter Server is very snappy and responsive, when moving from page to page things come up instantly and moving between different statistics and metrics on performance graphs is veryquick. Then, as time goes by, things begin to drag down a bit, tabs start to take a little bit longer to load when moving between them, performance graphs can throw exceptions and things just generally slow down.
Of course there are many issues that could be causing this to happen, but the most common that I have found is a fairly simple one, a neglected VMware vCenter DB that is full of fragmentation and in need of a little TLC. I don't want to go into what exactly database/index fragmentation is, but a good read if you have the time (and interest) is here. Also, VMware has released KB Article 1003990 as well, which covers off fragmentation within the vCenter DB.
And then there is also my explanation….
When I'm looking at tuning my vCenter DB, the tables that I find myself always defragmenting are as follows..
As you may guess, these tables hold historical and rolled up performance statistics and data. Since vCenter is always collecting data (depending of course on your Intervals and durations) these tables are constantly being updated (New stats coming in, old stats going out). Just as in file level defragmentation, the large number of writes, updates, and deletes causes some tables to become heavily fragmented.
I'm not going to go through defragmenting all of these as it is the same steps for each table/index. For this purpose I'll just go through VPX_HIST_STAT3. First off, to see the fragmentation inside a table just run the following command in SQL Server Management StudioUSE [vcenter database name] GO
You can retrieve the names of the indexes either from KB Article 1003990 or by expanding the Indexes folder in SSMS. Essentially this command translates to..Use VIM_VCDB
dbcc showcontig ('VPX_HIST_STAT3','PK_VPX_HIST_STAT3')
In my example this returns the following stats.
To summarize these results for a VI Admin, the lines that you should really be looking at is 'Scan Density' and 'Logical Scan Fragmentation'. In short, you want Scan Density to be as close to 100% as possible, and Logical Scan Fragmentation to be as close to 0% as possible. To defrag the indexes in this table we use the following command….dbcc indexdefrag('[databasename]’,'[tablename]’,'[indexname]')
so, after filling in the values we get…dbcc indexdefrag ('VIM_VCDB','VPX_HIST_STAT3','PK_VPX_HIST_STAT3')
As you can see we now have a lower Logical Scan Fragmentation and a higher Scan Density. This is the expected result we want from the defragmentation. Just repeat this step on all of the indexes and tables you want to defragment and you should be enjoying a much snappier, more responsive vCenter Server in no time. Keep in mind, some smaller indexes and tables will always be fragmented and not much can be done to correct that issue. Personally, I concentrate on the 8 tables listed above. Keep these tables/indexes happy and you are well on your way….
Over the last few weeks I’ve really had the opportunity to put Veeam Backup and Replication to the test. I’ve completed multiple restores (using Instant VM recovery, Entire VM and by picking just single vmdks) and it has saved us like a champ many times. Today we ran into a situation where another restore was needed, however this time, since some of our replicas were now on some decent storage I decided to give the failover a shot.
Failing over to a replica is pretty straight-forward and in most cases shouldn’t even need any supporting documentation. Where I ran into some questions was when the failover was complete. In essence, I had my VM up and running within minutes, end users were happy as they were able to access the application, but the VM in question was now running on some lower tier storage and was experiencing a little bit more latency than it normally would running on our production storage. I found documentation in the B&R user guide on how to ‘Undo’ a failover, but didn’t see any on how to commit a failover.
So, below is the process I followed to commit the failover and get this VM back onto production storage and get my replications jobs functioning again on the ‘new’ replica VM.
1. Remove the Veeam Backup Running Snapshot – Veeam creates a snapshot before powering on the replicated VM. By reverting to this snapshot you are able to get the VM back to its state before you performed the failover, thus giving you the option to undo. In this case, I had no plans of undoing this failover as I needed this replica for production use so I deleted it using the VI Client.
2. Storage vMotion your replicated VM to your production storage. – This does a couple of things. First off, if you are like me and have some lowered tier storage to host your replicas, it gets the VM to your faster spindles. Secondly, your replicated VM files are stored inside their own directory in a folder called VeeamBackup on your datastore. The storage vMotion will also do the ‘cleanup’ so to speak of your folder/file names, aligning them with your VM name.
3. Remove original VM out of the replica job – Simple task really, if you only have one vm / job, just delete your replica job. If you have multiple VMs / job, open up the properties of your job and remove the respective VM from the list.
4. Undo the failover – Don’t worry, this won’t power down your replicated VM as Veeam now has no idea where it went (since the svMotion). One note however, be sure to check the ‘Force Lock Removal’ box as shown below in order to release the locks on the replicated VM.
The following steps are completely optional…
5. Remove original VM replicas. – Find your VM under the replicas section, right click and delete from disk. This will clean up all of the replica restore points from the original VM. Depending on storage space available, I sometimes do not perform this step. By leaving the restore points there you it does give you the option to do more restores from the original VM if need be. Your choice really…
6. Add the newly recovered VM back into your replica job. – Easy one right! Add the new VM that you have just failed over to back into the replication job in order to protect it from whatever caused you to fail over in the first place.
These are just the steps that I take to commit my replicas. As always, I’m sure there are many different ways to do everything but this is the easiest way I’ve found to accomplish this task. Certainly, if I’m missing something or going way off the hook on something please leave feedback and let me know… Also, from what I’ve heard Veeam Backup & Replication v6 will have the options to permanently failover to a replica built right in!
When I first learned that I was going to be attending VMworld this year, I was very excited. Excited to be at the centre of all the new technology, excited to participate in the technical deep-dive sessions, excited to get my hands dirty with some of the new features of vSphere 5 in the hands on labs, excited to experience all that is Vegas for the first time. Just excited as a whole to be part of this large conference. However, upon my return from the fabulous Las Vegas and the even more fabulous VMworld I’ve realized that my initial expectations and views on the conference had somewhat changed.
Sure, I’m grateful to have experienced everything that I have listed above, but to me, as with most other conferences that I’ve attended, the most memorable, beneficial experiences have came from the conversations in the hallways. At VMworld, this point was elevated to new heights. After signing up for the conference I began to pay a little more attention to my twitter feeds, my rss feeds, and just began listening to the many podcasts available centred around VMware. This proved to be the best preparation that I could have possibly done for VMworld. I truly found out why VMware and its’ partners have always bragged and boasted about the community surrounding them.
What made my experience at my first VMworld memorable all began and ended with the community. The community that has assembled itself around virtualization, and more specifically VMware virtualization is one of the strongest and tightest that I’ve ever seen. It truly is a welcoming, helpful, and educational experience being involved with this community. I can go on all day about how much I appreciate the support of all the VMware community members, but I’d rather just stick to a few specific examples of how they helped me get the most of my first (and hopefully many) appearances at VMworld.
Example 1 – The Social Aspect
We all know that there are many parties hosted by many different community members and vendors at VMworld. When I first started to find out through social media and blog posts how to get into some of these parties I thought there was no way I’d get invites, being just a small customer and virtually unknown to the hosts. That was until I read a post by Christopher Kusek (@cxi). Christopher basically mentioned that if you haven’t received a ticket, whether the were tickets left or not, to just send an email….you never know. So, I sent an email off to Christopher regarding his CXIparty and immediately got a response with an invite. I was shocked, even more shocked when our email conversation went back and forth many times in the weeks leading up to VMworld. To me, Christopher is the perfect example of a truly dedicated community member. I barely know him, have only met him once, and he has gone out of his way to ensure that my experience (and even my wifes) at VMworld was truly epic.
Example 2 – Those humble rock-stars
No, I’m not talking about The Killers, I’m talking about all of those famous bloggers and tweeps out there that are consistently day after day rolling out blog posts and dedicating their personal time to speak at various events and VMUGs. After following some of these people on twitter and reading their blogs for the last few years I made it a point to go up and introduce myself and thank them for their commitment, time, and great content that they have been producing. Each and everyone one of them (and there are way too many to name) seemed to be somewhat humbled and appreciative to accept complements from me. They thanked me for saying hello, and it was really nice to put a real face to all of those twitter handles! I felt a bit awkward in a way, but quickly realized that I was not alone, as there were many other attendees there doing the same thing. Most of the ideas and conversations throughout the community are started by one of the many virtualization bloggers and that’s what makes them the rockstars!
Example 3 – John Troyer
Does this guy ever quit? It seemed that everywhere that I went the VMware social media evangelist was there…and with more energy than the rest of the room combined. The man arranged many vMeetups in the community lounge (including the one that caused me to start this blog), delivers a content rich podcast every Wednesday and really knows how to make splash at a party 🙂 Not to mention I’m sure he had many, many hours of planning leading up to VMworld. Kudos to John (@jtroyer) for all his hard work over the years.
These are only three examples of many that make the VMware community what it is today. I urge anyone who hasn’t invested some time in the community to do so, as it seems to be growing stronger and stronger every day. I would also like to thank the community for being so welcoming, it really did make my first VMworld experience one to remember.
Okay, here we are. A blog!?! I’ve been so busy reading everyone elses’ blogs that I never even thought of writing my own. That is until now!
So, a bit about @mwpreston, well, I’m from the great white north! A fairly rural city in Ontario, Canada eh? I’m a husband, father, and a huge Montreal Canadiens fan. I’ve been employeed in and around the I.T. industry for the last 12 years or so, with the last 6 focusing in public education. Throughout those 12 years I’ve mainly focussed on Applications development, Database administration, and as of late, virtualization. I have a passion for all things surrounding I.T (especially when they work) and I hope that will come through in the content you find here.
Speaking of content, I’m really not sure what you will find on this space yet! I hope to fill it up with my trials and tribulations of anything I have to deal with and resolve that I think may help others. More specifically I’m sure you will see most posts with a strong focus on virtualization with a little bit of application and database development thrown in there. I will try to keep my personal life out this space (mainly because I don’t think you would care to know anything about it) but I can’t help myself sometimes, and tend to go on the odd rant!
So you may find a hodge podge of almost anything here but my main goal will be to at the very least provide consistency. It may be hard during the long cold months that we here in Canada call hockey season, but I’m sure I could squeeze some sort of update in during the 2nd intermission. I wouldn’t count on anything valid being in the those 2nd intermission posts as they may turn out to be negative and ranty in nature, depending on the score of the Habs game.
I don’t claim to be an expert in anything, so if you find any error in what I am saying, or for that matter if you just plain feel like leaving a comment please do. They are always welcome. Also, I know for certain that I’m not the greatest writer in the world so hopefully people can bear with me as I produce content without proper punctuation and grammar. You are just going to have to deal with it…hey, if you want, comment on how horrible it is…I don’t care!
So, in the great words of David Davis (@davidmdavis) “With that said, let’s get started…”