Tag Archives: vCenter

VCSA 6.5 Migration deployment sizes limited!

Recently I finally bit the bullet and decided to bring the vCenter portion of a vSphere environment up to version 6.5.  Since the migration from a Windows based vCenter to the VCSA is now a supported path I thought it would also be a good time to migrate to the appliance as well.  So with that I ran through a few blogs I found in regards to the migration, checked out the vSphere Upgrade Guide and peeled through a number KB’s looking for gotchya’s.  With my knowledge in hand I headed into the migration.

At this point I had already migrated my external windows based PSC to version 6.5 and got started on the migration of the windows-based vCenter Server.  Following the wizard I was prompted for the typical SSO information along with where I would like to place the appliance.  The problem though came when I was prompted to select a deployment size for my new VCSA.  My only options available were Large and X-Large.  Might not be a big deal if in fact this environment required this amount of resources – Looking at the table below those deployment sizes are scoped to fit at a 1000 host and above mark.

DeploymentSize

Did this environment have 1000+ hosts and 10000+ VMs?  Absolutely not!  At its largest it contained maybe 70 hosts and a few hundred VMs running on them – a Small configuration at best, medium if you want to be conservative!  At first I thought maybe I was over provisioned in terms of resources on my current vCenter Server – but again, it only had 8 vCPU’s and 16GB of RAM.  With nothing out of the ordinary with vCenter itself I turned my attention to the database – and that’s where my attention stayed as it was currently sitting at a size of 200GB.  Honestly, this seemed super big to me and knowing that it had been through a number of upgrades over the years I figured I would make it my goal to shrink this down as small as possible before trying again!  TL;DR; version – The database was the culprit and I did end up with the “small” option –  but I did a number of things after a frenzy of Google’s and searches – all listed below…

WAIT!!!!  Don’t be that guy!  Make sure you have  solid backups and can restore if things here go sideways – engage VMware GSS if needed – don’t just “do what I do” 🙂

 

Reset the vpx provider

The vpx data provider basically supplies the object cache for vCenter – caching all inventory objects such as hosts, clusters, VMs, etc in order to provide that super-snappy response time in the vSphere Web Client 6.0 (Is this sarcasm?).  Anyways, resetting this essentially will reduce the size of our Inventory Database.  Now, the problem in versions prior to 5.5 Update 3 is that there was no way to reset individual data providers – in order to do one you had to do them all – and that meant losing all of your tags, storage profiles/policies, etc.  Thankfully, 5.5 U3 and 6.0 allows us to simply reset just vpx, leaving the rest of our environment in-tact.  In order to do so we must first get into the vSphere Inventory Managed Object Browser (MOB) and get the UUID of the vpx provider.  **NOTE, this is different than the MOB you may be used to logging into, see below ***

First, log into the Inventory Service MOB by pointing your browser to https://vCenterIP/invsvc/mob1/    From there, simply click the ‘RetrieveAllProviderConfigs’ link within the Methods section as shown below

invsvcprovider

In the pop up dialog, click ‘Invoke Method’, then run a search for vpx

vpxprovider

It’s the providerUuid string that we are looking for – go ahead and copy that string to your clipboard and return to https://vCenterIP/InvSvc/mob1/ – this time, clicking the ‘ResetProviderContent’ link under Methods.  In the pop up dialog, paste in your copied UUID and click ‘Invoke Method’ as shown below…

resetcontent

After a little while the window should refresh and hopefully you see no errors!   The process of resetting for myself took roughly 5 minutes to complete….

Getting rid of logs

Although vCenter does its own log rotation you may want to check out and see just how much space your logs are taking up on your current vCenter server before migrating as some of this data is processed during the migration/upgrade.  I freed up around 30GB of disk by purging some old logs – not a lot, but 30GB that didn’t need to be copied across the wire during the migration.  There is a great KB article here outlining the location and purpose of all of the vCenter Server log files – have a look at it and then peruse through your install and see what you may be able to get rid of.   For the windows version of vCenter you can find all of the logs in the %ALLUSERSPROFILE%\VMware\vCenterServer\logs\ folder.  I mostly purged anything that was gzipped and archived from most of the subfolders within this directory.  Again, not a difference maker in terms of unlocking my “Small” deployment option – but certainly a time-saver during the migration!  So what was culprit that was not allowing me to select “Small” – yeah, let’s get to that right now…

My Bloated vCenter Database

bloateddbYeah, 200GB is a little much right – even after resetting the vpx provider and shrinking the database files I was still sitting pretty high!  So, since I had no intention of migrating historical events, tasks and performance data I thought I’d look at purging it before hand!  Now if you have ever looked at the tables within your vCenter Server database you will find that VMware seems to create a lot of tables by  appending a number to the VPX_HIST_STAT table.  I had a lot of these – and going through them one by one wasn’t an option I felt like pursuing.  Thankfully, there’s a KB that provides a script to clean all of this up – you can find that here!  Go and get the MSSQL script in that KB and copy it over to your SQL Server.  Once you stop the vCenter Service we can simply run the following command via the command prompt on our SQL Server to peel through and purge our data.

sqlcmd -S IP-address-or-FQDN-of-the-database-machine\instance_name -U vCenter-Server-database-user -P password -d database-name -v TaskMaxAgeInDays=task-days -v EventMaxAgeInDays=event-days -v StatMaxAgeInDays=stat-days -i download-path\2110031_MS_SQL_task_event_stat.sql

Obviously you will need to assign some values to the parameters passed (TaskMaxAgeInDays, EventMaxAgeInDays, & StatMaxAgeInDays).  For these you have a few options.

  • -1 – skips the respective parameter and deletes no data
  • 1 or more – specifies that the data older than that amount of days will be purged
  • 0 – deletes it all!

For instance, I went with the 0, making my command look like the following….

sqlcmd -S IP-address-or-FQDN-of-the-database-machine\instance_name -U vCenter-Server-database-user -P password -d database-name -v TaskMaxAgeInDays=0 -v EventMaxAgeInDays=0 -v StatMaxAgeInDays=0 -i download-path\2110031_MS_SQL_task_event_stat.sql

After purging this data, and running a shrink on both my data and log files I finally had my vCenter database reduced in size – but only to 30GB.  Which, in all honesty still seemed a bit large to me – and after running the migration process again I still didn’t see my “Small” deployment option.   So I went looking for other large tables within the database and…..

Hello VPX_TEXT_ARRAY

It’s not very nice to meet you at all!!!  After finally getting down to this table – and running “sp_spaceused ‘VPX_TEXT_ARRAY’” I found that it was sitting a whopping 27GB.  Again, a flurry of Google!  What is VPX_TEXT_ARRAY and what data does it hold?  Can I purge it?  Well, yes….and no.  VPX_TEXT_ARRAY, from what I can gather keeps track of VM/Host/Datastore information – including information in regards to snapshots being performed on your VMs.  Also from what I can gather, from my environment anyways, is that this data exists within this table from, well, the beginning of time!  So, think about backup/replication products which constantly perform snapshots on VMs in order to protect them – yeah, this could cause that table to grow.  Also, if you are like me, and have a database that has been through a number of upgrades over the years you may end up having quite a bit of data and records within this table as it doesn’t seem to be processed in any sort of maintenance job.  In my case, 7 million records resided within VPX_TEXT_ARRAY.  Now, don’t just go and truncate that table as it most likely has current data residing in it – data vCenter needs in order to work – there’s a reason it tracks it all in the first place right?  Instead, we have to parse through the table, comparing the records with those that are in the VPX_ENTITY table, ensuring we only delete items which do not exist.  The SQL you can use to do so, below…

DELETE FROM VPX_TEXT_ARRAY
WHERE NOT EXISTS(SELECT 1 FROM VPX_ENTITY WHERE ID=VPX_TEXT_ARRAY.MO_ID)

A long and boring process – 18 hours later I was left with a mere 9000 records in my VPX_TEXT_ARRAY table.  Almost 7 Million removed.  Just a note, there is a KB outlining this information as well – in which it says to drop to SINGLE_USER mode – You can if you wish, but I simply just stopped my vCenter Server service and stayed in MULTI_USER so I could check in from time to time to ensure I was still actually removing records.  an sp_spaceused ‘VPX_TEXT_ARRAY’ in another query window will let you track just that.   Also, it might be easier, if you have the space, to set the initial size of your transaction logs something bigger than the amount of data in this table.  This allows SQL to not have to worry about growing them as it deletes records – you can always go back in the end and reset the initial size of the tlogs to 0 to shrink them.

So – a dozen coffees and a few days later I finally ran another shrink on both the data and log files, setting their initial sizes to 0 and voila – a 3GB database.  Another run at the migration and upgrade and there it was – the option to be “Small”!  Again, this worked in my environment – it may not work in yours – but it might help get you pointed in the right direction!  Do reach out if you have any questions and do ensure you have solid backups before you attempt any of this or anything you read on the net really Smile  Also, there’s always that Global Support Services thing that VMware provides if you want some help!   Thanks for reading!

VCSA 6.0 prompting for a manual fsck

One of my VCSA deployments, the only one running 6.0 experienced a switch failure and in result a network outage of roughly 5 minutes the other day.  Not a big deal, but unfortunately this was a very “cost effective” solution and the switch that hosted the production network also hosted the VLANs carrying all of the NFS traffic to the datastores the VCSA resided on as well!  In short, VCSA done got grumpy – after fixing the issues with the switch I ended up at the screen shown below…

VCSAFileSystemError

Not an overly complicated error – just stating that we need to run a file system check on the /dev/mapper/log_vg-log volume manually.  In the past, say with 5.5 I’d just drop to a bash shell and do so – however the default appliance shell in the 6.0 version of VCSA presents a few challenges in doing the same thing.  First off, if I went ahead and gave the root password to the VCSA I was presented with the default menu – the same menu you would receive if you ssh’d to the box under normal circumstances – that said, in the maintenance mode, the shell.set and shell.enable commands don’t work.  So in order to get to a point where we can actually execute fsck we need to do a couple of things…

Grub to bash

So the first thing we need to do is get our VCSA booting into a bash prompt.  To do so, hit CTRL+D at the presented screen and get the box to reboot.  When the boot loader appears we will need to hit the space bar or up/down keys to stop the auto boot process.  Once stopped we can selected ‘p’ to unlock the menu and enter the root password for the box.  We then want to select “e” to edit our boot sequence –  highlight the second line, the one that displays the kernel parameters and select “e” once again.  At the end of that line we will want to append “init=/bin/bash” as shown below – this will boot our system into a bash shell.  Once done, hit “enter” to save and “b” to boot.

grubmenu1

grubmenu2

After the system has booted you should now be sitting at a bash prompt.  On a normal day we would simply run our fsck command here however the file system we are looking to check is still not mounted at this point.  I tried numerous commands and options to try and get it mounted but came up short.  That said running the following command and rebooting our vCenter will switch the login shell for root back to the ‘normal bash’ and allow us to continue

chsh -s /bin/bash root

Once the command has been run and the server rebooted we will be brought back to the same error prompting us to enter the root password.  Go ahead and do that.  This time we will be brought directly to a bash prompt with log_vg-log being available to us!  So, without further ado go ahead and run the following command to complete the file system check.

fsck /dev/mapper/log_vg-log

More than likely you will get numerous prompts asking you whether or not to fix any errors that occur.  Use your discretion here, however I didn’t have much of a choice and needed to say ‘Yes’ to all.  After it’s done give the VCSA another reboot and everything should come back up normally (at least it did for me).  Hopefully this helps push someone in the right direction if they are experiencing similar issues 🙂

VMware vCenter Log Insight – Make your logs make sense!

VMware LogoToday VMware has introduced the world to VMware vCenter Log Insight, labeling it as a "new automated log management and analytics product for the cloud era".  In my opinion this is a great next step for VMware's management portfolio and if integrated correctly, could really compliment the analytics and performance data crunched by VMware vCenter Operations.  

More than just syslog?

From what I have seen, YES!  Although the underlying technology utilizes syslog collectors/receivers to receive the data, the visualizations and dashboards by which that data is presented to the end user is really where the value resides.  On average an ESXi host will dump roughly 250MB of data per day.  That's 250MB of data, that you, the end-user will need to parse and correlate line by line to try and make some sense out it.  I know I only understand about 25% (if that) of what is spit out in some of those logs.  vCenter Log Insight takes this data and with what they call 'content packs', presents the user with a bunch of predefined dashboards of some of the most relevant data that you may be looking for, along with common links to KB articles if any.

vCLI-Content

Easy transition from monitoring to troubleshooting

Hopefully we have all seen the power of vCenter Operations; How it correlates and analyzes all that data to really help us drill down and find out where any current (or future) problems exist.  If the issues are not evident, or if we are still unure of what the problem still is, the next viable step would be to jump into our logs to see what information we can find there.  With integration between vCOPs and vCenter Log Insight hopefully this will make that transition from our monitoring solutions into our log analyzing solutions a whole lot easier.  Again, saving us time and helping us discover root causes that much quicker.

vcli-integration

Even more for advanced users

For those that love to look at the raw log data (huh?!?!?!) you can do that as well.  A search type functionality, similar to that of Splunk is available as well.  Use this to parse and filter through all of your logs that vCenter Log Insight collects.  The main difference here is there is no need to learn any new "languages" to drill around in and query your data.  VMware seems to have really made a big effort to keep this product simple and easy to use, but powerful and extendable at the same time.  Also, the ability to generate alerts and send email notifications on a custom query is a very nice functionality to have.

vCLI-Query

More than just ESXi and vCenter

As mentioned above visualizations and presentations are provided by content packs.  These are easily exported and imported to and from vCenter Log Insight, in turn allowing third parties (including YOU) to easily develop, distribute and share.  So, hopefully, within time, we will see more than just ESXi and vCenter logs getting pumped into this.  On that note, we will probably see more than just VMware products being analyzed.  In my opinion the community will really need to take the lead on this one, and looking at past performance that the VMware community has, I'm sure they will!

So VMware says to expect to see some sort of GA in Q3 of this year, I'll let you guess the timeframe!  I hope to get a few more posts out about vCenter Log Insight as I delve more into the product but for now you can find some here, here and here.  Have a look for yourself and let me know if you think!

Backing up your vCenter DB – all three of’em

three-fingersWait!  What!  3!?!?!  Yes, you read correctly!  While in the days of vCenter 5.0 and below we only had to worry about 1 database, the release of 5.1 has tripled that!   This is something I hadn’t even thought about until recently attending a vBrownbag put on by Justin King (@vcenterGuy).

So there’s the SQL database from vCenter right – ok – no big, I know about that one and am backing it up – KB article on how to do that.

Then there is the SSO database – no problem, I knew about that one as well since I had to create it when I first upgraded to 5.1.  Again it’s a MS SQL DB, doesn’t change that much –  which is easy enough to backup…

But then Justin started talking about the Inventory service – remember, that’s the third requirement you had to install when upgrading.  Well guess what?  It has a database too!  It’s not SQL at all – it’s sitting on your vCenter Server in xDB format.  My first thought was what is even in this database – I can’t browse it like I can the SQL databases (or I just don’t know how to).  What I can gather from the What’s New docs and VMworld presentations the Inventory database holds things such as a read cache of all the objects that are accessed within the vSphere Web Client and all of your tags and categories that are setup from vCenter.  I’m sure there’s more but this is all I can find.

However back to my main objective, how do I back this thing up?  A little digging around and I found this KB article on how to backup/restore your vCenter Inventory database.  Basically it’s as follows.

Backing up the inventory database (WINDOWS)

  1. Navigate to the Inventory scripts folder (c:\Program Files\VMware\Infrastructure\Inventory Service\scripts)
  2. Run the following
    • backup.bat -file backup_filename

Restoring the inventory database (WINDOWS)

  1. Navigate to the Inventory scripts folder (c:\Program Files\VMware\Infrastructure\Inventory Service\scripts)
  2. Run the following
    • restore -backup backup_filename

So utterly simple yet so not talked about 🙂  Wait – but what if I’m using the vCSA?  Am I out of luck?  Absolutely not!  Use the following…

Backing up the inventory database (Linux)

  1. Navigate to the Inventory scripts folder (/usr/lib/vmware-vpx/inventoryservice/scripts/)
  2. Run the following
    • ./backup.sh -file backup_filename

Restoring the inventory database (Linux)

  1. Navigate to the Inventory scripts folder (/usr/lib/vmware-vpx/inventoryservice/scripts/)
  2. Run the following
    • ./restore.sh -backup backup_filename

So there you go!  You can now sleep at night knowing you aren’t going to lose all of your hard work setting up those tags!  Moral of the story – Pay attention and participate in the vBrownBags – there is always some great information and learning to be had.

My first vCenter Orchestrator Worlkflow – Part 1 – Introduction

Orchestrate all of the thingsvCenter Orchestrator in it’s basics is a workflow development tool that can be used to schedule and automate multiple tasks in your environment.  A tool that can be used to create and execute workflows not only on your vSphere environment, but with the use of plugins you can manage other types of applications such as Active Directory, SQL, etc.  In the past I’ve often heard of vCO being described as a hidden gem inside of your vCenter Server; meaning it’s usually already there and licensed  but for the most part, you are probably not using it.  Such is the case with myself. I’ve often thought about trying to learn this technology in order to execute some of my common day to day tasks through a more automated and scripted fashion but I’ve always resulted to things like PowerShell and PowerCLI to do the same job.  Why?  Well, comfort really.  I already have a good handle on Powershell technology and could get things done faster there rather than learning something new.  That being said with the introduction of vSphere 5.1 a bunch of new features and enhancements were made with the vCenter/vCO interoperability.  The biggest IMO was the ability to execute a vCO workflow while directly inside of the vSphere Web Client – contextually!!!  This integration is what enticed to have a closer look at vCO.  Basically I now have the ability to create workflows that can do pretty much anything and grant access to myself or to others to simply right-click a host from within vSphere and execute them.

contextualThus leads me to these series of posts where I will try and take you though my experiences with vCenter Orchestrator; and it couldn’t come at a more opportune time.  I have approx 50 hosts to configure and deploy within the next few months and in effort to keep them consistent I decided to do so with a PowerCLI script that I had written a while back.  The only difference being I will be executing this script through vCenter Orchestrator (to get that super awesome right-click functionality).  Now there is certainly some overlap here.  A lot of the actions that the PowerCLI script performs actually have workflows already created in vCO that do the same thing however in this case I’ve decided not to use them – baby steps right!  Also, it helps to highlight the power of the vCO plugins – I can in fact do things like execute PowerShell commands, run queries against SQL servers, move objects around in Active Directory, etc.

So with that said I guess you could call this post an introduction of sorts with nothing too technical included.  Be sure to check out Part 2 – Installing and configuring vCenter Orchestrator where we will dive a bit deeper into the setup of vCO.

My first vCenter Orchestrator Workflow

vCenter Operations Manager now sits at 5.7

VMware LogoIt should be no surprise to any of my regular readers or followers that I am a huge fan of vCenter Operations.  Being a VMware customer I find that it is a huge time-saver when trying to pin point performance issues within our environment, as well as giving us a great first step in trying to do capacity planning and figure out where we are going to need to go next.  So, it should also be no surprise that I get just a little excited when there is a new release of the product; be it only a .1 release, still super awesome none the less.

It goes without saying that you should see a few posts diving deeper into some of the new features listed below as well as in the official release notes, as well as a quckie about how to upgrade, but for now, without further ado, the newest features from vCenter Operations 5.7 from vmware.com…

More Flexibility with Capacity Planning

  • Assess capacity risk and plan by allocation and/or actual demand: Set policies based on your varying business needs to assess capacity risk, efficiency, and forecast. For example, different buffers, over-commit ratios, alert thresholds, business hours, etc., across production and test-dev environments.
  • New views for Cluster Capacity Risk: Quickly identify via color-coded Cluster capacity risk view which clusters grouped by business criteria, etc., are at capacity risk—facing a capacity shortfall now or in the near future or just not sized right. Drill down for each cluster in the Cluster Risk Detail view to analyze which resource is it constrained on and why.
  • New policies for common environments and workloads: New out-of-the-box policies, such as Production and Test-Dev policies, enable quick set-up of vCenter Operations Manager capacity settings for common types of environments. Additional new out-of-the-box policies, such as Batch workload, Interactive workload, and Ignore VMs policies, help fine-tune capacity configuration settings to accurately right size and analyze different workloads based on their performance characteristics.

Improved Self-Monitoring

This release introduces new diagnostics metrics to monitor the health and availability of vCenter Operations Manager components, such as Analytics, Collector, Active MQ, Web server, database, and operating system.

Widgets with Improved Flexibility and Usability

  • Health Tree Widget: Easy visualization for large number of objects.
  • Generic Scoreboard Widget: Support for Sparkline, string metrics, and metrics filtering by resource.
  • Metric Sparkline Widget: Configurable color ranges and units, support for resource type and label.
  • Resource Widget: Customizable to add metrics beyond health.
  • Top-N Analysis Widget: Support for analysis based on latest values.

New Custom Relationship Widget

Allows you to build a custom resource hierarchy and relationship view, just like the existing out-of-the-box vCenter Server view.

Custom UI Import and Export Changes for Dashboards and Super Metrics

  • Export format changed from binary (.bin) to XML (.xml): .bin formats are still supported for backward compatibility.
  • DBCLI Enhancements: Programmatically import and export Super Metrics.
  • Pre-population of Dashboard objects during import.

Balanced Metrics Profile

This release introduces a new metrics profile that reports a reduced set of metrics. Increase the scalability of vCenter Operations Manager to support more resources by changing the metrics profile to the new “Balanced” profile in vCenter Operations Manager Administration.

VMware vCenter Infrastructure Navigator Filtering Capability

You can configure how resources discovered by vCenter Infrastructure Navigator are displayed in vCenter Operations Manager. This release introduces a configurable filtering capability to the vCenter Infrastructure Navigator adapter to control Application service and Application resource reporting. For each resource type, you can configure either “blackList” or “whiteList” filtering in the configuration file filterList.txt.

  • blackList: The vCenter Infrastructure Navigator adapter ignores specified entries. If an Application Service name or an Application name is included in the “blackList,” it is not reported by the vCenter Infrastructure Navigator adapter. This is the default setting. The vCenter Infrastructure Navigator adapter filters unknown Application service names by default.
  • whiteList: The vCenter Infrastructure Navigator adapter reports only the specified entries. If there are no entries added to the whiteList mode, none of the resources of the corresponding resource type are displayed.

 New Browser Support

This release adds new support for the following browsers: Apple Safari version 6, Google Chrome versions 24 and 25, and Mozilla Firefox 18 and 19.

Security Hardening

This release includes additional security hardening and increases compliance with The Defense Information Systems Agency (DISA) and The Security Technical Implementation Guides (STIG) guidelines.

 Go and download a fully featured 60 day trial for yourself here.