Tag Archives: vCenter

VCSA 6.0 prompting for a manual fsck

One of my VCSA deployments, the only one running 6.0 experienced a switch failure and in result a network outage of roughly 5 minutes the other day.  Not a big deal, but unfortunately this was a very “cost effective” solution and the switch that hosted the production network also hosted the VLANs carrying all of the NFS traffic to the datastores the VCSA resided on as well!  In short, VCSA done got grumpy – after fixing the issues with the switch I ended up at the screen shown below…

VCSAFileSystemError

Not an overly complicated error – just stating that we need to run a file system check on the /dev/mapper/log_vg-log volume manually.  In the past, say with 5.5 I’d just drop to a bash shell and do so – however the default appliance shell in the 6.0 version of VCSA presents a few challenges in doing the same thing.  First off, if I went ahead and gave the root password to the VCSA I was presented with the default menu – the same menu you would receive if you ssh’d to the box under normal circumstances – that said, in the maintenance mode, the shell.set and shell.enable commands don’t work.  So in order to get to a point where we can actually execute fsck we need to do a couple of things…

Grub to bash

So the first thing we need to do is get our VCSA booting into a bash prompt.  To do so, hit CTRL+D at the presented screen and get the box to reboot.  When the boot loader appears we will need to hit the space bar or up/down keys to stop the auto boot process.  Once stopped we can selected ‘p’ to unlock the menu and enter the root password for the box.  We then want to select “e” to edit our boot sequence –  highlight the second line, the one that displays the kernel parameters and select “e” once again.  At the end of that line we will want to append “init=/bin/bash” as shown below – this will boot our system into a bash shell.  Once done, hit “enter” to save and “b” to boot.

grubmenu1

grubmenu2

After the system has booted you should now be sitting at a bash prompt.  On a normal day we would simply run our fsck command here however the file system we are looking to check is still not mounted at this point.  I tried numerous commands and options to try and get it mounted but came up short.  That said running the following command and rebooting our vCenter will switch the login shell for root back to the ‘normal bash’ and allow us to continue

chsh -s /bin/bash root

Once the command has been run and the server rebooted we will be brought back to the same error prompting us to enter the root password.  Go ahead and do that.  This time we will be brought directly to a bash prompt with log_vg-log being available to us!  So, without further ado go ahead and run the following command to complete the file system check.

fsck /dev/mapper/log_vg-log

More than likely you will get numerous prompts asking you whether or not to fix any errors that occur.  Use your discretion here, however I didn’t have much of a choice and needed to say ‘Yes’ to all.  After it’s done give the VCSA another reboot and everything should come back up normally (at least it did for me).  Hopefully this helps push someone in the right direction if they are experiencing similar issues 🙂

VMware vCenter Log Insight – Make your logs make sense!

VMware LogoToday VMware has introduced the world to VMware vCenter Log Insight, labeling it as a "new automated log management and analytics product for the cloud era".  In my opinion this is a great next step for VMware's management portfolio and if integrated correctly, could really compliment the analytics and performance data crunched by VMware vCenter Operations.  

More than just syslog?

From what I have seen, YES!  Although the underlying technology utilizes syslog collectors/receivers to receive the data, the visualizations and dashboards by which that data is presented to the end user is really where the value resides.  On average an ESXi host will dump roughly 250MB of data per day.  That's 250MB of data, that you, the end-user will need to parse and correlate line by line to try and make some sense out it.  I know I only understand about 25% (if that) of what is spit out in some of those logs.  vCenter Log Insight takes this data and with what they call 'content packs', presents the user with a bunch of predefined dashboards of some of the most relevant data that you may be looking for, along with common links to KB articles if any.

vCLI-Content

Easy transition from monitoring to troubleshooting

Hopefully we have all seen the power of vCenter Operations; How it correlates and analyzes all that data to really help us drill down and find out where any current (or future) problems exist.  If the issues are not evident, or if we are still unure of what the problem still is, the next viable step would be to jump into our logs to see what information we can find there.  With integration between vCOPs and vCenter Log Insight hopefully this will make that transition from our monitoring solutions into our log analyzing solutions a whole lot easier.  Again, saving us time and helping us discover root causes that much quicker.

vcli-integration

Even more for advanced users

For those that love to look at the raw log data (huh?!?!?!) you can do that as well.  A search type functionality, similar to that of Splunk is available as well.  Use this to parse and filter through all of your logs that vCenter Log Insight collects.  The main difference here is there is no need to learn any new "languages" to drill around in and query your data.  VMware seems to have really made a big effort to keep this product simple and easy to use, but powerful and extendable at the same time.  Also, the ability to generate alerts and send email notifications on a custom query is a very nice functionality to have.

vCLI-Query

More than just ESXi and vCenter

As mentioned above visualizations and presentations are provided by content packs.  These are easily exported and imported to and from vCenter Log Insight, in turn allowing third parties (including YOU) to easily develop, distribute and share.  So, hopefully, within time, we will see more than just ESXi and vCenter logs getting pumped into this.  On that note, we will probably see more than just VMware products being analyzed.  In my opinion the community will really need to take the lead on this one, and looking at past performance that the VMware community has, I'm sure they will!

So VMware says to expect to see some sort of GA in Q3 of this year, I'll let you guess the timeframe!  I hope to get a few more posts out about vCenter Log Insight as I delve more into the product but for now you can find some here, here and here.  Have a look for yourself and let me know if you think!

Backing up your vCenter DB – all three of’em

three-fingersWait!  What!  3!?!?!  Yes, you read correctly!  While in the days of vCenter 5.0 and below we only had to worry about 1 database, the release of 5.1 has tripled that!   This is something I hadn’t even thought about until recently attending a vBrownbag put on by Justin King (@vcenterGuy).

So there’s the SQL database from vCenter right – ok – no big, I know about that one and am backing it up – KB article on how to do that.

Then there is the SSO database – no problem, I knew about that one as well since I had to create it when I first upgraded to 5.1.  Again it’s a MS SQL DB, doesn’t change that much –  which is easy enough to backup…

But then Justin started talking about the Inventory service – remember, that’s the third requirement you had to install when upgrading.  Well guess what?  It has a database too!  It’s not SQL at all – it’s sitting on your vCenter Server in xDB format.  My first thought was what is even in this database – I can’t browse it like I can the SQL databases (or I just don’t know how to).  What I can gather from the What’s New docs and VMworld presentations the Inventory database holds things such as a read cache of all the objects that are accessed within the vSphere Web Client and all of your tags and categories that are setup from vCenter.  I’m sure there’s more but this is all I can find.

However back to my main objective, how do I back this thing up?  A little digging around and I found this KB article on how to backup/restore your vCenter Inventory database.  Basically it’s as follows.

Backing up the inventory database (WINDOWS)

  1. Navigate to the Inventory scripts folder (c:\Program Files\VMware\Infrastructure\Inventory Service\scripts)
  2. Run the following
    • backup.bat -file backup_filename

Restoring the inventory database (WINDOWS)

  1. Navigate to the Inventory scripts folder (c:\Program Files\VMware\Infrastructure\Inventory Service\scripts)
  2. Run the following
    • restore -backup backup_filename

So utterly simple yet so not talked about 🙂  Wait – but what if I’m using the vCSA?  Am I out of luck?  Absolutely not!  Use the following…

Backing up the inventory database (Linux)

  1. Navigate to the Inventory scripts folder (/usr/lib/vmware-vpx/inventoryservice/scripts/)
  2. Run the following
    • ./backup.sh -file backup_filename

Restoring the inventory database (Linux)

  1. Navigate to the Inventory scripts folder (/usr/lib/vmware-vpx/inventoryservice/scripts/)
  2. Run the following
    • ./restore.sh -backup backup_filename

So there you go!  You can now sleep at night knowing you aren’t going to lose all of your hard work setting up those tags!  Moral of the story – Pay attention and participate in the vBrownBags – there is always some great information and learning to be had.

My first vCenter Orchestrator Worlkflow – Part 1 – Introduction

Orchestrate all of the thingsvCenter Orchestrator in it’s basics is a workflow development tool that can be used to schedule and automate multiple tasks in your environment.  A tool that can be used to create and execute workflows not only on your vSphere environment, but with the use of plugins you can manage other types of applications such as Active Directory, SQL, etc.  In the past I’ve often heard of vCO being described as a hidden gem inside of your vCenter Server; meaning it’s usually already there and licensed  but for the most part, you are probably not using it.  Such is the case with myself. I’ve often thought about trying to learn this technology in order to execute some of my common day to day tasks through a more automated and scripted fashion but I’ve always resulted to things like PowerShell and PowerCLI to do the same job.  Why?  Well, comfort really.  I already have a good handle on Powershell technology and could get things done faster there rather than learning something new.  That being said with the introduction of vSphere 5.1 a bunch of new features and enhancements were made with the vCenter/vCO interoperability.  The biggest IMO was the ability to execute a vCO workflow while directly inside of the vSphere Web Client – contextually!!!  This integration is what enticed to have a closer look at vCO.  Basically I now have the ability to create workflows that can do pretty much anything and grant access to myself or to others to simply right-click a host from within vSphere and execute them.

contextualThus leads me to these series of posts where I will try and take you though my experiences with vCenter Orchestrator; and it couldn’t come at a more opportune time.  I have approx 50 hosts to configure and deploy within the next few months and in effort to keep them consistent I decided to do so with a PowerCLI script that I had written a while back.  The only difference being I will be executing this script through vCenter Orchestrator (to get that super awesome right-click functionality).  Now there is certainly some overlap here.  A lot of the actions that the PowerCLI script performs actually have workflows already created in vCO that do the same thing however in this case I’ve decided not to use them – baby steps right!  Also, it helps to highlight the power of the vCO plugins – I can in fact do things like execute PowerShell commands, run queries against SQL servers, move objects around in Active Directory, etc.

So with that said I guess you could call this post an introduction of sorts with nothing too technical included.  Be sure to check out Part 2 – Installing and configuring vCenter Orchestrator where we will dive a bit deeper into the setup of vCO.

My first vCenter Orchestrator Workflow

vCenter Operations Manager now sits at 5.7

VMware LogoIt should be no surprise to any of my regular readers or followers that I am a huge fan of vCenter Operations.  Being a VMware customer I find that it is a huge time-saver when trying to pin point performance issues within our environment, as well as giving us a great first step in trying to do capacity planning and figure out where we are going to need to go next.  So, it should also be no surprise that I get just a little excited when there is a new release of the product; be it only a .1 release, still super awesome none the less.

It goes without saying that you should see a few posts diving deeper into some of the new features listed below as well as in the official release notes, as well as a quckie about how to upgrade, but for now, without further ado, the newest features from vCenter Operations 5.7 from vmware.com…

More Flexibility with Capacity Planning

  • Assess capacity risk and plan by allocation and/or actual demand: Set policies based on your varying business needs to assess capacity risk, efficiency, and forecast. For example, different buffers, over-commit ratios, alert thresholds, business hours, etc., across production and test-dev environments.
  • New views for Cluster Capacity Risk: Quickly identify via color-coded Cluster capacity risk view which clusters grouped by business criteria, etc., are at capacity risk—facing a capacity shortfall now or in the near future or just not sized right. Drill down for each cluster in the Cluster Risk Detail view to analyze which resource is it constrained on and why.
  • New policies for common environments and workloads: New out-of-the-box policies, such as Production and Test-Dev policies, enable quick set-up of vCenter Operations Manager capacity settings for common types of environments. Additional new out-of-the-box policies, such as Batch workload, Interactive workload, and Ignore VMs policies, help fine-tune capacity configuration settings to accurately right size and analyze different workloads based on their performance characteristics.

Improved Self-Monitoring

This release introduces new diagnostics metrics to monitor the health and availability of vCenter Operations Manager components, such as Analytics, Collector, Active MQ, Web server, database, and operating system.

Widgets with Improved Flexibility and Usability

  • Health Tree Widget: Easy visualization for large number of objects.
  • Generic Scoreboard Widget: Support for Sparkline, string metrics, and metrics filtering by resource.
  • Metric Sparkline Widget: Configurable color ranges and units, support for resource type and label.
  • Resource Widget: Customizable to add metrics beyond health.
  • Top-N Analysis Widget: Support for analysis based on latest values.

New Custom Relationship Widget

Allows you to build a custom resource hierarchy and relationship view, just like the existing out-of-the-box vCenter Server view.

Custom UI Import and Export Changes for Dashboards and Super Metrics

  • Export format changed from binary (.bin) to XML (.xml): .bin formats are still supported for backward compatibility.
  • DBCLI Enhancements: Programmatically import and export Super Metrics.
  • Pre-population of Dashboard objects during import.

Balanced Metrics Profile

This release introduces a new metrics profile that reports a reduced set of metrics. Increase the scalability of vCenter Operations Manager to support more resources by changing the metrics profile to the new “Balanced” profile in vCenter Operations Manager Administration.

VMware vCenter Infrastructure Navigator Filtering Capability

You can configure how resources discovered by vCenter Infrastructure Navigator are displayed in vCenter Operations Manager. This release introduces a configurable filtering capability to the vCenter Infrastructure Navigator adapter to control Application service and Application resource reporting. For each resource type, you can configure either “blackList” or “whiteList” filtering in the configuration file filterList.txt.

  • blackList: The vCenter Infrastructure Navigator adapter ignores specified entries. If an Application Service name or an Application name is included in the “blackList,” it is not reported by the vCenter Infrastructure Navigator adapter. This is the default setting. The vCenter Infrastructure Navigator adapter filters unknown Application service names by default.
  • whiteList: The vCenter Infrastructure Navigator adapter reports only the specified entries. If there are no entries added to the whiteList mode, none of the resources of the corresponding resource type are displayed.

 New Browser Support

This release adds new support for the following browsers: Apple Safari version 6, Google Chrome versions 24 and 25, and Mozilla Firefox 18 and 19.

Security Hardening

This release includes additional security hardening and increases compliance with The Defense Information Systems Agency (DISA) and The Security Technical Implementation Guides (STIG) guidelines.

 Go and download a fully featured 60 day trial for yourself here.

Upgraded to vCenter 5.1 – now I need to login using DOMAIN\Username ?!?!?

VMware LogoSomething I’ve noticed on my 3 upgrades of vCenter so far is that after implementing SSO and upgrading to vCenter 5.1 users now have to log into the clients using the format DOMAIN\Username.  Not a big deal, but different than what they are previously used to, and different in most cases is not good.  Honestly, it drives me a little crazy too 🙂

WebClientLogin

Thankfully it’s not a big job to get things back to the way they were.  All that needs to be done is to actually set up your domain as one of the default domains, something which doesn’t seem to be applied by default (in my case anyways).  To do so you will need the new web client and you will need to login using your administrative SSO credentials that you setup during that install.  By default the username is admin@system-domain and the password is one that you have created (can’t tell you that 🙂 )

AddToDefaultDomainsOnce your in select Administration from the left hand navigational menu and then Configuration in the Sign-On and Discovery section.  First off you should see your domain configuration listed in the top section of the Identity Sources tab.  Simply select it and then click the Default Domains button (shown left) to add it to your list of default domains.  You may experience a warning at this pointing, something about locking out accounts – I ignored this and have not had any issues at all, but it’s your choice whether you want to investigate this further.

 

DefaultDomainsDoneOne more thing, do not forget to apply your changes by clicking the ‘Save’ icon – this one bit me about three times before I even noticed it was there 🙂

 

As always I appreciate any comments, questions and concerns below 🙂