Tag Archives: VMware

No vMotion for you! – A general system error occurred: vim.faultNotFound

vMotion is pretty awesome am I right?  Ever since I first saw my first VM migrate from one host to another without losing a beat I was pretty blown away – you always remember your first Smile  In my opinion it’s the vMotion feature that truly brought VMware to where they are today – laid the groundwork for all of the amazing features you see in the current release.  It’s something I’ve taken for granted as of late – which is why I was a little perplexed when all of a sudden, for only a few VMs, it just stopped working…

vMotionError

You can see above one of my VMs that just didn’t seem to want to budge!  Thankfully we get a very descriptive and helpful error message of “A general system error occurred: vim.faultNotFound” – you know, because that really helps a lot!  With my Google-Fu turning up no results and coming up empty handed in forum scouring I decided to take a step back to the VCP days and look at what the actual requirements of vMotion are – surely, this VM is not meeting one of them!  So with that, a simplified version of the requirements to vMotion…

  • Proper vSphere licencing
  • Compatible CPUs
  • Shared Storage (for normal vMotion)
  • vMotion portgroups on the hosts (min 1GB)
  • Sufficient Resources on target hosts
  • Same names for port groups

Licensing – check!  vCloud Suite

CPU Compatibility – check! Cluster of blades all identical

Shared Storage – check!  LUNs available on all hosts

vMotion interface – check!  Other VMs moved no problem

Sufficient Resources – check!  Lots of resources free!

Same names for port groups – check!  Using a distributed switch.

So, yeah, huh?

Since I’d already moved a couple dozen other VMs and the fact that this single VM was failing no matter what host I tried to move it to I ruled out the fact that there was anything host related causing this and focussed my attention to the single VM.  Firstly I thought maybe the VM was tied to the host somehow, using local resources of some sort – but the VM had no local storage attached to it, no CD ROMs mounted, nothing – it was the perfect candidate for vMotion but no matter what I tried I couldn’t get this VM to move!  I then turned my attention to networking – maybe there was an issue with the ports on the distributed switch, possibly having none available.

After a quick glance, there was lots of ports available, but there was another abnormality that reared its ugly head!  The VM was listed as being connected to the switch on the ‘VMs’ tab – however on the ‘Ports’ tab it was nowhere to be found!   So what port was this VM connected to?  Well, let’s ssh directly to the host to figure this one out…

To figure this out we need to run the “esxcli network vm port list” command and pass it the VMs worldID – to get that, we can simply execute the following

esxcli network vm list

From there, we can grab the world ID of our VM in question and run the following

esxcli network vm port list –w world_id

In my case, I came up with the following…

vmportid

Port 317!  Sounds normal right?  Not in my case.  In fact, I knew for certain from my documentation that the ports on this port group only went up to 309!  So, I had a VM, connected to the port group, on a port that essentially didn’t exist!

How about a TL;DR version?

Problem stemmed from the VM being connected to essentially a non-existent port!  Since I couldn’t have any downtime on this port my fix was to simply create a another port group on the dvSwitch, mimicking the settings from the first.  After attaching the VM to the newly built port group, then re-attaching back to the existing one I was finally attached to what I saw as a valid port, Port #271.

port-fixed

After doing this guess what finally started working again – that’s right, the wonderful and amazing vMotion Smile.  I’m sure you could achieve the same result by simply disconnecting and connecting, however you will experience downtime with that method – so I went the duplicate port group route.

Where there is one there’s many

All of this got me thinking – this can’t be the only VM that’s experiencing this issue is it?  I started looking around trying to find some PowerCLI scripts that I could piece together and as it turns out, knowing what the specific problem was certainly helps with the Google-Fu and I found a blog by Jason Coleman dealing with this exact same issue!  Wish I could’ve found that earlier Smile.  Anyways, Jason has  a great PowerCLI script attached to his post that peels through and detects which VMs in your environment are experiencing this exact problem!  He even has automated the creation of the temporary port groups as well!  Good work Jason!  After running it my conclusions were correct – there were about a dozen VMs that needed fixing in my environment.

How or why this occurred I have no idea – I’m just glad I found a way around it and as always, thought I’d share with intention of maybe helping others!  Also – it gave me a chance to throw in some Seinfeld action on the blog!  Thanks for reading!

VCSA 6.5 Migration deployment sizes limited!

Recently I finally bit the bullet and decided to bring the vCenter portion of a vSphere environment up to version 6.5.  Since the migration from a Windows based vCenter to the VCSA is now a supported path I thought it would also be a good time to migrate to the appliance as well.  So with that I ran through a few blogs I found in regards to the migration, checked out the vSphere Upgrade Guide and peeled through a number KB’s looking for gotchya’s.  With my knowledge in hand I headed into the migration.

At this point I had already migrated my external windows based PSC to version 6.5 and got started on the migration of the windows-based vCenter Server.  Following the wizard I was prompted for the typical SSO information along with where I would like to place the appliance.  The problem though came when I was prompted to select a deployment size for my new VCSA.  My only options available were Large and X-Large.  Might not be a big deal if in fact this environment required this amount of resources – Looking at the table below those deployment sizes are scoped to fit at a 1000 host and above mark.

DeploymentSize

Did this environment have 1000+ hosts and 10000+ VMs?  Absolutely not!  At its largest it contained maybe 70 hosts and a few hundred VMs running on them – a Small configuration at best, medium if you want to be conservative!  At first I thought maybe I was over provisioned in terms of resources on my current vCenter Server – but again, it only had 8 vCPU’s and 16GB of RAM.  With nothing out of the ordinary with vCenter itself I turned my attention to the database – and that’s where my attention stayed as it was currently sitting at a size of 200GB.  Honestly, this seemed super big to me and knowing that it had been through a number of upgrades over the years I figured I would make it my goal to shrink this down as small as possible before trying again!  TL;DR; version – The database was the culprit and I did end up with the “small” option –  but I did a number of things after a frenzy of Google’s and searches – all listed below…

WAIT!!!!  Don’t be that guy!  Make sure you have  solid backups and can restore if things here go sideways – engage VMware GSS if needed – don’t just “do what I do” 🙂

 

Reset the vpx provider

The vpx data provider basically supplies the object cache for vCenter – caching all inventory objects such as hosts, clusters, VMs, etc in order to provide that super-snappy response time in the vSphere Web Client 6.0 (Is this sarcasm?).  Anyways, resetting this essentially will reduce the size of our Inventory Database.  Now, the problem in versions prior to 5.5 Update 3 is that there was no way to reset individual data providers – in order to do one you had to do them all – and that meant losing all of your tags, storage profiles/policies, etc.  Thankfully, 5.5 U3 and 6.0 allows us to simply reset just vpx, leaving the rest of our environment in-tact.  In order to do so we must first get into the vSphere Inventory Managed Object Browser (MOB) and get the UUID of the vpx provider.  **NOTE, this is different than the MOB you may be used to logging into, see below ***

First, log into the Inventory Service MOB by pointing your browser to https://vCenterIP/invsvc/mob1/    From there, simply click the ‘RetrieveAllProviderConfigs’ link within the Methods section as shown below

invsvcprovider

In the pop up dialog, click ‘Invoke Method’, then run a search for vpx

vpxprovider

It’s the providerUuid string that we are looking for – go ahead and copy that string to your clipboard and return to https://vCenterIP/InvSvc/mob1/ – this time, clicking the ‘ResetProviderContent’ link under Methods.  In the pop up dialog, paste in your copied UUID and click ‘Invoke Method’ as shown below…

resetcontent

After a little while the window should refresh and hopefully you see no errors!   The process of resetting for myself took roughly 5 minutes to complete….

Getting rid of logs

Although vCenter does its own log rotation you may want to check out and see just how much space your logs are taking up on your current vCenter server before migrating as some of this data is processed during the migration/upgrade.  I freed up around 30GB of disk by purging some old logs – not a lot, but 30GB that didn’t need to be copied across the wire during the migration.  There is a great KB article here outlining the location and purpose of all of the vCenter Server log files – have a look at it and then peruse through your install and see what you may be able to get rid of.   For the windows version of vCenter you can find all of the logs in the %ALLUSERSPROFILE%\VMware\vCenterServer\logs\ folder.  I mostly purged anything that was gzipped and archived from most of the subfolders within this directory.  Again, not a difference maker in terms of unlocking my “Small” deployment option – but certainly a time-saver during the migration!  So what was culprit that was not allowing me to select “Small” – yeah, let’s get to that right now…

My Bloated vCenter Database

bloateddbYeah, 200GB is a little much right – even after resetting the vpx provider and shrinking the database files I was still sitting pretty high!  So, since I had no intention of migrating historical events, tasks and performance data I thought I’d look at purging it before hand!  Now if you have ever looked at the tables within your vCenter Server database you will find that VMware seems to create a lot of tables by  appending a number to the VPX_HIST_STAT table.  I had a lot of these – and going through them one by one wasn’t an option I felt like pursuing.  Thankfully, there’s a KB that provides a script to clean all of this up – you can find that here!  Go and get the MSSQL script in that KB and copy it over to your SQL Server.  Once you stop the vCenter Service we can simply run the following command via the command prompt on our SQL Server to peel through and purge our data.

sqlcmd -S IP-address-or-FQDN-of-the-database-machine\instance_name -U vCenter-Server-database-user -P password -d database-name -v TaskMaxAgeInDays=task-days -v EventMaxAgeInDays=event-days -v StatMaxAgeInDays=stat-days -i download-path\2110031_MS_SQL_task_event_stat.sql

Obviously you will need to assign some values to the parameters passed (TaskMaxAgeInDays, EventMaxAgeInDays, & StatMaxAgeInDays).  For these you have a few options.

  • -1 – skips the respective parameter and deletes no data
  • 1 or more – specifies that the data older than that amount of days will be purged
  • 0 – deletes it all!

For instance, I went with the 0, making my command look like the following….

sqlcmd -S IP-address-or-FQDN-of-the-database-machine\instance_name -U vCenter-Server-database-user -P password -d database-name -v TaskMaxAgeInDays=0 -v EventMaxAgeInDays=0 -v StatMaxAgeInDays=0 -i download-path\2110031_MS_SQL_task_event_stat.sql

After purging this data, and running a shrink on both my data and log files I finally had my vCenter database reduced in size – but only to 30GB.  Which, in all honesty still seemed a bit large to me – and after running the migration process again I still didn’t see my “Small” deployment option.   So I went looking for other large tables within the database and…..

Hello VPX_TEXT_ARRAY

It’s not very nice to meet you at all!!!  After finally getting down to this table – and running “sp_spaceused ‘VPX_TEXT_ARRAY’” I found that it was sitting a whopping 27GB.  Again, a flurry of Google!  What is VPX_TEXT_ARRAY and what data does it hold?  Can I purge it?  Well, yes….and no.  VPX_TEXT_ARRAY, from what I can gather keeps track of VM/Host/Datastore information – including information in regards to snapshots being performed on your VMs.  Also from what I can gather, from my environment anyways, is that this data exists within this table from, well, the beginning of time!  So, think about backup/replication products which constantly perform snapshots on VMs in order to protect them – yeah, this could cause that table to grow.  Also, if you are like me, and have a database that has been through a number of upgrades over the years you may end up having quite a bit of data and records within this table as it doesn’t seem to be processed in any sort of maintenance job.  In my case, 7 million records resided within VPX_TEXT_ARRAY.  Now, don’t just go and truncate that table as it most likely has current data residing in it – data vCenter needs in order to work – there’s a reason it tracks it all in the first place right?  Instead, we have to parse through the table, comparing the records with those that are in the VPX_ENTITY table, ensuring we only delete items which do not exist.  The SQL you can use to do so, below…

DELETE FROM VPX_TEXT_ARRAY
WHERE NOT EXISTS(SELECT 1 FROM VPX_ENTITY WHERE ID=VPX_TEXT_ARRAY.MO_ID)

A long and boring process – 18 hours later I was left with a mere 9000 records in my VPX_TEXT_ARRAY table.  Almost 7 Million removed.  Just a note, there is a KB outlining this information as well – in which it says to drop to SINGLE_USER mode – You can if you wish, but I simply just stopped my vCenter Server service and stayed in MULTI_USER so I could check in from time to time to ensure I was still actually removing records.  an sp_spaceused ‘VPX_TEXT_ARRAY’ in another query window will let you track just that.   Also, it might be easier, if you have the space, to set the initial size of your transaction logs something bigger than the amount of data in this table.  This allows SQL to not have to worry about growing them as it deletes records – you can always go back in the end and reset the initial size of the tlogs to 0 to shrink them.

So – a dozen coffees and a few days later I finally ran another shrink on both the data and log files, setting their initial sizes to 0 and voila – a 3GB database.  Another run at the migration and upgrade and there it was – the option to be “Small”!  Again, this worked in my environment – it may not work in yours – but it might help get you pointed in the right direction!  Do reach out if you have any questions and do ensure you have solid backups before you attempt any of this or anything you read on the net really Smile  Also, there’s always that Global Support Services thing that VMware provides if you want some help!   Thanks for reading!

Friday Shorts – VMware, Veeam, Docker and Community

Dreams are important!  Otherwise, sleep is just 8 hours of nothing. – MacGyver

vSphere 6.5 is here!

VMware LogoNDA’s have been lifted and the flood gates have opened in terms of bloggers around the world talking about vSphere 6.5!  Now that the product has finally been declared GA we can all go ahead and download it today!  Certainly the new HTML5 client, the addition of a more functional vCenter Server Appliance and built-in disk level encryption are enough to make me want to make the jump…eventually 🙂  That said there is a lot that is new with vCenter and ESXi – you can check it all out here.

Veeam Backup & Replication 9.5 is here!

veeamlogoAnd just as VMware releases it’s flagship hypervisor software to the masses Veeam follows behind one day later with the support and availability back it all up with their updated release of version 9.5 of Backup & Replication.  There is a lot that’s new within this release – check it all out here!  With Nimble support and a ton of integration with other Veeam products such as the Veeam Agents this release has a lot – but perhaps some of my favourite enhancements will be the ones that will probably not be written about the most and that’s all of the engine enhancements and things like the vSphere inventory cache.  As with most Veeam releases I’m sure this one will be well adopted!

Tech Field Day 12 – that’s a wrap!

tfd-logo-100x100I’ve just returned home from Tech Field Day 12 in beautiful San Jose!  I say beautiful but I was told it was cold by multiple people there – that said, I’m Canadian, and 18C in the middle of November is not my idea of cold 🙂  Either way I sat with 11 of my peers around the TFD table for a couple of days and got blasted by the fire hose.  There was some great presenting companies there; think Dell-EMC, DriveScale, Rubrik, Cohesity, StorageOS, Docker, and Igneous.  Expect more in this space about what these guys had to talk about but for now if you want to check out the videos you can do so – links are all over here.

Fancy some community recognition!

Ah, November – cooler air, the transition from Halloween to Christmas – and of course, a flurry of forms to fill out if you’d like to be included in any vendor/community recognition programs.  Most of these things require you to nominate yourself – so suck up any pride as you may have to get a little self absorbed while you try and brag yourself up.  Recognition programs are a funny thing – Some are great, some are so-so – I’m not going to go into the details of each.  If you want a great spot to see a list of them all Jim Jones has a great post outlining everything here.  And to get a glimpse into one of my favourties, the Veeam Vanguard program – check out Matt Crapes post here.  I’ve also been invited to the KEMP VIP program – which is new to me so expect to see more about that as well in the future.

Beetlejuice, Beetlejuice, Docker

docker-logo-300-71x60I feel like I can’t attend any presentation anymore without hearing the work Docker – honestly, containers and Docker is just popping up everywhere – with the oddest of technologies claiming they have support for Docker.  So, with all this my problem is, What the eff is Docker – I’ve never had a use-case to even start looking at containers within my day job – therefore, I don’t really have that much knowledge around the challenges and benefits of them.  After seeing them at TFD I can now say that I need to explore this further – and jump on this container bandwagon to learn what they are all about.  First stop, Mr Stephen Foskett’s blog where he tells us just “What the deal with containers are“.  If you are just learning, or just love containers – check it out!

Friday Shorts – #vDM30in30, AWS, Reading, Writing, AWS and more…

Would it kill you not to be so funny all the time? That’s all I’m askin’. This woman thinks I’m very funny and now you’re gonna be funny, so what am I gonna be? I’m gonna be a short bald guy with glasses who suddenly doesn’t seem so funny. – George Costanza

VMworld is over – vSphere 6.5 is near

VMware LogoWith VMworld EMEA sweeping up the remains of the solutions exchange the dawn of vSphere 6.5 is here!  And with that comes a multitude – and I mean a crap load of multitudes of blogs talking about all of their favourite features of the release to come!  With that said I can’t possible link to all of them that I have read – but what I will link to is to the linkmaster himself Eric Siebert!  He’s got a great roundup post here of all of the features within vSphere 6.5.  Now this post doesn’t have links, just quickly lists the features – but if you want to go deeper on a certain feature be sure to check out his ‘vSphere 6.5 Link-O-Rama’ page as well!

Renting vSphere on AWS

vmware_awsVMware and AWS announced a partnership – one which would allow vSphere to run on the Amazon cloud!  What this means for customers is that they will get the familiarity of vSphere and all it has to offer, but on a cloud-like scale, running on bare-metal within Amazon’s datacenters.  What’s in it for VMware – well, they get that global cloud they have been looking for, without having to actually build the datacenters behind it!  And Amazon – well, who knows where they will take it but I can’t see them complaining too much about being able to pitch certain services and products to those who chose to run VMware on AWS!  Anyways, Frank Denneman has a great article here taking a deeper look into everything – as well, a little bit of a preview on how the new service enables one pretty cool new feature – Elastic DRS.

Alastair teaches us to read!

Although this post is a couple months old I’m just getting around to sharing it now!  I wanted to be sure as I’m always interested in how everyone in this community stays current with so much going on all the time!  The amount of content getting created for and by the tech community is huge, and keeping track of it all is certainly becoming a skill!  I read a lot, a lot of blogs so any way of figuring out ways to streamline my content consumption always sparks my interest!  Alastair has already taken us through is process of creating content, but with this post he takes us through his process of consuming content!  This is a great post with excellent tips on how to stay current!

More than simply writing a file

There’s a guest blog post on Michael White’s blog written by Michael Cade outlining the various backup methods used within Veeam Backup & Replication and how each one of them affects the capacity and performance on your backup repositories.   If you are using VBR and you haven’t had an in-depth look at these backup methods I would suggest starting with this post – it does a great job at outlining just how each backup method works, what is actually contained within each VBR backup file type.  It’s not just simply copying data here 🙂

#VDM30in30 30 posts, 30 days!

Think you have what it takes to pump out 30 posts in 30 days – I can say that I certainly thought I did at one point in time, but quickly realized after just a few days that I most certainly did not!  But hey, kudos for trying right – and even though I have a lot going on this November I will most likely try again!  Don’t have a clue what I’m talking about – well, the folks from the Virtual Design Master competition are once again challenging the blogging community to crank out 30 posts in 30 days throughout the month of November!  Have a look at Eric Wright’s blog for a great description of what this is like!  Now I know some people may think that the quality of posts may not be there, or that the content won’t be relevant, but honestly, my blogging isn’t about perfecting every single thing I do (I’m sure that’s apparent).  It’s about sharing things I do, problems I run into, opinions I have – sometimes it may be relevant, sometimes it won’t – sometimes you may agree, sometimes you won’t – sometimes you may just not even understand why a post ends up on this blog!   To me writing is about passion, and whether that passion be virtualization, technology, canadian folk music, maple syrup, hunting, whatever, it doesn’t matter!   The point here is that when I write, I think –  and when I think I learn – and nobody can argue that learning is invaluable!  So heads up, you might see some odd posts here come this November!

Friday Shorts – #vDM, New Web Client, Linux Cleanup, Betas and more…

Is it flowing? I like flowing, cascading hair. Thick lustrous hair is very important to me. Let me ask you this. If you stick your hand in the hair is it easy to get it out?

George Costanza – Seinfeld

Virtual Design Master 4 looking for sponsors

vdmlogoIf you have never checked out the Virtual Design Master challenge I suggest you stop reading this and head over to their site and peruse the last 3 season, then com back here of course…. Anyways, the online, reality based challenge is back for Season 4 and they are looking for sponsors to help provide prizes, swag, infrastructure, etc for the upcoming season!  So if you work for a vendor and want to get your brand attached to VDM4, follow this link to indicate your interest!  They are looking to get everything firmed up to have a July/August competition.

New HTML5 vSphere Web Client!

VMware LogoWhy VMware feels the need to change how the lightening fast, crazy responsive, highly reliable vSphere Web Client that is currently out there is beyond me, but they are…  I hope you can detect the sarcasm in that last sentence.  Anyways, they have been hard at work (re)developing the vSphere Web Client, removing it’s reliance on flash and flex and providing the same functionality through code based on HTML5.  I’ve not yet had a chance to check this out, but from the reactions on the blogosphere and Twitter I’d say that they are on the right track!  They are releasing the vSphere Web Client 6.5 as a Fling, allowing the product to get out into everyone’s hands before it’s integrated into a vSphere version.  If you have a chance go and check it out – it’s simply a virtual appliance that integrates with your current environment.

Getting Linux ready for a vSphere Template!

tuxcloudFellow VFD4 delegate Larry Smith recently posted in regards to cleaning up your Ubuntu templates!  It’s a great post that covers off a lot of things than you can do to ensure you have a clean, prepped instance of Ubuntu to use a template within your vSphere environment.  That said, he takes it one step further, scripting out the complete cleanup in bash – and in Ansible.  If you deal with Linux/Ubuntu templates I would definitly recommend heading over to Larry’s blog and applying some of this scripty goodness.

vSphere.next – Beta Time!

VMware has announced that the next version of vSphere will enter a (limited) public beta. If you feel like you have the time and are ready to provide the effort in providing feedback, submitting bugs, etc to VMware in regards to the next release of vSphere then you can head here and indicate your interest in being a part of the beta.  As far as I know not everyone will be accepted – careful consideration will be taken on who is chosen to participate as they want to ensure they are getting valuable feedback and discovering any gotchya’s in the product before releasing it to the masses!

ZertoCon – The Premier Business Continuity Conference

ZertoCON_Ads_125x125Zerto has been a long time sponsor of this blog so I thought I’d place a shoutout to them and what they have in the works this spring!  You can join Zerto and many others from May 23-25 in beautiful Boston for ZertoCon.  Lately we have seen a lot of these smaller vendors opting to have their own conferences – and honestly if you use their products they are a must for you to go!  The VM/EMC Worlds are a great venue, but honestly, these smaller, laser focused conferences are absolutely fabulous if you are looking to gain more knowledge around certain vendors and their ecosystems!  I encourage you to check it out and sign up if you have the chance to go!

Setting up VVOLs on HP 3PAR

3par_arrayAs I’ve recently brought a HPE 3PAR 7200 into production with an ESXi 6.0 U2 cluster I thought what a better time than now to check out just how VVOLs are implemented by HPE.
Although the tasks to do so aren’t difficult by any means I find the documentation around doing so is a bit scattered in different KB’s and documents between VMware and HPE, especially if you have upgraded to their latest firmware (3.2.2 MU2).

Pre-reqs

As far as prerequisites go there really isn’t that many other than ensuring you are up to date on both your 3PAR firmware and ESXi versions.  For the 3PAR you will need to ensure you are running at the very least 3.2.1.  In terms of vSphere – 6.0 or higher.  Also don’t forget to check your HBA’s on the VMware HCL and ensure that they are actually  supported as well, and note the proper firmware/driver combinations recommenced by VMware.

After spending the day(s) updating firmware (ugh!) it’s finally time to get going.

Step 1 – Time

NTP is your friend for this step.  Before proceeding any further you need to ensure that all of your hosts, vCenter Server and 3PAR are all synced in terms of time.  If you have NTP setup and running then you are laughing here, but if you don’t, stop looking at VVOLs and set it up now!  It should be noted that the 3PAR and the VMware infrastructure can be set to different time zones, however they must still be synced in terms of time!

Step 2 – Can we see the protocol endpoint?

At this stage we should actually check our ESXi hosts to ensure we can see the protocol endpoint on the 3PAR.  To do so we will need to ensure that we see the same WWN after running a couple of different commands.  First, as shown below, the ‘showport’ command on our 3PAR.  Circled is the WWN of our 3PAR array.  Make note of this!

showport

With the WWN of our storage array in memory we can now head over to our ESXi hosts.  SSH in and run the ‘esxcli storage core device list –pe-only‘ command.  This command will return any Protocol Endpoints visible from the ESXi host.  If all goes well we should see the same WWN that we did with showport, and the ”Is VVOL PE’ flag set to true – as shown below

pe-only

As you can see, we have a match so at least we have some visibility from our hosts!

Step 3 – VASA

showvasaAs we all know the whole concept of VVOLs requires the array to support VASA 2.0 and act as a storage provider for vCenter – this is what allows us to create our VM profiles and have the array automatically provision VVOLs depending on what profile is selected.  On the 3PAR we can check the status of VASA by simply running the ‘showvasa’ command.  In the case shown we can see that it is already enabled and functioning properly, however this wasn’t always the case for me.  To enable the service I first tried the ‘startvasa’ command, however it was complaining about not having a certificate.  The easiest way, if you plan on using self-signed certificates to generate one is to simply run the ‘setvasa -reset’ command.  This will reset your VASA configuration and generate a self-signed cert.  After this you can simply run ‘startvasa’ to get everything up and running…

Step 4 – Create the storage container

Now if you are following the HPE VVOL integration guide you won’t see this step, mainly because it was created around the 3.2.1 firmware, which would have already had a default, and only one storage container created for you.  If you are running 3.2.2 though you have the option to create more than one storage container, and by default comes with, well, no storage containers.  So before we go and register our vCenter with the VASA provider we first need to create a storage container to host our VVOL datastore.  First, create a new Virtual Volume set with the following command

createvvset myvvolsetname

Then, let’s create our storage container and assign our newly created set to it

setvvolsc -create set:myvvolsetname

Again, these commands wouldn’t be required in 3.2.1 as far as I know, but are in 3.2.2

Step 5 – Register our VASA within vCenter

Now it’s time to head over the familiar, lightening fast interface we call the vSphere Web Client and register the 3PARs VASA implementation as a storage provider.  Make note of the ‘VASA_API2_URL’ shown in step 3 – you will need this when registering.  With your vCenter Server context selected, navigate to Manage->Storage Providers and click plus sign to add a new storage provider.

registerprovider

Enter your VASA URL from step 3, along with a name, username, and password and click ‘OK’.  For this instance I’ve used 3paradm, but you may be better off investigating creating a new account with just the ‘service’ role within the 3PAR.  Either way, get your new storage provider registered in vCenter and wait for the Status to show as Online and active.

Step 6 – The VVOL datastore

We are almost there I promise!  Before we can deploy VMs within a VVOL or assign storage profiles to match certain CPG’s within the 3par we need to have our VVOL datastore setup within vCenter.  I found the best spot to create this datastore is by right clicking the Cluster or ESXi host we want to have access to VVOLs and selecting Storage->New Datastore.  Instead of selecting VMFS or NFS as we normally would, select VVOL as the type as shown below

vvoltype

On the next screen simply give your datastore a name and select the storage container (this is what we made available in step 4).  Then, simply select the hosts you wish to have access to deploy VVOLs to and away you go!

Step 7 – Storage Profiles

At this point you could simply deploy VMs into your newly created VVOL datastore – the 3PAR will intelligently chose the best CPG to create the VVOL in, but the power really comes by being able to assign certain VM storage profiles to our disks, and having the VVOL go to the proper CPG depending on the array capabilities. Storage Profiles are created by clicking on the Home icon and navigating to Policies and Profiles within the web client.  In the VM Storage Profiles section simply click the ‘Create new storage profile’ button.  Give your new profile a name and continue on to the Rule-Set section.

profile

The Rule sets of my “Silver” VM storage profile are shown above.  As you can see, I’ve specified that I want this storage profile to place VM disks within my FastClass raid 5 CPG, and place their subsequent snapshots in the SSD tier CPG.  When you click next you will presented with a list of the compatible and incompatible storage.  Certainly select your compatible storage and click next.  Once we have all of the profiles we need we can simply assign them to our VMs disk as shown below…

vmprofile

As you can see I’ve selected our newly created “Silver” policy for our new VM.  What this states is that when this VM is created a new VVOL will be created on our FastClass disks on the 3PAR housing the VMs.

Step 8 – VVOL visibility

Although we are technically done with deploying VVOLs at this point I wanted to highlight the showvvvolvm command that we can utilize on the 3PAR in order to gain visibility into our VVOLs.  The first being simply listing out all of our VMs that reside on VVOLs within the 3PAR.

showvvolvm -sc

showvvol1

As you can see by the Num_vv column we have 3 VVOLs associated with our VM (MyNewVM).  But how do we get the information on those VVOLs individually – we can use the same command just with the -vv flag

showvvolvm -sc -vv

showvvol2

So now we can see that we have 1 VVOL dedicated for the config, 1 VVOL dedicated for the actual disk of the VM, and finally 1 VVOL hosting a snapshot that we have taken on the VM.

Anyways, that’s all I have for now – although I haven’t gone too deep into each step I hope this post helps someone along the way get their VVOLs deployed as I had a hard time finding all of this information in one spot.  For now I like what I see between HP and VMware concerning VVOLs – certainly they have a long road ahead of them in terms of adoption – we are still dealing with a 1.0 product from VMware here and there are a lot of things that need to be worked out concerning array based replication, VASA high availability, functionality without VASA, GUI integration, etc – but that will come with time.  Certainly VVOLs will change the way we manage our virtualized storage and I’m excited to see what happens – for now, it’s just fun to play with 🙂  Thanks for reading!

Quick Fix – Making your inactive NFS datastore active again!

Sticking to my rule of “If it happens more than once I’m blogging about it” I’m bringing you this quick post around an issue I’ve seen a few times in a certain environment.  Although this is solved by only a few esxcli commands I always find it easier for me to remember (and find) if I post it here… 🙂

Anyways, as it is I have a couple of NFS datastores that sometimes “act up” a bit in terms of their connections.  For the most part they are fine and dandy – however every now and then they show up within the vSphere client as “inactive” and ghosted.  After checking the network (I always try and pin things on the network) it appears that all the connections are fine – Host communicates with storage, storage with host – the same datastores are even functioning fine on other hosts.  Logically my next step is to remount them on the host in question but when trying to unmount and/or remount them through the vSphere client I usually end up with a “Filesystem busy” error.

Thankfully it doesn’t take a lot to fix this issue, but could certainly become tedious if you have many NFS datastores which you need to perform these commands on…

First up, list the NFS datastores you have mounted on the host with the following

esxcli storage nfs list

You should see that the ‘inactive’ datastores are indeed showing up with false under the accessible column.  Make note of the Volume Name, Share Name and Host as we will need this information for the next couple of commands….

Before we can add our datastore back we need to first get rid of it.  The following command takes care of that…

esxcli storage nfs remove -v DATASTORE_NAME

Depending on whether or not you have any VMs registered on the datastore and host you may get an error, you may not – I’ve found it varies 🙂  Anyways, lastly we simply need to mount the datastore back to our host using the following command…  Be sure to use the exact same values you gathered from the ‘nfs list’ command.

esxcli storage nfs add -H HOST -s ShareName/MountPoint -v DATASTORE_NAME

There you go!  You should now have a happy healthy baby NFS datastore back into your storage pool.  On a side note I’d love to see some sort of esxcli storage nfs remount -v DATASTORE_NAME command go into the command line in order to skip some of these steps – but, hey, for now I’ll just use three commands.

VCSA 6.0 prompting for a manual fsck

One of my VCSA deployments, the only one running 6.0 experienced a switch failure and in result a network outage of roughly 5 minutes the other day.  Not a big deal, but unfortunately this was a very “cost effective” solution and the switch that hosted the production network also hosted the VLANs carrying all of the NFS traffic to the datastores the VCSA resided on as well!  In short, VCSA done got grumpy – after fixing the issues with the switch I ended up at the screen shown below…

VCSAFileSystemError

Not an overly complicated error – just stating that we need to run a file system check on the /dev/mapper/log_vg-log volume manually.  In the past, say with 5.5 I’d just drop to a bash shell and do so – however the default appliance shell in the 6.0 version of VCSA presents a few challenges in doing the same thing.  First off, if I went ahead and gave the root password to the VCSA I was presented with the default menu – the same menu you would receive if you ssh’d to the box under normal circumstances – that said, in the maintenance mode, the shell.set and shell.enable commands don’t work.  So in order to get to a point where we can actually execute fsck we need to do a couple of things…

Grub to bash

So the first thing we need to do is get our VCSA booting into a bash prompt.  To do so, hit CTRL+D at the presented screen and get the box to reboot.  When the boot loader appears we will need to hit the space bar or up/down keys to stop the auto boot process.  Once stopped we can selected ‘p’ to unlock the menu and enter the root password for the box.  We then want to select “e” to edit our boot sequence –  highlight the second line, the one that displays the kernel parameters and select “e” once again.  At the end of that line we will want to append “init=/bin/bash” as shown below – this will boot our system into a bash shell.  Once done, hit “enter” to save and “b” to boot.

grubmenu1

grubmenu2

After the system has booted you should now be sitting at a bash prompt.  On a normal day we would simply run our fsck command here however the file system we are looking to check is still not mounted at this point.  I tried numerous commands and options to try and get it mounted but came up short.  That said running the following command and rebooting our vCenter will switch the login shell for root back to the ‘normal bash’ and allow us to continue

chsh -s /bin/bash root

Once the command has been run and the server rebooted we will be brought back to the same error prompting us to enter the root password.  Go ahead and do that.  This time we will be brought directly to a bash prompt with log_vg-log being available to us!  So, without further ado go ahead and run the following command to complete the file system check.

fsck /dev/mapper/log_vg-log

More than likely you will get numerous prompts asking you whether or not to fix any errors that occur.  Use your discretion here, however I didn’t have much of a choice and needed to say ‘Yes’ to all.  After it’s done give the VCSA another reboot and everything should come back up normally (at least it did for me).  Hopefully this helps push someone in the right direction if they are experiencing similar issues 🙂

Resizing the root partition of the vCenter Server Appliance (VCSA)

There are many side effects of a root file system filling up – server halts, unexpected application crashes, slowness, midnight wake up calls, etc.   And the root file system on the VCSA is no exception – in fact, I found it while trying to deploy a VM from a template into my environment – kept getting the dreaded 503 error that stated nothing useful to help with the resolution!  But, after a little bit of investigative work it appeared to me that the root file system on my VCSA was nearly full!  No keep in mind this was in my lab, and in all honesty you should probably investigate just why your file system is taking up so much space in the first place – but do to my impatience in getting my template deployed I decided to simply just grant a little more space to the root partition so it had a little room to breathe!  And below is the process I followed – may be right, may be wrong – but it worked!

outofspace

Step 1 – Make the disk bigger through the vSphere Client!

This is a no-brainer – if we don’t expand the space on the disk belonging to the VCSA that hosts the root partition before we can expand the root partition into that space!  So go ahead and log in to vCenter (or better yet the host on which your VCSA runs) and expand it’s underlying disk

biggerdisk

Once you have done this you may need to reboot your VCSA in order to get the newly expanded disk to show as expanded – I for one couldn’t find any solution that would rescan the disk within the VCSA to show the new space, but if you know, by all means let me know in the comments!!!

Step 2 – Rewrite the partition table

Things are about to get dicey here!  We are going to use fdisk in order to recreate the partition tables for the root filesystem – so relax, be careful and take your time!!!

First up, let’s have a look at our disk by running “fdisk –l /dev/sda”  As shown below we can see that it is no reporting at 25GB in size.

fdiskl

Next, we need to find the partition that our root filesystem resides on.  The picture of the “df-h” output at the beginning of this post confirms we are running on /dev/sda3 – this is the filesystem we will be working with…

So listed below is a slew of fdisk commands and options that we need to run – also, you can see my complete output shown at below….

First up, delete partition number 3 using the d option.

1
2
3
fdisk /dev/sda
d (for delete)
3 (for partition 3)

Now, let’s recreate the same partition with a new last sector – thankfully we don’t have to figure this out and should be fine utilizing all the defaults that fdisk provides…this time selecting the n option, p for partition, 3 for our partition number and accepting all of the defaults

1
2
3
n (for new)
p (for partition)
3 (for partition number 3)

After accepting all the defaults we need to make this partition bootable – again done inside of fdisk by using ‘a’ and then ‘3’ for our partition number

1
2
a (to toggle bootable flag)
3 (for partition number 3)

wholeprocess

As you can see in the message pictured above we need to perform a reboot in order for these newly created partition tables to take affect – so go ahead and reboot the VCSA.

Step 3 – Extend the filesystem

Well, the hard part is over and all we have left to do is resize the filesystem.  This is a relatively easy step executed using the resize2fs command shown below

resize2fs /dev/sda3

After this has complete a simple “df –h” should show that we now have the newly added space inside our root partition.

done

There may be other and better ways of doing this but this is the way I’ve chosen to go – honestly, it worked for me and I could now deploy my template so I’m happy!  Anytime you are using fdisk be very careful to not “mess” things up – take one of those VMware snapshotty thingies before cowboying around Smile  Thanks for reading!

Friday Shorts – #VMUG, nmcli, All flash VSAN, Altaro and more…

Why hello there – it’s been a while – It’s been a busy couple of months with work, conferences and home life and blogging has been put on the back burner for a bit.  I mean hey, I live in Canada and I need to get ready for the winter eh!  It’s a “Game of Thrones” winter around here!  Fear not over the past couple of months I’ve been doing some awesome things with Ravello, with a vSphere 6 upgrade, and some other awesome automation and orchestration stuff so I have a lot of posts filed under the idea category – so there is no lack of content to be written!  All that said for now let’s just have a look at some great community posts.

More advantage to the VMUG advantage

vmugVMUG Advantage has many benefits including free NFR software evals, discounted training, certification, and conference fees, discount codes for software and labs and more – but now we can add one more item to that list.  As of now VMUG is offering $600 of service credit with vCloud Air OnDemand.  I’ve reviewed vCloud Air OnDemand and can say that $600 is more than enough to get you in there and playing around for the year!  This is yet another great benefit to the VMUG Advantage program so if you haven’t bought it – do it!

Unexpected Signal: 11

VMware LogoDid you jump to get vSphere 5.5 Update 3 installed and running in your environment?  If so you might want to check out this VMware KB which outlines that the snapshot consolidation process may cause your VMs to fail with the above, well descripted, error message Smile  Sorry, nothing funny about if you are running any backup solution that may utilizing the VADP to free up disks for processing!   Anyways, downgrade, power off VMs and consolidate, or redeploy 5.5 are your resolution options for now!

Linux Networking through vRO

vmware-vcenter-orchestrator-vco-logo-150x150If you love vRO and automation and you don’t follow the vCOTeam blog then you should, do that first before continuing any further.  There, now that that’s out of the way have a look at this very detailed post in regards to configuring networking with Linux using nmcli, or better yet doing the whole thing through a vRO workflow – Awesome stuff!

All Flash VSAN in the homelab

tier-whatJason Langer (@jaslanger) has a great article about spinning (err flashing) up an All Flash VSAN setup in his homelab – showing you both the hard and the easy way this is a great guide for those looking to test out AF VSAN in their spare time (you know, when you aren’t building lego and what not Smile)

Rubrik and vRealize Orchestrator

rubrik_press_bg1Well, if you are a Rubrik customer and you are a vRO lover then I suggest you head over to Eric Shanks’ blog as he (and Nick Colyer) has a slew of blog posts related to vRO and Rubrik and how to do just about anything utilizing the API’s that Rubrik provides.

Speaking of backup – Altaro is now on the scene

altaro-vm-backup-500x257There’s a new player in the backup space when looking at protecting VMware virtual machines!  I had a chance to sit on the beta for the Altaro VMware backup and albeit I didn’t have a lot of time to check it all out I did get it installed and configured some backups and liked what I saw!  There have been a lot of community reviews of their software and first impressions are very positive – anyways, all the data protection junkies can check them out here.

Test driving vCloud Air On-Demand–Part 2

vmware-vcloud-air-virtual-private-cloud-ondemand-1-638In part 1 of my vCloud Air test drive we went through the vCloud Air UI as well as went over the steps it took to get a VM up and running in the cloud.  This is all great except for the fact that our VM had no connection to the internet – nor did we have any way of accessing our VM outside of the default console that vCloud Air provides.  This section will deal with just that – we will explore the NAT and firewall rules that need to be setup in order to get our VM access to the internet as well as port forward our public IP in order to provide ssh access into resources within vCloud Air.

Just a note – If you wanted to try out vCloud Air On-Demand on your own you can do so by following this url – and using the promo code Influencer2015.  This will get you $500 in service credits to burn in 90 days – more than enough credit to give it a valid test.  This code and url expires June 30, 2015 so be sure to register ASAP – also, it’s valid only for new MyVMware accounts meaning you will most likely need to register under a different email than you currently use.

Just as I did in the first post in this series, part 2 will have a video an an accompanying blog post.  The video, embedded below along with the blog post both accomplish the same result – so hopefully I’m covering off everyone’s content type of choice!

Connecting our cloud to the internet

As we seen in part 1 our VM was not connected to the internet by default.  Thankfully accomplishing this task is not that hard, even automated to a certain extent inside of vCloud Air (notice a recurring pattern here?), basically creating the NAT and firewall rules that we need in order to allow communication out.

Speaking of NAT let’s get a little of the vCloud Air terminology straight before we continue.   First up is our firewall – the vCloud Air firewall essentially is closed by default, meaning all traffic both in and out from the public IP is blocked by default.  In order to change this, we create what’s called Firewall Exceptions.  How vCloud Air interprets NAT is always based on the internal vCloud Air network, as follows

  • SNAT (Source NAT). – Deals with traffic originating from within the vCloud Air Network (source) destined for another network (ie Internet)
  • DNAT (Destination NAT) – Deals with traffic originating from another network (ie Internet) destined for the vCloud Air network (destination).

Now before we can get into any natting or firewalling we need to create a public ip address to nat out of.  This is done by browsing to your gateway configuration and then the ‘Public IPs’ tab.  The actual adding of the IP is done by clicking ‘Add IP Address’ – tricky huh?

Once we have our public IP setup we can do one of two things – we can create the SNAT and Firewall exceptions manually or we can right-click our VM and selected ‘Connect to Internet’  The latter option will automatically create the SNAT and Firewall exceptions that we need in order to allow outbound access from our newly created VM.

connecttointernet_thumb

Just a note here as well – I found it best to simply let the UI report back that everything has completed successfully – not just here but when doing things such as deleting VMs and creating data centers.  Sometimes navigating away from a page while a task was in process caused either the task to take and extremely long time or forced me to log back into the UI.  Anyways, after the task completes you should see the following rules that were created.

snatrulesforinternet_thumb[1]

firewallrulesforinternet_thumb[1]

Now if you are wondering how to manually create a firewall rule don’t worry because we are going  to do this as well.  Although the rules have been created to allow http/https/dns out of our network there is nothing created around ICMP or ping.  This is a commonly used method of testing connection to systems, so I’m not sure why this isn’t included in the ‘Connect to Internet’ workflow.  Either way it gives us the opportunity to go through the process.  Simply clicking the Add button will allow us to configure the following rule, which allows ping out not only from our VM (109.2) but our whole 109.0 network out to our external IP (shown below).

firewallforicmp_thumb[1]

At this point we are almost there in terms of access to the internet.  vCloud Air has statically assigned us and IP from our default IP pool, however it hasn’t done any configuration in regards to dns – so if you were to go try and ping google.ca at this point your VM would have no way of resolving it.  If you need to add some name servers to your Ubuntu VMs interface you can do so by running the following commands.

echo “dns-nameservers 8.8.8.8 8.8.4.4” >> /etc/network/interfaces

ifdown eth0 && ifup eth0

At this point, we should be successful in pinging google.ca or any other network address located on the internet – we have properly connected our cloud to the internet.

Connecting the Internet to our cloud.

Remember back in part 1 when I was griping a bit about not being able to send CTRL+ commands to my VM through the default console?  Well, one way around this might be to configure and allow ssh through our firewall, which would allow me to use putty or any other ssh client and issue CTRL+ to my hearts delight.  Keep in mind these scenarios would also work for Windows VMs and RDP by simply using port 3389 or whichever port you desire.

So, since our ssh traffic is going to be originating from the internet and destined to our vCloud network we first need to create a DNAT rule in order to port forward port 22 from our external IP to our internal Ubuntu server (Note:  The default Ubuntu image already is listening on port 22 for ssh).  The setup of the DNAT rule is shown below, remember to wait after clicking ‘Finish’ till vCloud Air reports success back.

dnatsetup4_thumb[1]

Even though we have our DNAT setup now we still need to allow access on port 22 on our external IP through our firewall – remember everything is blocked by default so the following firewall exception will need to be created.  I’m left the source IP and port as Any/Any, essentially allowing access from anywhere – if you had a specific IP that you would always be connecting from you could technically be a bit more secure and use that.  For my testing though, I don’t care so much…

firewallsshsetup_thumb[1]

And there you have it!  After waiting for the rules to apply (just wait) you should know be able to open up putty or your favorite ssh client, enter in your external (public) IP and log in to your VM.  Any other services and ports you want to open up?  Just simply repeat the following steps using whichever port you desire.

Although writing all this down after the fact seems pretty self explanatory and easy, to be honest, I struggled a bit during the networking portion.  Not to say it isn’t intuitive, but with everything else being a breeze within the vCloud Air UI I would’ve thought there would be some pre-built workflows around opening up services given the number of steps it takes.  Even if it was just common items such as ssh, rdp, or www.  That said, it is possible that maybe if I RTFM it might have been a bit easier – but I like to jump right in – helps me evaluate the usability.

All in all VMware has a great service in vCloud Air On-Demand.  It’s a piece that was originally missing from their cloud offerings.  Having a pay-as-you-go service where you don’t need to fork out long term commitments or budget is key, especially when you think in terms of timely workloads, dev/test, etc.  In the end vCloud Air has impressed me – a clean UI, easy to use solution without breaking the bank!

Again, if you want to test out vCloud Air On Demand for yourself go ahead and get a new MyVMware account and sign up at this URL using the promo code Influencer2015.  I know I’ve mentioned this a lot over the past two blog posts but it will get you $500 in service credits and that’s more than enough to get a solid judgment on the service.  Honestly, who doesn’t want free things!  Thanks for reading/watching.