Tag Archives: VMware

Quick Fix – Making your inactive NFS datastore active again!

Sticking to my rule of “If it happens more than once I’m blogging about it” I’m bringing you this quick post around an issue I’ve seen a few times in a certain environment.  Although this is solved by only a few esxcli commands I always find it easier for me to remember (and find) if I post it here… 🙂

Anyways, as it is I have a couple of NFS datastores that sometimes “act up” a bit in terms of their connections.  For the most part they are fine and dandy – however every now and then they show up within the vSphere client as “inactive” and ghosted.  After checking the network (I always try and pin things on the network) it appears that all the connections are fine – Host communicates with storage, storage with host – the same datastores are even functioning fine on other hosts.  Logically my next step is to remount them on the host in question but when trying to unmount and/or remount them through the vSphere client I usually end up with a “Filesystem busy” error.

Thankfully it doesn’t take a lot to fix this issue, but could certainly become tedious if you have many NFS datastores which you need to perform these commands on…

First up, list the NFS datastores you have mounted on the host with the following

esxcli storage nfs list

You should see that the ‘inactive’ datastores are indeed showing up with false under the accessible column.  Make note of the Volume Name, Share Name and Host as we will need this information for the next couple of commands….

Before we can add our datastore back we need to first get rid of it.  The following command takes care of that…

esxcli storage nfs remove -v DATASTORE_NAME

Depending on whether or not you have any VMs registered on the datastore and host you may get an error, you may not – I’ve found it varies 🙂  Anyways, lastly we simply need to mount the datastore back to our host using the following command…  Be sure to use the exact same values you gathered from the ‘nfs list’ command.

esxcli storage nfs add -H HOST -s ShareName/MountPoint -v DATASTORE_NAME

There you go!  You should now have a happy healthy baby NFS datastore back into your storage pool.  On a side note I’d love to see some sort of esxcli storage nfs remount -v DATASTORE_NAME command go into the command line in order to skip some of these steps – but, hey, for now I’ll just use three commands.

VCSA 6.0 prompting for a manual fsck

One of my VCSA deployments, the only one running 6.0 experienced a switch failure and in result a network outage of roughly 5 minutes the other day.  Not a big deal, but unfortunately this was a very “cost effective” solution and the switch that hosted the production network also hosted the VLANs carrying all of the NFS traffic to the datastores the VCSA resided on as well!  In short, VCSA done got grumpy – after fixing the issues with the switch I ended up at the screen shown below…

VCSAFileSystemError

Not an overly complicated error – just stating that we need to run a file system check on the /dev/mapper/log_vg-log volume manually.  In the past, say with 5.5 I’d just drop to a bash shell and do so – however the default appliance shell in the 6.0 version of VCSA presents a few challenges in doing the same thing.  First off, if I went ahead and gave the root password to the VCSA I was presented with the default menu – the same menu you would receive if you ssh’d to the box under normal circumstances – that said, in the maintenance mode, the shell.set and shell.enable commands don’t work.  So in order to get to a point where we can actually execute fsck we need to do a couple of things…

Grub to bash

So the first thing we need to do is get our VCSA booting into a bash prompt.  To do so, hit CTRL+D at the presented screen and get the box to reboot.  When the boot loader appears we will need to hit the space bar or up/down keys to stop the auto boot process.  Once stopped we can selected ‘p’ to unlock the menu and enter the root password for the box.  We then want to select “e” to edit our boot sequence –  highlight the second line, the one that displays the kernel parameters and select “e” once again.  At the end of that line we will want to append “init=/bin/bash” as shown below – this will boot our system into a bash shell.  Once done, hit “enter” to save and “b” to boot.

grubmenu1

grubmenu2

After the system has booted you should now be sitting at a bash prompt.  On a normal day we would simply run our fsck command here however the file system we are looking to check is still not mounted at this point.  I tried numerous commands and options to try and get it mounted but came up short.  That said running the following command and rebooting our vCenter will switch the login shell for root back to the ‘normal bash’ and allow us to continue

chsh -s /bin/bash root

Once the command has been run and the server rebooted we will be brought back to the same error prompting us to enter the root password.  Go ahead and do that.  This time we will be brought directly to a bash prompt with log_vg-log being available to us!  So, without further ado go ahead and run the following command to complete the file system check.

fsck /dev/mapper/log_vg-log

More than likely you will get numerous prompts asking you whether or not to fix any errors that occur.  Use your discretion here, however I didn’t have much of a choice and needed to say ‘Yes’ to all.  After it’s done give the VCSA another reboot and everything should come back up normally (at least it did for me).  Hopefully this helps push someone in the right direction if they are experiencing similar issues 🙂

Resizing the root partition of the vCenter Server Appliance (VCSA)

There are many side effects of a root file system filling up – server halts, unexpected application crashes, slowness, midnight wake up calls, etc.   And the root file system on the VCSA is no exception – in fact, I found it while trying to deploy a VM from a template into my environment – kept getting the dreaded 503 error that stated nothing useful to help with the resolution!  But, after a little bit of investigative work it appeared to me that the root file system on my VCSA was nearly full!  No keep in mind this was in my lab, and in all honesty you should probably investigate just why your file system is taking up so much space in the first place – but do to my impatience in getting my template deployed I decided to simply just grant a little more space to the root partition so it had a little room to breathe!  And below is the process I followed – may be right, may be wrong – but it worked!

outofspace

Step 1 – Make the disk bigger through the vSphere Client!

This is a no-brainer – if we don’t expand the space on the disk belonging to the VCSA that hosts the root partition before we can expand the root partition into that space!  So go ahead and log in to vCenter (or better yet the host on which your VCSA runs) and expand it’s underlying disk

biggerdisk

Once you have done this you may need to reboot your VCSA in order to get the newly expanded disk to show as expanded – I for one couldn’t find any solution that would rescan the disk within the VCSA to show the new space, but if you know, by all means let me know in the comments!!!

Step 2 – Rewrite the partition table

Things are about to get dicey here!  We are going to use fdisk in order to recreate the partition tables for the root filesystem – so relax, be careful and take your time!!!

First up, let’s have a look at our disk by running “fdisk –l /dev/sda”  As shown below we can see that it is no reporting at 25GB in size.

fdiskl

Next, we need to find the partition that our root filesystem resides on.  The picture of the “df-h” output at the beginning of this post confirms we are running on /dev/sda3 – this is the filesystem we will be working with…

So listed below is a slew of fdisk commands and options that we need to run – also, you can see my complete output shown at below….

First up, delete partition number 3 using the d option.

1
2
3
fdisk /dev/sda
d (for delete)
3 (for partition 3)

Now, let’s recreate the same partition with a new last sector – thankfully we don’t have to figure this out and should be fine utilizing all the defaults that fdisk provides…this time selecting the n option, p for partition, 3 for our partition number and accepting all of the defaults

1
2
3
n (for new)
p (for partition)
3 (for partition number 3)

After accepting all the defaults we need to make this partition bootable – again done inside of fdisk by using ‘a’ and then ‘3’ for our partition number

1
2
a (to toggle bootable flag)
3 (for partition number 3)

wholeprocess

As you can see in the message pictured above we need to perform a reboot in order for these newly created partition tables to take affect – so go ahead and reboot the VCSA.

Step 3 – Extend the filesystem

Well, the hard part is over and all we have left to do is resize the filesystem.  This is a relatively easy step executed using the resize2fs command shown below

resize2fs /dev/sda3

After this has complete a simple “df –h” should show that we now have the newly added space inside our root partition.

done

There may be other and better ways of doing this but this is the way I’ve chosen to go – honestly, it worked for me and I could now deploy my template so I’m happy!  Anytime you are using fdisk be very careful to not “mess” things up – take one of those VMware snapshotty thingies before cowboying around Smile  Thanks for reading!

Friday Shorts – #VMUG, nmcli, All flash VSAN, Altaro and more…

Why hello there – it’s been a while – It’s been a busy couple of months with work, conferences and home life and blogging has been put on the back burner for a bit.  I mean hey, I live in Canada and I need to get ready for the winter eh!  It’s a “Game of Thrones” winter around here!  Fear not over the past couple of months I’ve been doing some awesome things with Ravello, with a vSphere 6 upgrade, and some other awesome automation and orchestration stuff so I have a lot of posts filed under the idea category – so there is no lack of content to be written!  All that said for now let’s just have a look at some great community posts.

More advantage to the VMUG advantage

vmugVMUG Advantage has many benefits including free NFR software evals, discounted training, certification, and conference fees, discount codes for software and labs and more – but now we can add one more item to that list.  As of now VMUG is offering $600 of service credit with vCloud Air OnDemand.  I’ve reviewed vCloud Air OnDemand and can say that $600 is more than enough to get you in there and playing around for the year!  This is yet another great benefit to the VMUG Advantage program so if you haven’t bought it – do it!

Unexpected Signal: 11

VMware LogoDid you jump to get vSphere 5.5 Update 3 installed and running in your environment?  If so you might want to check out this VMware KB which outlines that the snapshot consolidation process may cause your VMs to fail with the above, well descripted, error message Smile  Sorry, nothing funny about if you are running any backup solution that may utilizing the VADP to free up disks for processing!   Anyways, downgrade, power off VMs and consolidate, or redeploy 5.5 are your resolution options for now!

Linux Networking through vRO

vmware-vcenter-orchestrator-vco-logo-150x150If you love vRO and automation and you don’t follow the vCOTeam blog then you should, do that first before continuing any further.  There, now that that’s out of the way have a look at this very detailed post in regards to configuring networking with Linux using nmcli, or better yet doing the whole thing through a vRO workflow – Awesome stuff!

All Flash VSAN in the homelab

tier-whatJason Langer (@jaslanger) has a great article about spinning (err flashing) up an All Flash VSAN setup in his homelab – showing you both the hard and the easy way this is a great guide for those looking to test out AF VSAN in their spare time (you know, when you aren’t building lego and what not Smile)

Rubrik and vRealize Orchestrator

rubrik_press_bg1Well, if you are a Rubrik customer and you are a vRO lover then I suggest you head over to Eric Shanks’ blog as he (and Nick Colyer) has a slew of blog posts related to vRO and Rubrik and how to do just about anything utilizing the API’s that Rubrik provides.

Speaking of backup – Altaro is now on the scene

altaro-vm-backup-500x257There’s a new player in the backup space when looking at protecting VMware virtual machines!  I had a chance to sit on the beta for the Altaro VMware backup and albeit I didn’t have a lot of time to check it all out I did get it installed and configured some backups and liked what I saw!  There have been a lot of community reviews of their software and first impressions are very positive – anyways, all the data protection junkies can check them out here.

Test driving vCloud Air On-Demand–Part 2

vmware-vcloud-air-virtual-private-cloud-ondemand-1-638In part 1 of my vCloud Air test drive we went through the vCloud Air UI as well as went over the steps it took to get a VM up and running in the cloud.  This is all great except for the fact that our VM had no connection to the internet – nor did we have any way of accessing our VM outside of the default console that vCloud Air provides.  This section will deal with just that – we will explore the NAT and firewall rules that need to be setup in order to get our VM access to the internet as well as port forward our public IP in order to provide ssh access into resources within vCloud Air.

Just a note – If you wanted to try out vCloud Air On-Demand on your own you can do so by following this url – and using the promo code Influencer2015.  This will get you $500 in service credits to burn in 90 days – more than enough credit to give it a valid test.  This code and url expires June 30, 2015 so be sure to register ASAP – also, it’s valid only for new MyVMware accounts meaning you will most likely need to register under a different email than you currently use.

Just as I did in the first post in this series, part 2 will have a video an an accompanying blog post.  The video, embedded below along with the blog post both accomplish the same result – so hopefully I’m covering off everyone’s content type of choice!

Connecting our cloud to the internet

As we seen in part 1 our VM was not connected to the internet by default.  Thankfully accomplishing this task is not that hard, even automated to a certain extent inside of vCloud Air (notice a recurring pattern here?), basically creating the NAT and firewall rules that we need in order to allow communication out.

Speaking of NAT let’s get a little of the vCloud Air terminology straight before we continue.   First up is our firewall – the vCloud Air firewall essentially is closed by default, meaning all traffic both in and out from the public IP is blocked by default.  In order to change this, we create what’s called Firewall Exceptions.  How vCloud Air interprets NAT is always based on the internal vCloud Air network, as follows

  • SNAT (Source NAT). – Deals with traffic originating from within the vCloud Air Network (source) destined for another network (ie Internet)
  • DNAT (Destination NAT) – Deals with traffic originating from another network (ie Internet) destined for the vCloud Air network (destination).

Now before we can get into any natting or firewalling we need to create a public ip address to nat out of.  This is done by browsing to your gateway configuration and then the ‘Public IPs’ tab.  The actual adding of the IP is done by clicking ‘Add IP Address’ – tricky huh?

Once we have our public IP setup we can do one of two things – we can create the SNAT and Firewall exceptions manually or we can right-click our VM and selected ‘Connect to Internet’  The latter option will automatically create the SNAT and Firewall exceptions that we need in order to allow outbound access from our newly created VM.

connecttointernet_thumb

Just a note here as well – I found it best to simply let the UI report back that everything has completed successfully – not just here but when doing things such as deleting VMs and creating data centers.  Sometimes navigating away from a page while a task was in process caused either the task to take and extremely long time or forced me to log back into the UI.  Anyways, after the task completes you should see the following rules that were created.

snatrulesforinternet_thumb[1]

firewallrulesforinternet_thumb[1]

Now if you are wondering how to manually create a firewall rule don’t worry because we are going  to do this as well.  Although the rules have been created to allow http/https/dns out of our network there is nothing created around ICMP or ping.  This is a commonly used method of testing connection to systems, so I’m not sure why this isn’t included in the ‘Connect to Internet’ workflow.  Either way it gives us the opportunity to go through the process.  Simply clicking the Add button will allow us to configure the following rule, which allows ping out not only from our VM (109.2) but our whole 109.0 network out to our external IP (shown below).

firewallforicmp_thumb[1]

At this point we are almost there in terms of access to the internet.  vCloud Air has statically assigned us and IP from our default IP pool, however it hasn’t done any configuration in regards to dns – so if you were to go try and ping google.ca at this point your VM would have no way of resolving it.  If you need to add some name servers to your Ubuntu VMs interface you can do so by running the following commands.

echo “dns-nameservers 8.8.8.8 8.8.4.4” >> /etc/network/interfaces

ifdown eth0 && ifup eth0

At this point, we should be successful in pinging google.ca or any other network address located on the internet – we have properly connected our cloud to the internet.

Connecting the Internet to our cloud.

Remember back in part 1 when I was griping a bit about not being able to send CTRL+ commands to my VM through the default console?  Well, one way around this might be to configure and allow ssh through our firewall, which would allow me to use putty or any other ssh client and issue CTRL+ to my hearts delight.  Keep in mind these scenarios would also work for Windows VMs and RDP by simply using port 3389 or whichever port you desire.

So, since our ssh traffic is going to be originating from the internet and destined to our vCloud network we first need to create a DNAT rule in order to port forward port 22 from our external IP to our internal Ubuntu server (Note:  The default Ubuntu image already is listening on port 22 for ssh).  The setup of the DNAT rule is shown below, remember to wait after clicking ‘Finish’ till vCloud Air reports success back.

dnatsetup4_thumb[1]

Even though we have our DNAT setup now we still need to allow access on port 22 on our external IP through our firewall – remember everything is blocked by default so the following firewall exception will need to be created.  I’m left the source IP and port as Any/Any, essentially allowing access from anywhere – if you had a specific IP that you would always be connecting from you could technically be a bit more secure and use that.  For my testing though, I don’t care so much…

firewallsshsetup_thumb[1]

And there you have it!  After waiting for the rules to apply (just wait) you should know be able to open up putty or your favorite ssh client, enter in your external (public) IP and log in to your VM.  Any other services and ports you want to open up?  Just simply repeat the following steps using whichever port you desire.

Although writing all this down after the fact seems pretty self explanatory and easy, to be honest, I struggled a bit during the networking portion.  Not to say it isn’t intuitive, but with everything else being a breeze within the vCloud Air UI I would’ve thought there would be some pre-built workflows around opening up services given the number of steps it takes.  Even if it was just common items such as ssh, rdp, or www.  That said, it is possible that maybe if I RTFM it might have been a bit easier – but I like to jump right in – helps me evaluate the usability.

All in all VMware has a great service in vCloud Air On-Demand.  It’s a piece that was originally missing from their cloud offerings.  Having a pay-as-you-go service where you don’t need to fork out long term commitments or budget is key, especially when you think in terms of timely workloads, dev/test, etc.  In the end vCloud Air has impressed me – a clean UI, easy to use solution without breaking the bank!

Again, if you want to test out vCloud Air On Demand for yourself go ahead and get a new MyVMware account and sign up at this URL using the promo code Influencer2015.  I know I’ve mentioned this a lot over the past two blog posts but it will get you $500 in service credits and that’s more than enough to get a solid judgment on the service.  Honestly, who doesn’t want free things!  Thanks for reading/watching.

Test driving vCloud Air On-Demand–Part 1

vmware-vcloud-air-virtual-private-cloud-ondemand-1-638

As a vExpert I tend to get a number of opportunities to evaluate different pieces of software and platforms – and as much as I’d like to simply look at every one I just don’t have the time to do so.  That said, when the vCloud team reached out with an offer to have a go at their vCloud Air On-Demand service I rearranged some of my priorities – partly because cloud is interesting to me, but mostly because they also gave me the chance to let my readers have the same opportunity!  VMware offers everyone $300 in service credits to evaluate vCloud Air On-Demand, but they gave me an extra $200 – and the promotional code to give you guys the same!  So, if you register at using this exact link – and use the promo code Influencer2015, you too can have a total of $500 in service credits to play with.  Just a note – you have 90 days to use up your credits before they expire – oh, and you need to register before June 30th, 2015 – so hurry!  Another caveat, this offer is valid for NEW MyVMware accounts only – so, ummm, uh, yeah, find another email to register with Smile

On to the evaluation

So I’ve recorded a couple of videos in regards to what I’ve done inside of vCloud Air, the first one, attached just below this paragraph takes us through a little tour of the vCloud Air web UI, and shows us the steps to get our first VM up and running.

 

Now if you don’t feel like listening to my Canadian accenty, cold-infested, whispering (I had a house full of sleeping kids) voice I’ve written the process down as well.  Hey, we all learn in different ways right – some people like videos and others can’t stand them – so here’s both.

Judging a book by its’ cover

A simple, clean interface can go a long way when it comes to peoples reaction and opinions on the software that they use.  The vCloud Air team certainly kept this in mind when developing the UI supporting their on-demand service.  It’s very clean – showing only the basic information that one would really need to see to get a handle on their virtual data centers and VMs.  If you have ever used vCloud Director (vCD) you know just how many different tabs and options are available within VMware’s cloud offering – there are a ton of them, and I find the vCD interface cumbersome and hard to use.  It’s nice to see that VMware has taken some of the basic functionality that vCD provides, and abstracted it away to the vCloud Air UI – allowing their customers to perform common tasks such as power operations, network setup, and VM creation/snapshotting without having to ever set foot inside of vCD.

Let’s Cloud Bro!

Let’s get to it!  The first step after logging into the vCloud Air portal is to create a virtual datacenter.  Before we do that though we have to determine exactly what region we want to work in.  As shown below we have some options as to where we would like our virtual datacenter to be located – I’ve chosen Virginia for some of my testing – but if you are following along, chose one close to you.

vd

To create our Virtual Data Center select the + icon next to the Virtual Data Centers label.  As you can see there isn’t a whole lot of configuration required in this step, simply a name.  Also you can see that each VDC allows for 50 VMs containing 130 GHz CPU, 100GB of RAM and 2TB of both SSD accelerated storage and standard storage.

createVDC_thumb[1]

At this point automation kicks in and our virtual data center is created.  Once it’s complete we can see that a number of components will be created and configured by default for us.  Selecting our VDC from the left hand menu and clicking on the ‘Networks’ tab we can see a number of these pre-configured items such as our public gateway IP address, the default gateway IP for our internal network, as well as the IP range that will be handed out to VMs within our VDC.  We can also create new networks from directly within the vCloud Air UI, however if you need to delve a little deeper into the services offered you can do so by using the ‘Manage in vCloud Director’ link in the top right hand corner.  This will open an already authenticated vCloud Director session where you can manage your networks and add services such as DHCP, load balancing, etc. Essentially all of the functionality that you would normally have when running a full instance of vCD.

networkstab_thumb[1][1]

In order to create firewall rules, nat rules, and assign an accessible public IP to our gateway we need to select our default gateway under the ‘Gateways’ tab.  Again, we can break out into a vCloud Director window here as well.   We will come back to this section in part 2 of this series to connect our VM to the internet and grant ssh access but for now its just good to know where this information is located.

gateway_thumb[1][1]

Speaking of VMs let’s get on with the show here and get our first VM created.  This is done on the ‘Virtual Machines’ tab (Use the giant “Create your first virtual machine” button).  When creating a VM you can select from the catalog which has been provided by VMware, or by creating a catalog, uploading and ISO and creating your VM from scratch.  For the sake of this evaluation I just used the 32 bit Ubuntu server provided by VMware.

createVM_thumb[1][1]

After selecting your VM from the catalog you can then name it and customize the cpu/memory/storage to your choosing.  vCloud Air will default these settings to their preferred amounts but you can change them using their respective sliders.  What’s nice about his screen is that you can see how s simple CPU, RAM and Storage change can affect your price per hour.  In my case, this Ubuntu VM with 1 CPU, 2GB RAM and 10 GB of accelerated storage is a mere 5 cents/hour – not bad

changeVMHardware_thumb[1][1]

Once the VM has been created it should now be listed under the Virtual Machines tab.  Right-clicking the VM will bring up a context menu showing all the actions available, including power options, console access, snapshotting, etc..

vmoptions_thumb[1][1]

Clicking on the VMs name within our list will also bring us into more details in regards to that VM.  The ‘Resource Usage’ tab showing estimated costs, ‘Settings’ tab showing various configurable items, and the ‘Networks’ tab showing the networking information for the VM.  As shown below we can see that our new Ubuntu VM has claimed the first address within our IP pool – 192.168.109.2.

vmnetwork_thumb[1][1]

Another important note about the ‘Settings’ tab is the ‘Guest OS Password’ section.   In order to login to our newly created VM we will need the root password.  This can be revealed by clicking ‘show initial password’.  By default, all the VMs from the default catalog provided by vCloud Air will prompt you to change the default password after first login.  Let’s make note of this password and go ahead and open a console to change it.

changepassword_thumb[1][1]

As we can see below the console provided by the vCloud Air UI is pretty barebones – allowing us to simply provide input to the VM and a button to send CTRL+ALT+DEL to the VM.  I found this a little frustrating at times, especially since I was using a Linux VM.  There were times where I had to direct a CTRL+C command to the VM but had no way of doing so, instead I had to proceed with a complete reboot of the VM.  An on-screen keyboard may be a better solution here.

keyboard

At this point we are done with part 1 of my test drive.  My goal here was simply to get a VM up and running and we’ve certainly accomplished that.  So far my opinion around vCloud Air On-Demand is a good one – Aside from a little hiccup of trying to send CTRL+ commands to the VM through the built-in console everything else has been a breeze.  I really like the UI – how they have taken some of the complexity involved with trying to certain tasks within vCD and provided a one-click, automated solution without ever having to touch vCD – yet still giving users the option to move into vCD if needed.  In part 2 we will have a look at setting up some of the networking and firewalling in our virtual data center – things will get a bit more complicated as we explore the NAT and firewall rules inside our gateway.

If you have any experience or thoughts about vCloud Air I’d love to hear them – leave a comment below or find me on twitter.  And as mentioned before if you wanted to evaluate vCloud Air On-Demand yourself go ahead and register here, using the Influencer2015 promotional code to get yourself $500.00 in service credits.

Don’t forget to read Test Driving vCloud Air On-Demand Part 2