There are many side effects of a root file system filling up – server halts, unexpected application crashes, slowness, midnight wake up calls, etc. And the root file system on the VCSA is no exception – in fact, I found it while trying to deploy a VM from a template into my environment – kept getting the dreaded 503 error that stated nothing useful to help with the resolution! But, after a little bit of investigative work it appeared to me that the root file system on my VCSA was nearly full! No keep in mind this was in my lab, and in all honesty you should probably investigate just why your file system is taking up so much space in the first place – but do to my impatience in getting my template deployed I decided to simply just grant a little more space to the root partition so it had a little room to breathe! And below is the process I followed – may be right, may be wrong – but it worked!
Step 1 – Make the disk bigger through the vSphere Client!
This is a no-brainer – if we don’t expand the space on the disk belonging to the VCSA that hosts the root partition before we can expand the root partition into that space! So go ahead and log in to vCenter (or better yet the host on which your VCSA runs) and expand it’s underlying disk
Once you have done this you may need to reboot your VCSA in order to get the newly expanded disk to show as expanded – I for one couldn’t find any solution that would rescan the disk within the VCSA to show the new space, but if you know, by all means let me know in the comments!!!
Step 2 – Rewrite the partition table
Things are about to get dicey here! We are going to use fdisk in order to recreate the partition tables for the root filesystem – so relax, be careful and take your time!!!
First up, let’s have a look at our disk by running “fdisk –l /dev/sda” As shown below we can see that it is no reporting at 25GB in size.
Next, we need to find the partition that our root filesystem resides on. The picture of the “df-h” output at the beginning of this post confirms we are running on /dev/sda3 – this is the filesystem we will be working with…
So listed below is a slew of fdisk commands and options that we need to run – also, you can see my complete output shown at below….
First up, delete partition number 3 using the d option.
1 2 3 |
fdisk /dev/sda d (for delete) 3 (for partition 3) |
Now, let’s recreate the same partition with a new last sector – thankfully we don’t have to figure this out and should be fine utilizing all the defaults that fdisk provides…this time selecting the n option, p for partition, 3 for our partition number and accepting all of the defaults
1 2 3 |
n (for new) p (for partition) 3 (for partition number 3) |
After accepting all the defaults we need to make this partition bootable – again done inside of fdisk by using ‘a’ and then ‘3’ for our partition number
1 2 |
a (to toggle bootable flag) 3 (for partition number 3) |
As you can see in the message pictured above we need to perform a reboot in order for these newly created partition tables to take affect – so go ahead and reboot the VCSA.
Step 3 – Extend the filesystem
Well, the hard part is over and all we have left to do is resize the filesystem. This is a relatively easy step executed using the resize2fs command shown below
resize2fs /dev/sda3
After this has complete a simple “df –h” should show that we now have the newly added space inside our root partition.
There may be other and better ways of doing this but this is the way I’ve chosen to go – honestly, it worked for me and I could now deploy my template so I’m happy! Anytime you are using fdisk be very careful to not “mess” things up – take one of those VMware snapshotty thingies before cowboying around Thanks for reading!
Many Thanks man, you saved my day, Many Thanks, Ufff….
It Worked for me too, Great Stuff…
Thank you, this solved my issue of not being able to update to the latest U3 for version 6
Mike, thanks for this article. Perfectly simple to follow and resolve this issue. At the end of Step 1 you mention you couldn’t find a command to refresh the disk space. I was able to run “vpxd_servicecfg storage lvm autogrow” and get it to refresh without rebooting. Hope that helps someone else. Thanks again!
I attempted this and now my vcenter server is dead. Just comes up with a 503 error and veeam can’t connect either.
503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x7f966405be30] _serverNamespace = /vsphere-client _isRedirect = false _port = 9090)
Same issue I am having. Did you find a fix?
After waiting a bit it should come up again. VM Web service can take time to init.
Perfect!!
Hi, with these commands you SHOULD be able to see the expanded disk without reboot
ls /sys/class/scsi_device/
echo 1 > /sys/class/scsi_device/0:0:0:0/device/rescan
partprobe -s
Thank you! This solved my issue updating to 6.0 U3
Thanks for your contribution!