Tag Archives: vSphere 5
First off I want to thank Tom Verhaeg (blog/twitter) for providing this scenario. Tom had gotten in contact with myself and wanted to do what he can to help our with the 8 weeks of #VCAP series as he is going through a similar type process as me in studying for the VCAP5-DCA. So props to Tom for taking the time and initiative to give back. Hopefully we see more from him in the coming weeks! Even better for myself as I can run through some scenarios that I didn't make up 🙂 Be sure to follow Tom on Twitter and check out his blog Thanks for the help Tom!!!
Your company leverages the full Enterprise Plus licensing and has set up a Distributed vSwitch. Recently, the number of ports needed on a particular portgroup exceeded the number configured. You are tasked with creating a new Portgroup, called DvS_ProductionNetwork which only connects the running VM’s and also functions when vCenter is down.
Off we go again. So, let’s recall. There are 3 different options of port binding on a DvS.
Static binding – Which creates a port group with a manual set number of ports. A port is assigned whenever a vNIC is added to a VM. You can connect a vNIC static binding only through vCenter.
Dynamic binding (Deprecated in vSphere 5.0!) – A port is assigned to a vNIC when the VM is powered on, and it’s vNIC is in a connected state. You can connect this dynamic binding only through vCenter.
Empheral binding – A port is assigned to a vNIC when the VM is powered on, and it’s vNIC is in a connected state. This binding method allows the bypass of vCenter, allowing you to manage virtual machine networking when vCenter is down.
So, that’s the one we need! Empheral binding! Luckily, it’s quite simple to configure. Hop over to the networking inventory (Ctrl + Shift + N) and create the new port group. Give it a name and leave the number of ports on the default of 128.
Now edit the settings of this port group, and select the Empheral binding under the port binding dropdown. Also note, that the number of ports is greyed out now.
Hopefully you all enjoyed the last scenario based post because you are about to get another one 🙂 Kind of a different take on covering the remaining skills from the storage section, section 1. So, here we go!
A coworker has come to you complaining that every time he performs storage related functions from within the vSphere client, VMware kicks off these long running rescan operations. He's downright sick of seeing them and wants them to stop, saying he will rescan when he feels the need to, rather than having vSphere decide when to do it. Make it happen!
So, quite the guy your coworker, thinking he's smarter than the inner workings of vSphere but luckily we have a way we can help him. And also the functions we are going to perform are also part of the VCAP blueprint as well – coincidence? Either way, the answer to our coworkers prayers is something called vCenter Server storage filters and there are 4 of them, explained below…
RDM Filter (config.vpxd.filter.rdmFilter) – filters out LUNs that are already mapped as an RDM
VMFS Filter (config.vpxd.filter.vmfsFilter) – filters out LUNs that are already used as a VMFS datastore
Same Hosts and Transports Filter (config.vpxd.filter.sameHostsAndTransporstFilter) – Filters out LUNS that cannot be used as a datastore extent
Host Rescan Filter (config.vpxd.filter.hostRescanFilter) – Automatically rescans storage adapters after storage-related management functions are performed.
As you might of concluded it's the Host Rescan Filter that we will need to setup. Also, you may have concluded that these are advanced vCenter Server settings, judging by the config.vpxd prefixes. What is conclusive is that all of these settings are enabled by default – so if we need to disable one, such as the Host Rescan Filter, we will need to set the corresponding key to false. Another funny thing is that we won't see these setup by default. Basically they are silently enabled. Anyways, let's get on to solving our coworkers issue.
Head into the advanced settings of vCenter Server (Home-vCenter Server Settings->Advanced Options). From here, disabling the host rescan filter is as easy as adding the config.vpxd.filter.hostRescanFilter and false values to the text boxes near the bottom of the screen and clicking 'Add' – see below
You work for the mayors office in the largest city in Canada. The mayor himself has told you that he installed some SSD into a host last night and it is showing as mpx.vmhba1:C0:T0:L0 – but not being picked up as SSD! You mention that you think that is simply SAS disks but he persists it isn't (what is this guy on crack :)). Either way, you are asked if there is anything you can do to somehow 'trick' vSphere into thinking that this is in fact an SSD.
Ok, so this one isn't that bad really, a whole lot of words for one task. Although most SSD devices will be tagged as SSD by default there are times when they aren't. Obviously this datastore isn't an SSD device, but the thing is we can tag it as SSD if we want to. To start, we need to find the identifier of the device we wish to tag. This time I'm going to run esxcfg-scsidevs to do so (with -c to show a compact display).
From there I'll grab the UUID of the device I wish to tag, in my case mpx.vmhba1:C0:T0:L0 – (crazy Rob Ford). Now if I have a look at that device with the esxcli command I can see that it is most certainly not ssd.
esxcli storage core device list -d mpx.vmhba1:C0:T0:L0
esxcli storage nmp device list -d mpx.vmhba1:C0:T0:L0
esxcli storage nmp satp rule add -s VMW_SATP_LOCAL -d mpx.vmhba1:C0:T0:L0 -o enable_ssd
And from here we need to reclaim the device
esxcli storage core claiming reclaim -d mpx.vmhba1:C0:T0:L0
And, another look at our listing out of the device should now show us that we are dealing with a device that is SSD.
esxcli storage core device list -d mpx.vmhba1:C0:T0:L0
And that's all for now 🙂
So my 8 weeks of #VCAP is quickly turning into just under 4 weeks of #VCAP so as I attempt to learn and practice everything on the blueprint you might find that I'm jumping around quite a bit. Also, I thought I would try presenting myself with a scenario with this post. Now all of the prep for the scenario is made by myself, therefore it's a pretty simple thing for me to solve, but none the less it will help get me into the act of reading a scenario and performing the tasks that are on it. So, this post will cover a bunch of random storage skills listed in Objective 1 of the blueprint – without ado, the scenario
Let's say we've been tasked with the following. We have an iSCSI datastore (iSCSI2) which utlizes iSCSI port bonding to provide multiple paths to our array. We want to change the default PSP for iSCSI2 from mru to fixed, and set the preferred path to travel down CO:T1:L0 – only one problem, C0:T1:L0 doesn't seem to be available at the moment. Fix the issues with C0:T1:L0 and change the PSP on iSCSI2 and set the preferred path.
Alright, so to start this one off let's have a look first why we can't see that second path to our datastore. If browsing through the GUI you aren't even seeing the path at all, the first place I would look at is claimrules (now how did I know that 🙂 ) and make sure that the path isn't masked away – remember the LUN Masking section. So ssh on into your host and run the following command.
esxcli storage core claimrule list
As you can see from my output lun masking is most certainly the cause of why we can't see the path. Rule 5001 loads the MASK_PATH plugin on the exact path that is in question. So, do you remember from the LUN Masking post how we get rid of it? If not, we are going to go ahead and do it here again.
First step, we need to remove that rule. That's done using the following command.
esxcli storage core claimrule remove -r 5001
Now that its gone we can load that current list into runtime with the following command
esxcli storage core claimrule load
But we aren't done yet! Instead of waiting for the next reclaim to happen or the next reboot, let's go ahead and unclaim that path from the MASK_PATH plugin. Again, we use esxcli to do so
esxcli storage core claiming unclaim -t location -A vmhba33 -C 0 -T 1 -L 0
And rescan that hba in question – why not just do it via command line since we are already there…
And voila – flip back into your Manage Paths section of iSCSI2 and you should see both paths are now available. Now we can move on to the next task, which is switching the PSP on iSCSI2 from MRU to Fixed. Now we will be doing this a bit later via the command line, and if you went into the GUI to check your path status, and since we are only doing it on one LUN we probably can get away with simply changing this via the vSphere Client. Honestly, it's all about just selecting a dropdown at this point – see below.
I circled the 'Change' button on this screenshot because it's pretty easy to simply select from the drop down and go and hit close. Nothing will happen until you actually press 'Change' so don't forget that. Also, remember, PSP is done on a per-host basis. So if you have more than one host and the VCAP didn't specify to do it on only one host, you will have to go and duplicate everything you did on the other host. Oh, and setting the preferred path is as easy as right-clicking the desired path and marking it as preferred. And, this scenario is completed!
The storage team thanks you very much for doing that but requirements have changed and they now wish for all of the iSCSI datastores, both current and any newly added datastores, to utilize the Round Robin PSP. How real life is that, people changing their mind 🙂
No problem you might say! We can simply change the PSP on each and every iSCSI datastore – not a big deal, there's only three of them. Well, you could do this, but the question specifically mentions that we need to have the PSP set to Round Robin on all newly added iSCSI datastores as well, so there's a bit of command line work we have to do. And, since we used the vSphere Client to set the PSP in the last scenario, we'll do it via command line in this one.
First up, let's switch over our existing iSCSI datastores (iSCSI1, iSCSI2, iSCSI3). To do this we will need their identifier which we can get from the GUI, however since we are doing the work inside the CLI, why not utilize it to do the mappings. To have a look at identifiers and their corresponding datastore names we can run the following
As you can see there are three datastores we will be targeting here. The identifier that we need will be the first string field listed beginning with t10 and ending with :1 (although we don't need the :1). Once you have the string identifier of the device we want to alter we can change its' PSP with the following command.
esxcli storage nmp device set -d t10.FreeBSD_iSCSI_Disk______000c299f1aec010_________________ -P VMW_PSP_RR
So, just do this three times, once for each datastore. Now, to handle any newly added datastores to defaulr to round robin we need to first figure out what SATP the iSCSI datastores are utilizing, then associate the VMW_PSP_RR PSP to it. We can use the following command to see which SATP is associated with our devices.
esxcli storage nmp device list
As you can see, our iSCSI datastores are being claimed by the VMW_SATP_DEFAULT_AA SATP. So, our next step would be to associate the VMW_PSP_RR PSP with this SATP – I know, crazy acronyms! To do that we can use the following command.
esxcli storage nmp satp set -s VMW_SATP_DEFAULT_AA -P VMW_PSP_RR
This command will ensure that any newly added iSCSI datastores claimed by the default AA SATP will get the round robin PSP.
At this point we are done this scenario but while I was doing this I realized there might be a quicker way to to change those PSP's on our existing LUNs. If we set associate our SATP with our PSP first then we can simply utilized the following command on each of our datastores to force them to change their PSP back to default (which will be RR since we just changed it).
esxcli storage nmp device set -d t10.FreeBSD_iSCSI_Disk______000c299f1aec010_________________ -E
Of course we have to run this on each datastore as well – oh, and on every host 😉
Big Joe, your coworker just finished reading a ton of vSphere related material because his poor little SQL server on his iSCSI datastore just isn't cutting it in terms of performance. He read some best practices which stated that the max IOPs for the Round Robin policy should be changed to 1. He requested that you do so for his datastore (iSCSI1). The storage team has given you the go ahead but said not to touch any of the other datastores or your fired.
Nice, so there is really only one thing to do in this scenario – change our default max IOPs setting for the SCSI1 device. So, first off, let's get our identifier for SCSI1
Once we have our identifier we can take a look on the roundrobin settings for that device with the following command
esxcli storage nmp psp roundrobin deviceconfig get -d t10.FreeBSD_iSCSI_Disk______000c299f1aec000_________________
As we can see, the IOOperation Limit is 1000, meaning it will send 1000 IOPs down each path before switching to the next. The storage team is pretty adamant we switch this to 1, so let's go ahead and do that with the following command.
esxcli storage nmp psp roundrobin deviceconfig set -d t10.FreeBSD_iSCSI_Disk______000c299f1aec000_________________ -t iops -I 1
Basically what we define with the above command is that we will change that 1000 to 1, and specify that the type of switching we will use is iops (-t). This could also be set with a -t bytes and entering the number of bytes to send before switching.
So, that's basically it for this post! Let me know if you like the scenario based posts over me just rambling on about how to do a certain task! I've still got lots more to cover so I'd rather put it out there in a format that you all prefer! Use the comments box below! Good Luck!
My plan is to go over all the skills in Objective 1.3 but before we get into PSA commands and what not let's first configure iSCSI port bonding – this way we will have a datastore with multiple paths that we can fiddle around with 🙂
First off iSCSI port binding basically takes two separate paths to an iSCSI target (the paths are defined by vmkernel ports) and bonds them together. So, we need two vmkernel ports. They can be on the same switch or separate switches, but the key is that you can only have one network adapter assigned to it. Meaning the vSwitch can contain multiple nics, but you need to ensure that the config is overridden on the vmkernel level to only have one NIC active. Let's have a look at this. Below you will see the current setup of my vmkernel ports (IPStore1 and IPStore2).
As you can see, my configuration here is actually wrong and needs to be adjusted – remember, one nic per vmkernel port. So, with a little click magic we can turn it into what you see below.
Basically, for IPStore1 I have overridden the default switch config on the vmkernel port, setting vmnic0 as active and vmnic1 as unused. For IPStore2 we will do the same except the opposite (hehe, nice, that makes no sense) – basically, override but this time set vmnic1 as active and vmnic0 as unused. This way we are left with two vmkernel ports, each utilizing a different NIC.
Now that we have the requirements setup and configured we can go ahead and get started on bonding the vmkernel ports together. This is not a hard thing to do! What we are going to want to do is right-click on our software iSCSI initiator and select 'Properties'. From there we can browse to the 'Network Configuration' tab and simply click 'Add'. We should now see something similar to below.
As you can see above, our VMkernel adapters are listed. If they weren't, that would indicate that they are not compatible to be bonded, meaning we haven't met the requirements outlined earlier. By selecting IPStore1 and then going back in and selecting IPStore2 ( I know, you can't do it at the same time 🙂 ), then selecting OK, then performing the recommended rescan you will have completed the task. We can now see that below inside of our 'Manage Paths' section for a datastore that has been mounted with our iSCSI initiator we have some nifty multipath options. First, we have an additional channel and path listed, as well, we are able to switch our PSP to thinks like Round Robin!
And kapow! That's it! We are done! In the next post we will look at how to perform some PSP/PSA related commands against this bad boy!
As some of you may now for the past, what feels like years but is probably closer to 6 months or so I have been working on a book project revolving around troubleshooting storage in a vSphere environment. At last I'm happy to say that the book is finally published and sitting on a variety of websites (Packt, Amazon) waiting to be purchased and consumed by you 🙂 ! The book, cleverly titled 'Troubleshooting vSphere Storage' is 150 pages straight to the point exercises that a vSphere admin can take when dealing with storage visibility, contention, and capacity issues.
Early on when I was pondering the idea of doing this I had no idea about the amount of work and time commitment that writing a book would consume! I most certainly have a new found respect for the rock stars that are putting out 500 page books out there! It really takes a major commitment from the authors, reviewers, and editors to get everything done! Speaking of reviewers, my technical reviewers, Angelo Luciani ( blog / twitter ), Jason Langer ( blog / twitter ), and Eric Wright ( blog / twitter ) were key to me actually finishing this project. Their feedback was awesome and without it, well, who knows what state the book would be in. So a big thanks goes out to them for all their help!
Needless to say I'm pretty excited to have a published piece of work out there – and if it helps just one person, well, then I guess I've done what I set out to do 🙂
Alright, continuing on the realm of security let's have a look at the built in firewall on ESXi. This post will relate directly to Objective 7.2 on the blueprint! Basically, a lot of this work can be done in either the GUI or the CLI, so chose what you are most comfortable with. I'll be jumping back and forth from both! Some things are just easier in the GUI I find….anyways, I only have like 4 weeks to go so let's get going…
First up, enable/disable pre configured services
Easy/Peasy! Hit up the 'Security Profile' on a hosts configuration tab and select 'Properties' in the 'Services' section. You should see something similar to that of below
I guess as far as enabling/disabling you would simply stop the service and set it to manual automation.
Speaking of automation, that's the second skill
As you can see above we have a few options in regards to automation behavior. We can Start/Stop with the host (basically on startup and shutdown), Start/Stop manually (we will go in here and do it), or Start automatically when …( I have no idea what this means 🙂 sorry – let me know in the comments 🙂 ). Anyways, that's all there is to this!
We are flying through this, Open/Close Ports
Same spot as above just hit the 'Properties' link on the Firewall section this time. Again, this is just as easy – just check/uncheck the boxes beside the service containing the port you want to open or close! Have a look below – it's pretty simple!
Another releavant spot here is the 'Firewall' button at the bottom. Aside from opening and closing a port, we can also specify which networks are able to get through if our port is open. Below I'm allowing access only from the 192.168.1.0/24 network.
That's what I get for talk about the CLI, custom services!
Aha! Too much talk of the CLI leads us to a task that can only be completed via the CLI; Custom Services. Basically, if you have a service that utilizes ports that aren't covered off by the default services you need to create your own spiffy little service so you can enable/disable it and open/close those ports and allow access to it. So, off to the CLI we go…
The services in the ESXi firewall are defined by XML files located in /etc/vmware/firewall The service.xml file contains the bulk of them and you can define yours in there, or you can simply add any xml file in the directory and it will be picked up (so long as it is defined properly). If you have enabled HA you are in luck – you will see an fdm.xml file there. Since the VCAP is time sensitive this might be your quickest way out as you can just copy that file, rename it to your service and modify as it fits. If not, then you will have to get into service.xml and copy text out of there. I'm going to assume HA is enabled and go the copy/modify route.
So, copy fdm.xml to your service name
cp fdm.xml mynewservice.xml
Before modifying mynewservice.xml you will need to give root access to write to it, use the following to do so…
chmod o+w mynewservice.xml
Now vi mynewservice.xml – if you don't know how to use 'vi', well, you better just learn, go find a site 🙂 Let's say we have a requirement to open up inbound tcp/udp 8000 and tcp/udp 8001 on the outbound. We would make that file look as follows, simply replacing the name and ports and setting the enabled flag.
Alright, save that bad boy, and probably it's a good idea to run 'chmod o-w mynewservice.xml' and take away that write permission. If you go and look at your services, or simply run 'esxcli network firewall ruleset list' you might say, "hey, where's my new service?" Well, it won't show up until you refresh the firewall – to do so, use the following command..
esxcli network firewall refresh
Now you can go check in the GUI or do the following to list out your services…
esxcli network firewall ruleset list
Woot! Woot! It's there! But wait, it's disabled. No biggie, we can go ahead and enable it just as we did the others in the steps earlier in this post – or, hey, since we are in the CLI let's just do it now!
esxcli network firewall ruleset set -r mynewservice -e true
And that's that! You are done! If asked to set the allowedIP information, I'd probably just jump back to the GUI and do that!
Set firewall security level – More CLI goodness
Well before we can set the firewall security level let's first understand what security levels are available to us. ESXi gives us three…
High – This is the default – basically, firewall blocks all incoming and outgoing ports except for the the essential ports it needs to run.
Medium – All Incoming is blocked, except for any port you open – outgoing is a free for all
Low – Nada – have at it, everything is open.
Anyway, we can get the default action by specifying
esxcli network firewall get
and to change it we have a few options… Passing '-d false' would set us to DROP (the default HIGH security level), passing a '-d true' will set us up to PASS traffic (I think this would be the medium security) and setting a '-e false' will disable the firewall completely (the low settings). So, to switch to medium we could do the following
esxcli network firewall set -d true
I could be wrong here, so if I am just let me know and I'll update it 🙂
And guess what? We are done with the firewall! I would practice this stuff as it's easy measurable and can be quickly identified as you doing something right or wrong – I'd bet this will be on the exam in one way or another. Good Luck!