Monthly Archives: December 2013

Winners of the Troubleshooting vSphere Storage eBook

2062EN_mockupcover_normal_thumb.pngWell the dust has settled and three winners have been randomly picked to receive an eBook copy of Troubleshooting vSphere Storage.  For the rest of you don't be saddened – Packt is running a $5.00 eBook sale from now through Jan 3rd so you can go on over to the books landing page and pick yourself up a copy for only 5 bucks πŸ™‚

Thanks so much to everyone who entered.  This was certainly the most participated contest that I have had on this blog thus far.  Thank you all for your support and kind words.  Now I've heard back from 2 of the 3 winners, so if I fail to hear back from the third I'll most certainly pick another winner!

And the winners are….

Eric Beach

Bonnie Bauder

Sean Thulin

Also, starting December 30th I will be opening up my annual #HappyNewSphere contest so be sure to check back.  I've got some great sponsors this year including VMware Press and Pluralsight – so you can imagine what the prizes might be πŸ™‚

8 weeks of #VCAP – The rest of Section 2 – Port Binding, CLI, and DPIO

Section 2 of the blueprint is a pretty big one, and some of the pieces warranted their own post – however there are a lot of small little skills that don’t really require a complete tutorial so I thought I would just slam them all in here!

Determine use cases for and apply Port Binding settings

vSphere offers three types of port binding in their vSwitch settings (Distributed Virtual Switch only)– all of which are explained below

  • Static – the port will be assigned immediately on connection to the vSwitch.  The VM will stay connected to this port even when it’s powered off.  The only way to free up the port is to explicitly remove the NIC from the VM.  Static Ports are managed through vCenter Server
  • Dynamic – Port is connected when the VM is powered on and then disconnected when the VM is powered off.  Dynamic ports are managed through vCenter Server.  This method has been depreciated in vSphere 5.x
  • Ephemeral – Both static and dynamic port binding has a set number of ports, in ephemeral, the ports are actually created and destroyed on the VM power on/power off event therefore requiring a bit more overhead.  That said, these are managed by the host, therefore, networking can still be connected/disconnected in the event that vCenter Server is unavailable.

Choosing a port binding method is pretty easy – Right click on your port group, chose edit settings and it should be front and centre in the General section.

Image 1

As far as use-cases go, really ephemeral only needs to be used in recovery purposes since they are a bit more demanding in terms of overhead.  Also, ephemeral does not maintain port-level permissions and controls when a VM is rebooted, since the port will be destroyed and recreated.  For the most part it’s best to use Static port binding – and since 5.0 offers an auto expand feature to dynamically grow the number of ports by a specified interval, you shouldn’t have to worry about running out of ports.

Command Line goodness

The networking section references the ability to use command line tools to manage both standard and distributed virtual switches.  Obviously I can’t go over every command and every switch.  Just be sure to know how to use esxcfg-vswitch, esxcfg-vmknic, esxcfg-route, the networking namespaces in esxcli, as well as some of the PowerCLI cmdlets around networking (Get-VirtualSwitch, Get-NetworkAdapter, Get-VMHostNetwork, etc).

Hint – for the PowerShell command line stuff you can quickly find the PowerCLI commands associated with networking (or anything for the matter) by utilizing the Get-VICommand cmdlet and passing a search string.  IE, to return all cmdlets containing ‘net’ you can use the following

Get-VICommand –Name *Net*

Determine use cases for and applying VMware DirectPath I/O

I’ve never used DPIO – that said, there it is on the blueprint so I’d better figure it out.  As for use cases, honestly I haven’t seen many.  For the most part utilizing the virtualized hardware seems to perform well enough, but if you need the tiny bit performance improvement it claims to provide there are a couple of steps to get it running.

First up we need to configure pass-through on the host itself.  This is done on the Configuration tab under ‘Advanced Settings’.  Simply select ‘Configure Pass-through’ and select the device you want to present to a VM.

dpio1

Once you are done this you will need to restart the host in order to complete the next step, so go ahead and do that.

As for presenting the pass-through device to the VM this is done just as you would do any other piece of hardware (In ‘Edit Settings’ of a VM).  Simply select PCI Device as your hardware and follow the wizard.  You should see your device that you had setup for pass-through earlier in the dropdown box as shown below.

dpio2

From here you will need to ensure that your guest OS has the correct drivers in order to install this hardware as it is presented directly to the VM.  Aside from creating a memory reservation on your VM there are also a ton of features that are unavailable when you utilize DPIO.  Things such as vMotion, HA, DRS, Snapshots, Hot add, Fault tolerance are all not supported – probably why there is such low adoption.

And I think that should just about wrap up networking.  There is some teaming information mentioned, but honestly I find this to be VCP level knowledge and I’m just going to assume you already know it πŸ™‚  Good Luck!

Kerberos authentication for the PowerShell plugin in vCO 5.5

1 The ability to have vCO kick off PowerShell scripts is pretty awesome!  And the fact that you can kick these off contextually inside of the vSphere Web Client is even more awesome!  Even more awesome than that, yes, that’s a lot of awesome is the new features offered with vCenter Orchestrator 5.5 – So, I’ve taken the plunge on one of my environments and upgraded.  Since then I’ve been slowly migrating workflows over – one of which utilized the PowerShell plug-in.  Now, since the appliance mode of vCO requires you to do a rip and replace rather than an upgrade (because I’m using the embedded database) I had to reinstall the PS plugin, therefore forcing me to reconfigure the Kerberos settings on vCO.   During this I realized that things are a little bit different than when I first blogged about vCO and PowerShell here.  Below is how I got it to work…

First up is the WinRM setup on your PowerShell host.  This process  hasn’t changed from 5.1, however I’ll still include the steps and commands that need to be run below.  Remember these are to be executed on the Windows box that you wish to run the PowerShell script from.

  • To create a winrm listener and open any required firewall ports
  • winrm quickconfig
  • To enable kerberos authentication
  • winrm set winrm/config/service/auth @{Kerberos=”true”}
  • Allow transfer of unencrypted data
  • winrm set winrm/config/service @{AllowUnencrypted=”true”}
  • Up the max memory per shell – I needed to do this to get things working
  • winrm set winrm/config/winrs @{MaxMemoryPerShellMB=”2048″}

No on to the krb5.conf file – this is where things get a bit different.  In vCO 5.1 we were required to edit the krb5.conf file located in /opt/vmo/jre/lib/security/ – well, if you go looking for that directory on 5.5 you won’t find it.  Instead, we need to create our krb5.conf file in /usr/java/jre-vmware/lib/security/  As far as what goes in the file it is the same and is listed below…(obviosoly substituting your own domain for lab.local and your own dc for the kdc definition).

[libdefaults]
default_realm = LAB.LOCAL
udp_preferences_limit = 1   [realms]
LAB.LOCAL = {
kdc = dc.LAB.LOCAL
default_domain = LAB.LOCAL
}   [domain_realms]
.lab.local=LAB.LOCAL
lab.local=LAB.LOCAL

After you have saved the file in the proper directory we need to modify the permissions.  The following line should get you the proper permissions to get everything working.

chmod 644 /usr/java/jre-vmware/lib/security/krb5.conf

Just a few other notes!  You might want to modify your /etc/hosts file and be sure that you are able to resolve the fqdn’s of both your dc and the PowerShell host you plan to use.  Also, when adding the PowerShell host be sure to select Kerberos as your authentication type and enter in your credentials using the ‘[email protected]’ format.

For now, that should get you automating like a champ!

8 weeks of #VCAP – CDP and LLDP

Well, 8 weeks of VCAP has dwindled down into a serious 8 days of VCAP – and for now, how about a little bit of random information from the Networking section of the blueprint.

First up, CDP and LLDP

These are relatively easy to configure, however there are a few different modes that they can be run in, therefore I thought it would be best if I write them down in hopes that maybe I’ll remember them if any scenarios require me to configure them.

Basically the functionality of the two protocols is identical – they both provide discovery of ports connected to a virtual switch.  CDP however supports just Cisco physical switches whereas LLDP supports any switch supporting LLDP.  Another note, CDP can be enabled on both vSphere Standard Switches and vSphere Distributed Switches – LLDP – dvSwitch only!

So let’s have a look at the dvSwitch config first.  Like I mentioned earlier it’s pretty simple. From the properties tab of a vSphere Distributed Switch select ‘Advanced’.  From here its as simple as setting the status to Enabled, the type to either CDP or LLDP, and the Operation mode (explained below).

  • Listen – ESXi detects and displays information from the associated physical switch port, but all information in regards to the virtual switch is not available to the physical switch.
  • Advertise – ESXi presents information in regards to the virtual switch available to the physical switch, but doesn’t detect any information in regards to the physical switch port
  • Both – Does both advertise and listen.

dvswitch

Now that we are enabled we can view what information we receive inside of the Networking section of a hosts configuration tab.  To do so, simply expand out your physical uplinks and click the information icon (shown below).

lldp 

And that’s all there is for that – with the distributed switch anyways.  To get CDP working on a standard switch we are once again back into the command line interface.  Probably good to brush up on these commands anyways since its also mentioned in the blueprint.  So, Let’s say we wanted to configure CDP on a vSphere Standard Switch called vSwitch0 to a value of Both.  We could use the following command

esxcli network vswitch standard set –v vSwitch0 –c both

And that’s all there is to that – valid options for –c would be both, listen, advertise or down.  To view we could use the same process as above.

8 weeks of #VCAP – Syslog scenario by @tomverhaeg

Company policies state that every syslog capable device or server should send these logs to an appropriate syslog collector. Your colleague has already set up the VMware syslog collector on a separate machine, located at 10.10.20.45. You have been tasked with setting up the syslog clients on the ESXi hosts, and ensuring that syslogs arrive on the syslog server.

To configure the syslog collector on the ESXi hosts, we will be using the esxcli system syslog namespace. This allows us to set different options regarding the local and remote (which is what we want) syslog.

Let’s review the default config first by using the following command:

~ # esxcli system syslog config get

Default Rotation Size: 1024

Default Rotations: 8

Log Output: /scratch/log

Log To Unique Subdirectory: false

Remote Host: <none>

We see that no remote syslog is being used. Let’s configure one, using this command:

~ # esxcli system syslog config set –loghost=10.10.20.45

Now that we have configure a remote loghost, we need to reload the syslog daemon to apply the configuration changes. Esxcli can help us once again:

~ # esxcli system syslog reload

You might think that we’re ready now, but when we check our syslog, we don’t see syslog yet. Bummer! For this problem, I’ll reference to the ESXi firewall post (http://blog.mwpreston.net/2013/11/19/8-weeks-of-vcap-the-esxi-firewall/) as with the default security level, this outgoing traffic will be dropped. We need to enable the firewall rule for syslog (udp/514, tcp/1514).

~ # esxcli network firewall ruleset set -r syslog -e true

And reload our changes:

~ # esxcli network firewall refresh

And now, we see our host logs coming in. The VMware syslog collector stores it logs by default in C:\ProgramData\VMware\VMware Syslog Collector\Data

clip_image002

Troubleshooting vSphere Storage eBook giveaway!

2062EN_mockupcover_normalA few weeks ago I released a post in regards to my finishing of the Troubleshooting vSphere Storage book.  This has been a lot of work and I’ve had a lot of help from the community in getting this book completed.

Writing a book is not a simple task.  It involves research, lab time, and focus.  And then there is the editing, both grammatically and technically – which can be even more work than the writing itself! 

I tried to gear this book towards the vSphere Admin.  The “jack of all trades” system administrator.  Hopefully readers will find the knowledge and how to within the book to solve common storage issues that tend to spring up within a vSphere environment.  The book is broken into three main subjects of focus; troubleshooting connectivity, troubleshooting contention and troubleshooting capacity.  If you’d like to purchase the book you can do so by visiting the landing page on Packt’s website and following the various channels.

Tis the season for winning

That said – Tis the season right?  The season for giving!  That’s why I’m happy to announce that I have three eBook versions of Troubleshooting vSphere Storage to giveaway over the next week or so.  A little light reading to enjoy over the xmas holidays!

As you can see by the little widget below, I’ve opted to use PunchTab to gather the entries for this contest.  It’s hectic trying to follow Twitter hashtags and what not so I thought “Hey, why not use a CaaS (contest as a service) solution.  Sorry for all the PunchTab branding but this is what you get when you use a free service πŸ™‚

So how do you enter?  It’s easy, leave a comment, send a tweet and follow me!  That will get you three entries!  Just make sure you do it through the widget below.  I’ll have PunchTab pick three random winners at the end of the contest (end of Sunday, December 15th) and announce the winners shortly thereafter.  Good luck and let me know what you think of the book!

8 weeks of #VCAP – Fault Tolerance by @tomverhaeg

You might know VMware Fault Tolerance already, since the VCAP exam builds on the VCP knowledge. But still, it is in the blueprint, so it might be wise to go over it.

Fault Tolerance, often abbreviated as FT is a technique in which a shadow VM of a running VM is kept in lockstep with the primary. This basically means that all memory and CPU calculations on the primary VM also will be executed on the secondary VM.

In case of a host failover, a VM with fault tolerance enabled can switch over from the primary to the second VM in a matter of seconds, taking right over where the primary stopped. This allows for a better uptime of that VM and avoids the VM restart that HA would do.

There are a few host requirements for running FT:
-> You need to have a cluster where HA is enabled

-> All hosts needs to access the same (shared) datastores

-> There needs to be physical processor support

-> VMkernel ports need to be configured for vMotion and FT logging

There are also some VM requirements for running FT:

-> The VM can only have one (1) vCPU, so no vSMP

-> The VM disks need to be eager zeroed thick provisioned

-> No non re-playable devices (CD ROM, USB devices etc).

-> No snapshots

Configuring the VMkernel port for FT logging

Conform VMware best practices for FT, it it wise to use a dedicated NIC for FT logging (preferably even 10 gigabit), but configuring FT logging is as easy as selecting a checkbox on a VMkernel port:

clip_image002

Enabling FT on a VM

Enabling FT is rather simple, right-click the VM -> Fault Tolerance -> Turn on Fault Tolerance. You might get a popup saying that a reservation (memory) will be created for the full memory allocation of this VM, and that the disk will be eager zeroed out.

clip_image004

After it walks through the process of enabling fault tolerance, you get a nice blue icon in your inventory:

clip_image006

After powering on the FT VM, on the summary page, you also see some info about the FT status:

clip_image008

Testing VMware FT

Now that we have a running FT VM, we might as well test it. We have 2 options for testing it:

Test failover – The primary VM does a failover to the primary VM, and then spawns up a new secondary VM.

Test restart secondary – The secondary VM is re-spawned and the FT configuration is protected again.

clip_image010

After doing a failover of the primary VM, a new secondary VM will be spawned, so the status after doing the failover might be like this:

clip_image012

Troubleshooting VMware FT

So, all is happy, but since we’re doing the VCAP exam, we might expect some troubleshooting.

On the summary page of the host, you can see if the host is configured and ready for FT. If it isn’t, the reason why will also be mentioned:

clip_image013

In the image above, there isn’t a VMkernel port configured for FT logging. So go into your networking and check that FT logging box.

Also, when the VM mentions something like this, the secondary VM is not running, so do a restart or migrate secondary:

clip_image014

8 weeks of #VCAP – Host Cache Scenario by @tomverhaeg

Big thanks to Tom Verhaeg ( BLOG / TWITTER ) for another awesome practice scenario for the VCAP5-DCA

You recently acquired some SSD drives for in your hosts. You’re not running vSphere 5.5 yet so vFRC is not an option. You read something about swap to host cache, and you think it might be wise to configure your SSD drive for usage as host cache.

Well, the process of configuring this isn’t that hard. The swap to host cache will be used as a last resort and a replacement of swapping to “disk”. Remember that vSphere has 4 main memory management techniques:

1) Transparent page sharing: Eliminates redundant copies of memory pages by removing them from memory and creating a reference instead.

2) Memory ballooning: In times of contention, the balloon driver (comes with VMware Tools) will ask the guest OS for unused memory and returns this back to vSphere

3) Memory compression: After ballooning runs out, try compressing the memory (basically gzipping it).

4) Swap to disk / host cache: Swap memory to a disk of some sort.

So, the swapping itself comes last in a process of memory management. While it’s still not wanted, swapping to an SSD is still better than to storage or slow local storage.

You configure this by offering up a (portion of a) n SSD tagged datastore as host cache. Go to Configuration -> Host cache configuration

clip_image002

All devices that are being recognized as SSD drive will show up here. You can right click the datastores and set the amount of disk space that you are willing to spend on host cache. If you haven’t formatted a datastore yet, but do have an SSD in place, you can use the Add storage wizard mentioned above.

clip_image003

Once you’ve configured this, you can browse the datastore which you have (partially) allocated to Host cache. On your datastore, you will find a hashed folder, and in that folder a folder named hostCache.

Something like this: 5241d252-0687-cf96-f89a-10ddb1eabcf5/hostCache

In this folder, you will find as much .vswp files as the total number of GB’s that you have allocated to host cache.

Hurray!

8 weeks of #VCAP – vSphere Network I/O Control

Alright – here we go, Network I/O Control – Objective 2.4 of the blueprint lists this as a skill you must know.  Honestly, I've never used this before writing this post…thankfully, it's a very very easy thing to configure.  Unless I'm missing something, in which case I'm in for some trouble come exam time πŸ™‚

First up, let's have a look at the requirements.

  • Enterprise Plus licensing – since you need a distributed switch to use NIOC, in turn you need Ent+ licenses.

OK, maybe I should of said requirement – not plural.  I can't seem to find any other requirements for using NIOC.  Anyways, the first step in getting NIOC setup is to enable it, and this in itself is a matter of checking a box.  From within the Networking inventory view on the Resource Allocation tab select ‘Properties’ and check the box πŸ™‚

nioc-enable

 

System Network Resource Pools

Easy enough right!  Now on to our Network resource pools.  As you can see, there are some default system network resource pools already setup within NIOC.

  • Fault Tolerance
  • iSCSI
  • Management Traffic
  • Virtual Machine Traffic
  • vMotion
  • vSphere Replication

I’ll leave it to your imagination as to what traffic these represent.  Basically these resource pools are automatically applied to their corresponding traffic type when we enable NIOC.   NIOC utilizes the same type of sharing mechanism that resource pools utilize.  Meaning each resource pool is assigned a share value, one that will apply relatively to the other pools during network contention.  Thus, if going by the example in the Networking guide, if we assign FT and iSCSI a share value of 100, while all other resource pools having 50 shares, iSCSI and FT would each get 25% while the remaining resource pools would receive 12.5% of the available bandwidth (during contention).  The table below should help with that formula

Resource Pool Shares Total Shares Percentage
iSCSI 100 400 25%
FT 100 400 25%
Management 50 400 12.5%
VM 50 400 12.5%
vMotion 50 400 12.5%
Replication 50 400 12.5%

What if I want to further segregate my VM traffic?

A valid question.  To resolve this NIOC allows us to create our own User-defined network resource pools. Again, this is a very easy process.  Selecting ‘New Network Resource Pool’ will get the dialog box open that we need.  See Below..

newresourcepool

As you can see, we can create our own resource pool, assign either a predefined (high, normal, low) share value to it (or we can set a custom number) as well as a QoS priority tag if we need to tag outbound QoS from our virtual switch.  Just a note, we can change the values and QoS tags on our system defined resource pools as well if need be.

Now that we have our resource pool created there’s only one final step in applying it.  Using the ‘Manage Port Groups’ link we can assign our newly created resource pool to one of our dvPortGroups.  Below I’ve done just that by assigning ‘My Server Traffic’ to dvServers.

assignportgroup

And that’s all there is to NIOC really.  Again, not too hard, but something I’ve never touched before now.  Also, something that could of caught me off guard on the exam – the last thing i want to do is spend time reading documentation!  Good luck studying!