Balance your vCenter Log Insight syslogs with KEMP Virtual LoadMaster

When vCenter Log Insight 2.0 was released it brought with it many useful features.  One of which has been dubbed cluster mode.  Cluster mode allows us to deploy vCenter Log Insight in a scale out architecture, where we can attach many worker nodes to a single master node.  Aside from providing us with high availability, cluster mode also allows us to support over and above the 7500 maximum events per second that is supported within just a single instance.

Although cluster mode does open up a lot of doors in terms of scale, it does not do much in terms of evenly distributing load between your vCenter Log Insight instances.  In fact, any load balancing in terms of splitting up the syslog connections from our ESXi hosts will need to be done manually.  Not a feasible solution, especially if you are scaling due to the number of nodes sending data.  So what we are essentially left with is a void that load balancing companies are eager to fill

Enter KEMP

KEMP Technologies seen a problem here and have addressed it with an add-on pack designed specifically for vCenter Log Insight that can be installed directly into their current LoadMaster offering.  Essentially the  KEMP LoadMaster will act as a central aggregation point for all syslog activity, in turn, distributing the logs evenly across all of the master/worker Log Insight instances which sit behind it.  However KEMP does this differently than most other load balancing products.  One of the biggest features of the KEMP LoadMaster is its ability to balance Log Insight nodes no matter what collection method or protocol you are using, in an efficient and easy manner.  UDP is a no brainer, most all load balancing solutions will be able to support this.  However syslog through TCP can definitely present challenges to load balancers, mainly due to clients sending many messages at once, resulting in very long sessions.  KEMP can address these issues utilizing it’s understanding of syslog, as well as it’s application centric layer 7 visibility into the incoming traffic.

You can pull down trials of both vCenter Log Insight as well as the KEMP LoadMaster to see it in action for yourself.

We’ve covered What, Where, and Why – so How?

So let’s just dive right into this!  Both vCenter Log Insight and the KEMP LoadMaster are distributed as a virtual appliance, so they are very easy to deploy into our environments.  I’m not going to go into detail about setting up a Log Insight cluster as I think the installation wizards are intuitive enough to get you going.  In fact, the KEMP LoadMaster install is just as easy, but let’s have a look at it anyways.

As mentioned earlier the KEMP LoadMaster is shipped as a pre-packaged OVF appliance and is very easy to install.  Simply right-click on our datacenter object and select ‘Deploy OVF’.  After supplying the wizard with the basic hostname and network information we should be good to move on to the configuration.

You will need at lease version 7.1-20 of the KEMP LoadMaster to support the add-on package for vCenter Log Insight.  It is available for download here.

Next we will need to install our Log Insight add-on which  in the tools section of KEMP’s site.  We will install this by selecting System Configuration > System Administration > Update Software from the main navigational menu down the right hand side of the administration page and use the ‘Installed Addon Packages’ section a shown above.

Once we have installed our addon it’s a good idea to reboot. – This can be initiated by clicking the ‘Reboot’ button located on the System Configuration->System Administration->System Reboot page.

Now that we have our software pieces in place it’s time to start configuring some parameters and virtual services that will handle the load balancing to our Log Insight nodes.

First up is the Log Insight Message Split Interval.  This interval defines how many syslog messages we would like to send to a specific Log Insight node before moving on to the next node within the cluster.  For my purposes I left this value at it’s default of 10, but it can be anywhere between 1 and 100.

Our next step solely depends on what collection method we intend to use to send our logs to Log Insight.  For the purposes of this tutorial I’ve only included UDP, but if you are using TCP or even the Log Ingestion API you can find all the information that you will need within the KEMP Log Insight Manager Deployment Guide.

To support our UDP syslog balancing we will first need to create a UDP syslog Virtual Service.  To do so, navigate to Virtual Services->Add New from the navigational menu.  As shown in the screenshot below you can see there are a handful of parameters we need to supply.  First, a Virtual Address – this is simply a reachable IP on your network (not the same as Log Insight nodes or the LoadMaster node) that we will direct our logs to.  Also, we need to provide 514 as our port, UDP as our protocol and a descriptive Service Name.

After adding our Virtual Service we will need to expand the ‘Standard Options’ to setup a few more parameters before continuing (shown below).  First, we will want to be sure we check the “Force L7” checkbox.  What this does is allow the Virtual Service to run in Layer 7 of the OSI instead of Layer 4, essentially, allowing KEMP to keep the source IP Address ( that of our host ), but change the destination address from what would normally be our virtual service to that of our Log Insight node we have forwarded the packet to, allowing a more transparent load balancing experience.  Also in this section we will want to set our Idle Connection Timeout to 1 and select the ‘Set Idle Timeout’ button.  We will also want to ensure that ‘Transparency’ has been checked and Round Robin has been selected as our Scheduling Method.

Now we need to expand the ‘Real Servers’ section and add our Log Insight nodes.  First, ensure that ICMP Ping is selected as our check parameter and click ‘Add New’ to add a real server (Log Insight node).

Here we simply need to add as many “Real Servers” as we have Log Insight nodes.  This is a pretty simple process where we just supply the IP/DNS of our Log Insight node and select “Add this real server”. – again, repeating for each Log Insight node we have.

At this point we are done with the configuration and setup of the KEMP LoadMaster.  It’s just a matter of configuring (or re-configuring) our ESXi hosts and other devices forwarding logs to Log Insight to point at the IP address of our Virtual Service we just created.  To do so, we can modify the Syslog.global.loghost setting within the ‘Advanced Settings’ of our host to include udp://IP_OF_VIRTUAL_SERVICE:514/.  Also, do not forget to have a look at your hosts Security Profile to ensure that syslog traffic is indeed allowed through the local firewall contained on ESXi.  If you need more information on configuring syslog on ESXi, or are looking for ways to automate this I’ve provided the many different ways to do it near the bottom of this previous post.  After pointing your hosts towards our Virtual Service IP the magic of load balancing would have already began.

The data to back it up

Once we have our hosts pointing towards are Virtual Service we can go ahead and checkout the Statistics section inside of our KEMP LoadMaster.  Below you can see a shot of my setup.  From the Virtual Services section we can see that all connections are split 50/50 between the two Log Insight nodes I have configured.  To get even more detailed information per node, we can select the ‘Real Servers’ section and see each individual nodes total connections, bytes, and packets.

Conclusion

If you are looking to put a third party load balancer in front of your vCenter Log Insight installation I would definitely have a look at what KEMP has to offer.  In fact, the LoadMaster from KEMP will balance almost any application out there, so don’t think you are just limited to Log Insight. Be sure to have a look at the KEMP LoadMaster product page here to review all of the functionality and features included.   Aside from being a great load balancer (which is required), the biggest selling point from the KEMP LoadMaster that I have noticed throughout this process was definitely the ease of use.  Honestly, I went from downloading the OVF file to a fully functional load balanced Log Insight environment in less than 30 minutes.  Not being an “official” network guy and not having a whole lot of time in my day job this is very important to me.  However I did only setup the UDP Virtual Service and not the TCP and Log Ingestion services so one could argue that I didn’t do a complete setup.  That said, KEMP does have something called Templates that you can import into your LoadMaster making the setup as simple as supplying a Virtual Service IP and some Real Servers (Log Insight IP’s).  As mentioned before the KEMP LoadMaster is one of the only load balancers that can truly balance Log Insight syslog traffic over UDP, TCP, and through the Log Ingestion API.  KEMP will also monitor the health of our Log Insight nodes and dynamically remove and re-add nodes as they become unavailable and available.  Nodes can also be manually removed from our pool to perform things such as troubleshooting or maintenance, all while ensuring we still have complete access to our Log Insight infrastructure.   If you are looking to scale your Log Insight environment, better balance your syslog connections or simply maintain a higher level of high availability I would certainly recommend and take a look at what KEMP has to offer.

Don’t just take my word for it, give it a try!  You can get a fully functional trial of the KEMP Virtual LoadMaster for 30 days here.  With a setup of less than 30 minutes, 30 days is more than enough time to take it for a spin .

THIS REVIEW WAS SPONSORED BY KEMP TECHNOLOGIES INC.