Monthly Archives: September 2014
When vCenter Log Insight 2.0 was released it brought with it many useful features. One of which has been dubbed cluster mode. Cluster mode allows us to deploy vCenter Log Insight in a scale out architecture, where we can attach many worker nodes to a single master node. Aside from providing us with high availability, cluster mode also allows us to support over and above the 7500 maximum events per second that is supported within just a single instance.
Although cluster mode does open up a lot of doors in terms of scale, it does not do much in terms of evenly distributing load between your vCenter Log Insight instances. In fact, any load balancing in terms of splitting up the syslog connections from our ESXi hosts will need to be done manually. Not a feasible solution, especially if you are scaling due to the number of nodes sending data. So what we are essentially left with is a void that load balancing companies are eager to fill
KEMP Technologies seen a problem here and have addressed it with an add-on pack designed specifically for vCenter Log Insight that can be installed directly into their current LoadMaster offering. Essentially the KEMP LoadMaster will act as a central aggregation point for all syslog activity, in turn, distributing the logs evenly across all of the master/worker Log Insight instances which sit behind it. However KEMP does this differently than most other load balancing products. One of the biggest features of the KEMP LoadMaster is its ability to balance Log Insight nodes no matter what collection method or protocol you are using, in an efficient and easy manner. UDP is a no brainer, most all load balancing solutions will be able to support this. However syslog through TCP can definitely present challenges to load balancers, mainly due to clients sending many messages at once, resulting in very long sessions. KEMP can address these issues utilizing it’s understanding of syslog, as well as it’s application centric layer 7 visibility into the incoming traffic.
We’ve covered What, Where, and Why – so How?
So let’s just dive right into this! Both vCenter Log Insight and the KEMP LoadMaster are distributed as a virtual appliance, so they are very easy to deploy into our environments. I’m not going to go into detail about setting up a Log Insight cluster as I think the installation wizards are intuitive enough to get you going. In fact, the KEMP LoadMaster install is just as easy, but let’s have a look at it anyways.
As mentioned earlier the KEMP LoadMaster is shipped as a pre-packaged OVF appliance and is very easy to install. Simply right-click on our datacenter object and select ‘Deploy OVF’. After supplying the wizard with the basic hostname and network information we should be good to move on to the configuration.
You will need at lease version 7.1-20 of the KEMP LoadMaster to support the add-on package for vCenter Log Insight. It is available for download here.
Next we will need to install our Log Insight add-on which in the tools section of KEMP’s site. We will install this by selecting System Configuration > System Administration > Update Software from the main navigational menu down the right hand side of the administration page and use the ‘Installed Addon Packages’ section a shown above.
Once we have installed our addon it’s a good idea to reboot. – This can be initiated by clicking the ‘Reboot’ button located on the System Configuration->System Administration->System Reboot page.
Now that we have our software pieces in place it’s time to start configuring some parameters and virtual services that will handle the load balancing to our Log Insight nodes.
First up is the Log Insight Message Split Interval. This interval defines how many syslog messages we would like to send to a specific Log Insight node before moving on to the next node within the cluster. For my purposes I left this value at it’s default of 10, but it can be anywhere between 1 and 100.
Our next step solely depends on what collection method we intend to use to send our logs to Log Insight. For the purposes of this tutorial I’ve only included UDP, but if you are using TCP or even the Log Ingestion API you can find all the information that you will need within the KEMP Log Insight Manager Deployment Guide.
To support our UDP syslog balancing we will first need to create a UDP syslog Virtual Service. To do so, navigate to Virtual Services->Add New from the navigational menu. As shown in the screenshot below you can see there are a handful of parameters we need to supply. First, a Virtual Address – this is simply a reachable IP on your network (not the same as Log Insight nodes or the LoadMaster node) that we will direct our logs to. Also, we need to provide 514 as our port, UDP as our protocol and a descriptive Service Name.
After adding our Virtual Service we will need to expand the ‘Standard Options’ to setup a few more parameters before continuing (shown below). First, we will want to be sure we check the “Force L7” checkbox. What this does is allow the Virtual Service to run in Layer 7 of the OSI instead of Layer 4, essentially, allowing KEMP to keep the source IP Address ( that of our host ), but change the destination address from what would normally be our virtual service to that of our Log Insight node we have forwarded the packet to, allowing a more transparent load balancing experience. Also in this section we will want to set our Idle Connection Timeout to 1 and select the ‘Set Idle Timeout’ button. We will also want to ensure that ‘Transparency’ has been checked and Round Robin has been selected as our Scheduling Method.
Now we need to expand the ‘Real Servers’ section and add our Log Insight nodes. First, ensure that ICMP Ping is selected as our check parameter and click ‘Add New’ to add a real server (Log Insight node).
Here we simply need to add as many “Real Servers” as we have Log Insight nodes. This is a pretty simple process where we just supply the IP/DNS of our Log Insight node and select “Add this real server”. – again, repeating for each Log Insight node we have.
At this point we are done with the configuration and setup of the KEMP LoadMaster. It’s just a matter of configuring (or re-configuring) our ESXi hosts and other devices forwarding logs to Log Insight to point at the IP address of our Virtual Service we just created. To do so, we can modify the Syslog.global.loghost setting within the ‘Advanced Settings’ of our host to include udp://IP_OF_VIRTUAL_SERVICE:514/. Also, do not forget to have a look at your hosts Security Profile to ensure that syslog traffic is indeed allowed through the local firewall contained on ESXi. If you need more information on configuring syslog on ESXi, or are looking for ways to automate this I’ve provided the many different ways to do it near the bottom of this previous post. After pointing your hosts towards our Virtual Service IP the magic of load balancing would have already began.
The data to back it up
Once we have our hosts pointing towards are Virtual Service we can go ahead and checkout the Statistics section inside of our KEMP LoadMaster. Below you can see a shot of my setup. From the Virtual Services section we can see that all connections are split 50/50 between the two Log Insight nodes I have configured. To get even more detailed information per node, we can select the ‘Real Servers’ section and see each individual nodes total connections, bytes, and packets.
If you are looking to put a third party load balancer in front of your vCenter Log Insight installation I would definitely have a look at what KEMP has to offer. In fact, the LoadMaster from KEMP will balance almost any application out there, so don’t think you are just limited to Log Insight. Be sure to have a look at the KEMP LoadMaster product page here to review all of the functionality and features included. Aside from being a great load balancer (which is required), the biggest selling point from the KEMP LoadMaster that I have noticed throughout this process was definitely the ease of use. Honestly, I went from downloading the OVF file to a fully functional load balanced Log Insight environment in less than 30 minutes. Not being an “official” network guy and not having a whole lot of time in my day job this is very important to me. However I did only setup the UDP Virtual Service and not the TCP and Log Ingestion services so one could argue that I didn’t do a complete setup. That said, KEMP does have something called Templates that you can import into your LoadMaster making the setup as simple as supplying a Virtual Service IP and some Real Servers (Log Insight IP’s). As mentioned before the KEMP LoadMaster is one of the only load balancers that can truly balance Log Insight syslog traffic over UDP, TCP, and through the Log Ingestion API. KEMP will also monitor the health of our Log Insight nodes and dynamically remove and re-add nodes as they become unavailable and available. Nodes can also be manually removed from our pool to perform things such as troubleshooting or maintenance, all while ensuring we still have complete access to our Log Insight infrastructure. If you are looking to scale your Log Insight environment, better balance your syslog connections or simply maintain a higher level of high availability I would certainly recommend and take a look at what KEMP has to offer.
Don’t just take my word for it, give it a try! You can get a fully functional trial of the KEMP Virtual LoadMaster for 30 days here. With a setup of less than 30 minutes, 30 days is more than enough time to take it for a spin .
THIS REVIEW WAS SPONSORED BY KEMP TECHNOLOGIES INC.
I’m super excited to say that I will be attending the first annual VeeamON conference coming up in October in fabulous Las Vegas! I’ve been a long time supporter of Veeam and their flagship product Veeam Backup and Replication so when I heard they were going to have three full days of deep technical sessions I knew right away that I needed to be there!
What might I mean by deep technical sessions? Well, you can take a peak at the session catalog to get a feel for what we are in store for. Topics like working with the Veeam REST api, best practices for backing up SQL/Exchange/AD, and Storage/Network best practices are certainly ones that jump out at me. I’ve had a lot of ideas for blog posts and projects centred around some of the subjects so hopefully having access to all the Veeam Experts at the conference will give me the final push to finally finish some of these up.
Also, we have already seen some of the new features that Veeam Backup and Replication v8 has to offer. Thinks like Replication from Backups, Failover Plans and Backup encryption have all been well documented and the product is well into it’s beta phase. That said, I get the feeling that we haven’t heard the whole story around Veeam v8 and I bet there will be some super awesome features unveiled during VeeamON – don’t hold me to that though, it’s simply my personal opinion.
If you haven’t had a look at the agenda I suggest you do that as well. Veaam has landed some great speakers for keynotes and general sessions and it should make out to be a great show! If you’re coming, definitly hit me up on Twitter – I’d love to hang out. If you can’t make it, well, that totally sucks – but don’t fret – I’ll do my best to try and live blog and capture the buzz on this blog. In this case, what happens in Vegas – get’s shared with the world.
But down to business – Solarwinds and Stephen Foskett have granted myself with the honor of being a thwack ambassador for the month of September – and my topic – database analysis and performance. I know what you might be thinking its odd for this virtualization geek to be talking about database performance, but the fact of the matter is I’m responsible for many databases within my day job so I couldn’t be more excited! Excited to share what I know in an area that isn’t directly related to virtualization – but even more so, excited to learn more about an area that I feel I could improve in.
So, if you have some time head on over to the thwack community and check out my first two posts; “It’s always the databases fault” and “Making performance metrics make sense to your business“. Leave a comment and you may just win yourself a jambox!
thwack isn’t just about giving away a prize to hammer you with info on Solarwinds products -Honestly I’m very impressed with the the content over on thwack! There are some great conversations going on over there dealing with everything from virtualization monitoring to cloud to mobile to, well, database analysis. There’s a lot to learn and thwack is definitely a great community to join in order to not just get answers to your problems, but to genuinely expand your knowledge – Be sure to check it out!