Tag Archives: Ravello

Ravello expanding HVX platform to bare metal.

I have had the chance to work with Ravello quite  a lot in the last few years – participating in their initial vExpert access program testing out their beta of running nested ESXi – as well as getting a nifty deep dive into their HVX offering at a Tech Field Day a few years ago.  The tech is awesome – the people are great.

One thing I will admit though – when they were purchased by Oracle I was worried.

You see, sometimes when giant corporations such as Oracle swallow up a start-up such as Ravello we see some of the tech disappear – meaning, the bigger company takes the cut of meat they want off of the bone and tosses the rest into the garbage never to be seen again!

All that said this certainly was not the case with Oracle and Ravello – the vExpert access that they so graciously offered up, well it remained – and over the past couple weeks I got a sneak peek into what exactly Oracle and Ravello have been up to – and to be honest, it’s only gotten better!

Ravello on Oracle Cloud Infrastructure

Ravello’s HVX was a key component of us being able to forklift our VMware VMs as they are and place them into either the Amazon or Google Cloud computing environments – or, if we fancied we could simply turn on a flag inside Ravello and install ESXi directly in the cloud.   In the end, we had this sort of Hypervisor inception if you will – a KVM fork running HVX which in turn ran ESXi and then our VMs.  As you can imagine there were some performance bottlenecks in doing so – not to say that Ravello was slow by any means – to be honest, the performance really surprised me when I started working with it.    In order to properly execute the underlying instructions in the traditional Ravello environments they used something called binary translation.  This had to be used due to underlying hardware instruction sets which were not exposed to Ravello – therefore Ravello would essentially invent their own through software.  The binary translation, coupled with the “nestedness” of the solutions, and with some hard limits on number of CPUs/Memory per VM tended to place Ravello into the Testing/Lab/Pre-Production folder of workload types…

Well, today, that all changes – today we now have a couple other options when it comes to deploying our VMs inside of Ravello.  We now have regions which are tied to the Oracle Cloud Infrastructure (OCI).  OCI is basically Oracles cloud, however in the sake of Ravello and HVX we have a couple different flavours of platforms we can deploy on.  First, now that Ravello essentially owns the infrastructure HVX is running on we can use something called Hardware assisted Nested Virtualization – in short, the hardware underneath supports exposing those instruction sets directly up to the HVX hypervisor – thus improving speeds and performance with that.

But Ravello on OCI doesn’t stop here either.  Today we also have a new flag that we can apply to our workloads within Ravello – The preferPhysicalHost parameter can now be set to true on a per-VM basis, and when doing so we essentially can run our workload directly on top of HVX, which in turn is running directly on bare metal.  Here we can execute our physical instruction set directly on bare metal hardware, without the need for any sort of software or hardware translations at all.  As shown below certainly the bare-metal option gives us the best performance out of the three options as there is now software or hardware translations happening whatsoever.

So what type of performance are we talking about?

As I mentioned earlier I really didn’t have an issue with the Ravello performance before the Oracle acquisition – maybe I’m just a patient guy but I think I just didn’t know what else could be done.  To get a bit of a baseline on performance I decided to execute the PassMark benchmark on the three different Ravello regions – and ended up with the following results

Software based nested Virtualization (AWS/GCE) Oracle Cloud Infrastructure Hardware Assisted nested virtualization Oracle Cloud Infrastructure Bare Metal
passmark-oraclehardwareassist passmark-normal Ravello-BareMetal

As you can see we gain some great performance when moving into hardware assisted virtualization, however the jump from software based instruction sets to bare-metal is nearly double the PassMark benchmark score – a pretty valid increase!  Now I know sometimes these benchmarks mean very little but there has to be some truth to the increase in the numbers right?  Either way, the difference in performance is most certainly noticeable – to be honest, I started the bare metal test directly after the software based test – and it nearly finished first.

So with these new performance increases, and support to larger VMs consuming more resources I’m sure we will see Ravello on Oracle Cloud Infrastructure make a bigger push into the enterprise, looking to migrate some of those production workloads into the cloud.    I’m happy for Ravello and happy that Oracle has put forth the resources they have to make Ravello and HVX even better than it was.  It really is a solid service which provides some very unique and interesting opportunities as it applies to networking and cross cloud configurations.  You can find the official Ravello announcement here – or if looking for more community blogs give #RavelloOCI a search on Twitter!  Thanks for reading!

Friday Shorts – #TOVMUG, Ravello, Veeam Vanguard, vExpert and more…

Doughnuts!  I told you I don’t like ethnic food! – Mr Burns, The Simpsons

This edition of Friday Shorts isn’t really going to be that short at all – It’s been a while since I’ve done one and there is a ton of content out there that I wanted to re-share for people so buckle up, here it is!

2016 Toronto VMUG UserCon !

tovmugThursday, February 25th marked the date for the 2016 Toronto VMUG Full Day UserCon and I couldn’t be happier as to how it turned out!  Honestly, as a leader I was interested the whole day about how many attendees we had and as it turned out we had our best year ever with 800+ registrations and 605 brains that walked through the doors!  Last year we had 601 and I’m sure it would have been more for this year but you know, Canadian winters eh!  We had Nick Marshall do the morning keynote and his talk was awesome – I heard nothing but good things from the attendees in regards to the content Nick had to talk about (BTW it was dubbed Building your IT Career)  If you get a chance to have Nick out for your VMUG do so!  He’s awesome!  For our lunchtime keynote we had a VCDX panel – Tim Antonowitz, James Wirth, Joe Silvagi and Eiad Al-Aqqad graciously volunteered to sit up on stage as we had some great discussion with the attendees at the conference from everything dealing with certification through to NSX integration!  All in all it was a great day and if you were there I hope you enjoyed it – if you weren’t, come next year!

What does Docker, vExpert, Ravello and Netflix have in common?

@h0bbel that’s what!  More explanation – well, Christian is a vExpert, vExperts get some free CPU hours on Ravello Systems (Ravello is awesome BTW – be nice Oracle!), and he has a great post on his blog on how to setup Dockerflix within Ravello on Photon in order to test (get around) those geo-blocking Netflix firewall “you can’t watch this” setups!   So if you are say a vExpert blogger that lives in Canada that really fancies yourself a couple of hours of Legally Blonde I suggest you set the Maple Syrup ladle down and head over to Christians post!.

Speaking of Ravello – How’s about some community cmdlets?

PowerShell-LogoAnd when you think of community in the sense of this blurb just think of Luc Dekens being a one-man community!  If you are Ravello user and fancy yourself some PowerShell cmdlets Luc has an updated module on his blog available for download.  Luc has certainly put a lot of effort into this module which contains hooks into almost every single API Ravello offers!  I certainly find this module very useful for my work with Ravello and just want to give Luc a big Thank You for this!

Running commands on the VCSA without SSH

commandlineIf I was being shipped off to a deserted island and could only take with me the RSS feed of one virtualization blog I have to think that as of today that blog would be William Lams virtuallyGhetto!  William just seems really good at figuring things out – is that a line that can go on a resume?  I don’t know, either way, his recent post on how to run commands on the VCSA remotely without enabling ssh is pretty awesome!   He utilizes the Guest Operations API through a vSphere SDK to do so!  Just go read it, it’s gold Jerry, it’s gold!  goes here it is

 #ChrisWahlFacts – He doesn’t mess around when it comes to dropping knowledge about PowerShell and REST APIs

apiOver yonder at WahlNetwork.com Chris Wahl has a great series going on dubbed Automation for Operations – Honestly the whole series is great but its the last 4 parts that have really helped me out…a lot!  Chris has done a great job at explaining a bunch of concepts around utilizing PowerShell to connect to RESTful API’s, including Authentication, Processing GET Requests, Sending data with POST/PUT/PATCH, and his latest, creating hashtables for JSON payloads!

Veeam Vanguard nominations are open!

veeam_vanguard-700x224Ever wake up and think “Hey, Why isn’t so and so a Veeam Vanguard?”  or “Why am I not a Veeam Vanguard?”  Well, so long as you wake up wondering about that before March 30th you have a chance to throw your name or someone whom you think is worthy into the mix!  You can check out the official Veeam post here.

vExpert stuff!

vmware-vexpert-2013We all know that being a vExpert isn’t about what you get, but more about what you give – buuuuutttt, the fact of the matter is you do get stuff, sometimes lots of stuff, and it’s hard to keep track of it all!  Thankfully for the vExpert community Andrea Mauro is doing a great job at keeping track of it all for you!  Without posts like this there is no way I’d be able to keep track of it!  So, Thanks Andrea!

Ravello on my wrist – Pebble in the cloud

You can probably get the gist as to what this post might be about by the title but it does leave a little to the imagination.  For those who hate waiting for the point go ahead an watch this small video…

Before I get right into what I’ve done let me first provide a little background information as to why I’ve done this aside from just looking for “something geeky to do”.

Ravello-Systems-LogoFirst up I’ve pretty much let everyone know how much I heart Ravello Systems .  Not to go too deep but as I build up labs and environments for this blog and for other interests I really like to break things.  Why?  That’s how I learn best, breaking things, fixing them, then writing them down.  The problem is I seem to always be rebuilding or fixing before I can move onto my next project.  Ravello solves that issue for me – with Ravello I’m able to keep multiple blueprints of completely configured vSphere labs (different versions, different hardware configs) in the cloud.  When I’m feeling geeky I can simply deploy one of these as an application to either Google or Amazon and away I go.  If I break it to the point of no return it’s no biggie, I can simply redeploy!  Point and case it’s a time-saver for me!

Secondly I love to write code – it’s an odd passion of mine but it’s something I actually went to school for and never 100% pursued.  Meaning I love to write code….casually!  I couldn’t imagine dedicating my whole career to it, but having the knowledge of how to do it casually sure has helped me with almost every position I’ve held.

pebbleThirdly a little while I ago I purchased a Pebble watch.  I’m still not sure why I wanted a smartwatch but I knew if I had one I’d want it to be somewhat “open” and Pebble met those needs.  Using a service called CloudPebble and by turning on the development mode on the iPhone app I’m able to deploy custom applications to my Pebble – so that was a big seller when I was looking at watches – oh, and the fact that it’s only like $100 helps as well…

So on to the problem – I mentioned I love Ravello and have multiple applications setup within the service.  The applications are great, however it takes a good amount of time after powering one on before you are able to start using it.  Those vSphere services need time to initialize and boot.  My usual routine involves me logging into Ravello and powering on what I might need for the night before I leave work.  That way the initialization can happen during my commute, supper with my family, and bedtime routines and is ready to go when I am.  There are times though when I get half way home and realize I forgot to power on my labs, or I’m not near a computer and can’t be bothered to use the small iPhone screen.

There’s an app for that!

PowerOnRavelloFor these reasons I decided to try and figure out the Ravello APIs and the Pebble SDK and see if it was possible to create a small application to simply login into Ravello, select an existing application, and power it on!  It sounds simple enough but took a lot of trial and error – I had no clue what I was doing but in the end I was left with the solution below – and it works so I guess you could call it a success.


There’s a few pieces that need to fall into place before any of this will work.  First up you wiill need a CloudPebble account.  CloudPebble is a development environment that allows us to write applications for use on the Pebble watch in either JavaScript or C.  You can use an existing Pebble account to log into CloudPebble or simply setup a new account – either way you need one and it’s free!

devmodeSecondly you will need to enable developer connections within the Pebble app on your phone.  This is easlily done by selecting ‘Developer’ within the main menu and sliding the switcher over.  Honestly, it’s a phone app I’m sure you can figure it out.

Thirdly lets go ahead and setup a project within CloudPebble You can do this by simply importing mine, or manually by giving your new project a name and select PebbleJS as your Project Type.  Once created you should be at a screen similar to that shown below…


As you can see we have one source file (app.js).  This is the only source file we will need for this project.  If you imported my project you are done for now, but if you created a new project manually this file will be full of a bunch of example code on how to perform various functions and respond to different events within the Pebble interface – we won’t need any of this so go ahead and delete all the code within the file, but not the file itself.  We will replace it with all of this syntax –  explained in the next section.

The code

If you simply just want all the code to go through on your own go head and get that here.  For the rest of us I’ll try and explain the different blocks of code below…

// import required libraries
var UI = require('ui');
var ajax = require('ajax');
var Vector2 = require('vector2');
var Vibe = require('ui/vibe');

Lines 1 through 5 simply deal with importing the libraries we will be working with – UI will give us access to the Pebble UI, Ajax is what we will use for the Ravello API calls, Vector2 for use with positioning on items on the watch, and Vibe is simply so we can access the vibration features of the watch.

// setup authentication information
var encodedLogin = "mybiglongencodedstring";
var expirationTimeInSeconds = 600; // timeout for app

Lines 8 and 9 set up a couple of variables for the application.  First up, encodedLogin represents a base64 encoded string of the username and password you use to login to Ravello, with a “:” between them.  You can grab this by heading to https://www.base64encode.org/ and grabbing the encoded string using UTF-8 as the output – just don’t forget to place the : between (ie. I encoded “mwpreston@myemail.com:supersecretpassword”).  Copy the result and place assign it to the encodedLogin variable on Line 8

Line 9 deals with our expiration time – When we power on an application within Ravello we need to specify an auto power off parameter which states how long we want before the application powers itself down.  You don’t want to use up all those valuable CPU hours right?  The variable defined on line 9 is matched to that, however in seconds so get your calculator out and come up with a number.

// main window
var splashWindow = new UI.Window();
var text = new UI.Text({
position: new Vector2(0,0),
size: new Vector2(144,168),
text: 'Logging into Ravello Sytems, please wait...',
// Add to splashWindow and show

Lines 11 through 25 simply define the first splash window we will see in the application.  Kind of a message to show the user as we are making the API calls and gathering the application lists.  You can start to see some of the Pebble object functions and parameters here…

As we move into ajax calls starting on Line 28 we can start to see the URLs and API calls to Ravello and how they are formatted when using PebbleJS.  From here each API call that is sent to Ravello is nested within the previous – this was the only way I could get this to work.   You can go ahead and read the docs on the ajax function here – I still don’t really completely understand the values being returned but hey, it works!

Anyways, back to the task at hand – As shown below lines 28-30 makes are login request, passing basic authorization and our encodedLogin variable within the header.  After parsing the response on Line 34 we display yet another splash screen (Lines 35-44) with a success.

//login to Ravello
ajax({ url: 'https://cloud.ravellosystems.com/api/v1/login',method: 'post',
headers: {Authorization: "Basic " + encodedlogin,
Accept: "application/json"}
function(data,status,obj) {
// success into Ravello
var contents = JSON.parse(data);
var text2 = new UI.Text({
position: new Vector2(0,0),
size: new Vector2(144,168),
text: 'Hello ' + contents.name + ', you are now logged in! - Fetching applications, please wait...',

Another API call, this one to gather our application lists takes place on lines 46 and 47.  From there lines 51 through 74 build a menu to display the application listing, hide our previous screen, and display our newly formed menu.

ajax({ url: 'https://cloud.ravellosystems.com/api/v1/applications',method: 'get',
headers: {Accept: "application/json"}
// success application list
var apps = JSON.parse(data); 
var count = Object.keys(apps).length;
var menuItems = [];
var appname;
var appid;
for(var i = 0; i < count; i++) {
appname = apps[i].name;
appid = apps[i].id;
// Construct Application menu to show to user
var resultsMenu = new UI.Menu({
sections: [{
title: 'My Applications',
items: menuItems
// Show the Menu, hide the splash

At this point we are waiting on user interaction – the user needs to select the application they want powered on.  Line 76 defines that exact event listener, triggered once the user hits the select button.

// Add an action for SELECT
resultsMenu.on('select', function(e) {
console.log('Item number ' + e.itemIndex + ' was pressed!');
// this is where magic happens and we translate which item was pressed into turning on applications
var detailCard = new UI.Card
{title: "Starting Lab", subtitle: e.item.title }
detailCard.body('Setting lab power off time to ' + expirationTimeInSeconds.toString() + ' seconds...');
var ExpURL = 'https://cloud.ravellosystems.com/api/v1/applications/'+e.item.subtitle+'/setExpiration';
// set expiration time for selected application
var expbody = { "expirationFromNowSeconds": + expirationTimeInSeconds };
url: ExpURL,type: "json",method: "post",headers: { Accept: "application/json" }, data: expbody
// success setting expiration time
detailCard.body('Setting lab power off time to ' + expirationTimeInSeconds.toString() + ' seconds...'+ 'DONE!\nPowering on lab...');
var StartURL = 'https://cloud.ravellosystems.com/api/v1/applications/'+e.item.subtitle+'/start';
url: StartURL,type: "json",method:"post",headers: {Accept: "application/json"}

Once an application is selected lines 78 through 84 display some status messages as to what is happening, and beginning on line 89 we start the API calls to power on the application.  First (Line 92) sets the expiration time for the selected application.  Then, line 102 sends the actual Power Up command to Ravello.

// success starting application
console.log("Success on start:" + status);
detailCard.body('Setting lab power off time to ' + expirationTimeInSeconds.toString() + ' seconds...'+ 'DONE!\nPowering on lab...' + 'DONE!\nLab Powered On' );

Lines 108 and 109 simply display some success messages to the user and send a short vibrate command to the Pebble watch.

I’ve done my best to explain the code – it’s probably not the cleanest or best way to do all this but guess what?  I can power on a Ravello application from my watch so that’s all that matters…  Please feel free to steal all the code if you want it – or here is the complete CloudPebble project if you are in a hurry and just want to skip the copy/paste.  I’d love any feedback anyone may have on this.  For now this is where the project sits but I’d love to expand it further and integrate with more of the Ravello APIs available.  At the moment I’m happy with powering on my Ravello labs from my wrist!

Ravello Systems – Inception without the kick!

Ravello-Systems-LogoIf you have at all visited this blog in the last 4 or so months you shouldn’t be surprised to hear that I’m a pretty big Ravello Systems fan!  I was part of their beta for nested ESXi and I’ve wrote about my thoughts on that plenty of times.  With the beta out of the way and access granted to all the vExperts, Ravello Systems took hold of the clicker at VFD5 in Boston for their first of what I hope is many Tech Field Day presentations.

Disclaimer: As a Virtualization Field Day 5 delegate all of my flight, travel, accommodations, eats, and drinks are paid for. However I did not receive any compensation nor am I required to write anything in regards to the event or the sponsors. I  have also been granted early access to Ravello Systems ESXi beta in the past as well, and have received free access as a vExpert.  All that said, this is done at my own discretion.

As I mentioned earlier I’ve written plenty about what I’ve done utilizing Ravello Systems.  The platform is great for configuration validations, home lab type stuff, and for just exploring different functionality within vSphere.   You know, user type stuff.  At VFD5 Ravello went pretty deep in regards to how their software functions within Google and AWS, so I thought I’d take a different approach and try and dive a little deeper into how their technology functions this time around…to the point that my brain started to hurt.

HVX – A hypervisor that runs hypervisors, designed to run on a hypervisor – huh?!?!

Ravello’s magic sauce, HVX is custom built from the ground up to be a high performance hypervisor to run applications (and other hypervisors) while in itself running on a hypervisor (in public cloud).  To say Ravello would know a thing or two about developing a hypervisor would be a major understatement – Ravello’s co-founders, Benny Schnaider and Rami Tamir were once the co-founders of another start-up called Qumranet.  You know, the same Qumranet that originally authored this little known thing called the Kernel-based Virtual Machine, or better known as….yeah, KVM.  So needless to say that have a little experience in the hypervisor world.

The first dream within a dream

As we know Amazon’s EC2 is essentially an instance of Xen, where-as Google’s Cloud utilizes KVM.  So when we publish our application inside of Ravello we essentially deploy an instance of HVX, installed within a VM that has been spun up on either Xen or KVM – once our HVX hypervisor has been instantiated on our cloud hypervisor, then our images or VMs within Ravello are deployed on top of HVX.  So even without yet touching ESXi within Ravello we are 2 levels deep!  Now in terms of a native ESXi deployment we know that we can take advantage of common virtualization extensions such as Intel-VT and AMD SVM, however in HVX, since we have already been abstracted away from the hardware by the Cloud hypervisor we don’t have these – instead, HVX implements a technology called Binary Translation to translate any executable code from the guests that is deemed “unsafe”.  Coupling this with something called Direct Execution, which basically allows any code that need not be translated to run directly on the CPU.  Honestly, if you want to dive deeper into binary translation and direct execution Ravello has a great blog outlining it in a lot more detail than can fit into my maple syrup soiled, hockey statistic filled Canadian brain.  Aside from the performance features, HVX also passes up emulated hardware – the same hardware that we as VMware administrators are all used to – things like PVSCSI, VMXNet3, LSI, etc – this is all available to our guests running on top of HVX, even to our guests running on top of our ESXi guests on top of HVX – I know right!


So, what actually happens when we click that ‘publish’ button from within the Ravello interface is somewhat unique – we know we need to install HVX into our cloud VM but how many instances of HVX actually get deployed?  I’m not going to try and understand their algorithms around how they size their hypervisor but I’m just going to say it depends on the resource allocation on the VMs within your application.  You could end up with a single VM running on one instance of HVX or you could end up with 6 VMs running on 2 instances of HVX – however the deployment scenario plays out you can be ensured that it will in only be VMs belonging to that single application that get deployed on the HVX instances – no VMs from other peoples applications, not even any VMs from other applications that you may have.

That networking though!

Perhaps one of Ravello’s major strong points is how it exposes a complete L2 network to the applications running on top of it!  By that I mean we have access to everything L2 provides, we have services available such as VLANS, broadcasting, multicasting, etc within the overlay network Ravello implements.  As we mentioned before, depending on the size of the application being deployed, we may or may not have multiple instances of HVX instantiated within the cloud provider.  If we are limited to a single HVX instance, then the networking is “simple” in terms that it doesn’t have to leave their hypervisor – all switching, routing, etc can be performed within the one HVX instance.  However when an application spans multiple HVX instances creative technologies come into play as shown below.  Ravello has essentially built their own distributed virtual switching mechanism which can tunnel the traffic between HVX instances or Cloud VMs via UDP connectivity.


And storage…

The last challenge as it pertains to running Ravello applications inside the cloud comes in terms of storage performance.  Having HVX slotted in-between the the running applications and AWS allows Ravello to take advantages of the object storage capabilities of S3, but yet still present the underlying storage to the VMs as a block device.  Essentially, when we import a VM into Ravello Systems, it’s stored in it’s native format on top of HVX and that appears to be a block device, but under the covers the HVX file system is essentially storing this information in object storage.  Aside from all this abstraction HVX implements a Copy-on-write file system, delaying the actual allocation of storage until it is absolutely needed – in then end we are left with the ability to take very fast snapshots of our images and applications we deploy, easily duplicating environments and allowing people like myself to “frequently mess things up Smile


The Ravello presentation at VFD5 was one of my favorites from a technology standpoint – they did a great job outlining just what it is they do, how they do it, and how they are chosing to deliver their solution.  There was some questions around performance that were met head on with a whiteboard and overall it was a great couple hours.  Certainly check out some of the other great community posts below centered around Ravello to get some more nested goodness..

Ravello has a great product which honestly completely blows my mind when I try and wrap my head around it – We have our VMs, running on ESXi, running on HVX, running on Xen, running on some piece of physical hardware inside an Amazon data center – attaching to both Amazon EBS and S3 – we are snapshotting these things, saving as blueprints, redeploying to Google Clouds which completely flip the underlying storage and hypervisor!!  It’s exporting VMs out from our current vSphere environments and deploying them into the public cloud, complete with all of their underlying networking – already setup for you!  Ravello has coined their nested virtualization capabilities as Inception, and if you have ever seen the movie I’d say it certainly lives up to the name.  It has this magic about it – where you are in so deep yet still in control.  If you have a chance check out their VFD5 videos and sign up for a free trial to check them out for yourself.

#VFD5 Preview–Ravello Systems

v2Ravello_Logo_large-wpcf_100x27Ravello Systems have certainly had there fair share of buzz lately and rightly so – the shear fact that you can run a 64 bit VM, on top of a nested ESXi host, on top of their hypervisor (HVX), on either Amazon or Google Cloud is to say at the least – the bomb!

Disclaimer: As a Virtualization Field Day 5 delegate all of my flight, travel, accommodations, eats, and drinks are paid for. However I do not receive any compensation nor am I required to write anything in regards to the event or the sponsors. This is done at my own discretion.

I’ve had the chance to work with Ravello during their nested ESXi beta along with a few other bloggers and was blown away by the performance they provided while doing the exact scenario as described above.  I did a few posts on Ravello, one which involved a vMotion from Amazon AWS to Google Cloud if you’d like to check it out!  Needless to say I’m excited to see Ravello IRL at VFD5 on June 26 in Boston.  Also, I’ve heard through the grapevine that long-time Toronto VMUG attendee and friend Kyle Bassett will be part of the presentation – Kyle is a brilliant mind so you won’t want to miss it!

A home lab replacement?

In a lot of ways I can get the performance that I need in order to replace my home lab!  That said I’m no where near as extravegant when it comes to homelabs as a lot of people in these communities.  When it comes down to it though, a lot that I do within the lab is configuration validation, testing different setups, etc.  All of this is easily accomplished in Ravello!  In fact in some ways I can do a lot more within Ravello then I can within my own home lab.  Stringing together two datacenters, one in Google, one in Amazon via VXLAN for example!  For the most part I’m finding myself working more in cloud platforms than in my basement any more.

Bells and whistles

I would be selling Ravello short if I just said they allowed you to run nested ESXi in Amazon – they have a lot of value add, bells and whistles so to speak that make the service what it is.

Firstly they have what’s called an application – an application is essentially one or more VMs that perform some sort of function.  You could think of a couple ESXi hosts, a vCenter Server and some sort of iSCSI storage appliance as an application.  Applications and be started and stopped as whole unit, rather than each individual VM.

Secondly they have blueprints.  We can think of a blueprint as a point in time snapshot of any application.  Basically, blueprints allow you to save a configuration of an application to your library, which you can then deploy to either another application or another cloud.  Think of a blueprint being a base install of your ESXi/vCenter setup – you know before you go mangling inside of it.  If your original application ever breaks, or you’d like to explore new features without affecting your current setup you could simply save your application as a blueprint and deploy a new instance of it.  One newly released feature is the Ravello Repo, which allows customers to essentially share their blueprints with others, saving a lot of time when it comes to building up test and use cases.

Thirdly is pricing!  Honestly I’m not sure what hard costs I’ve incurred as I have gotten 1000 CPU hours/month for free – If you are a vExpert you can too as well as they have just extended this offer to all vExperts – very generous!  Not a vExpert, no problem, you can still get a free fully functioning trial here, good for 14 days worth of all you can eat cloud.  Although I’ve never seen my pricing I have looked at their pricing calculator – selecting 12 vCPU’s, 20GB of RAM and a TB of storage it comes out to around $1.32/hour – which too me is more than enough resources to get a small lab up and running and is more than affordable for what you get.  Plus you don’t deal with Amazon or Google at all – Ravello takes care of all of that.

What Ravello has in store for us on June 26 we don’t know, but I can assure you that it will be a treat to watch.  Speaking of watching, if you want to follow along with all the action you can do so by watching the live stream on the Tech Field Day page or on my VFD5 event page where all my content will live.

VXLAN on Ravello between Google and Amazon EC2

v2Ravello_Logo_largeA week or so ago I did a post around a cross vCenter vMotion lab that I had setup utilizing both Amazon EC2 and Google Cloud through Ravello Systems new beta which allows us to run nested ESXi.  It was a fun project to work on, migrating VMs back and forth through the clouds, but I tried to keep a lot of the technical detail out of the post – focusing more on what Ravello had to offer.  One key aspect of the setup was creating a VXLAN tunnel in order to bridge the two VM networks inside of each cloud – allowing me to complete the vMotion without performing any additional network configuration on my VMs once they had migrated.  Anyways, I thought I’d go into a little more detail on how I accomplished this.

Now, keep in mind here I don’t claim to be any sort of network guy – it’s probably my biggest lack in terms of IT skill-sets, therefore I could be going about this all wrong – any advice, comments, just leave the in the box below – I always appreciate any feedback I get.  I also appreciate any help I get, and had a lot with this project – CTO of Ravello Systems Alex Fishman had a couple of calls with me offering up his experience (very very smart guy).  Also, there’s a blog post on Ravello’s blog which goes over the setup as well – that said, I thought go into a little more detail in the case that someone else at the same level of network knowledge as myself might be looking for help.

So to start let’s have a brief look at the Ravello setup.  Firstly we need a couple of applications, one published to the EC2 cloud and one published to the Google Cloud.  Each application (in this case) contains two VMs – one to act as a client and one to act as the gateway/vxlan bridge.  I’ve used Ubuntu 14.04 server for these but you could use any OS you like, so long as the vxlan module is loaded and supported.  The table below outlines each VM and the networking associated with it in my test setup.

Cloud VM Network IP Notes
Amazon EC2 ec2-vxlan eth0 IP:
GW: None
Network Gateway
eth1 IP:
External network w/ Elastic IP attached
ec2-client eth0 IP:
GW: (in OS) None in Ravello
Google google-vxlan eth0 IP:
GW: None
Internal Network Gateway
eth1 IP:
External Network w/ Elastic IP attached
google-client eth0 IP:
GW: (in OS) None in Ravello

I’ve used a /29 subnet on the external networks as I really only need 3 total IPs available, one for each of the vxlan VMs as well as a third for a gateway – Honestly again you could use whatever you wanted here.  I understand that sometimes a picture is worth a thousand words so here is a side by side of both the Amazon and Google network canvas.

ec2networking googlenet

So a pretty simply setup when looking at the canvas – essentially the vxlan VM will need two NICs, one connected to the internal lan (eth0 in this case) and one connected to an externally routed network (eth1).  Before we finish up within the Ravello canvas and establish a tunnel let’s first look at the EC2 side of things so we can better understand the settings on the vxlan and client VMs that end up making our network canvas look like the above.

ec2-nic1 ec2-nic2

Looking closer at the network configuration of the ec2-vxlan VM (sorry, couldn’t get it all on one screen cap so I put them side-by-side) we can notice a couple of things; eth0, the internal lan (which will act as a gateway for the client VMs) is setup on the network with no gateway.  Also, we have selected public IP for this nic under external access but left the ‘Even without external services’ unchecked.  What this does is ensure that this VM can only be accessed by routing through our vxlan tunnel and cannot be accessed directly from the internet.  The second nic (eth1) is the nic we will use to establish our vxlan bridge.  This nic is subnetted in a way that allows very few IPs within the network, as we only need three anyways (one for ec2, one for google, and one for a gateway).  This nic has an Elastic IP tied to it within the External Access settings.  Since we will need to use this public IP later when we establish the tunnel it’s best that it not change often, and an elastic IP will never change, thus why we used it.


Another note in regards to the vxlan VM is the external services provided.  For this case I’ve simply allowed all external traffic on all ports into eth1 – probably not the greatest feat in terms of security but it sure does ensure I get the communication I need. (might be able to change to just other IP).

Note that you will need to setup the Google side of things inside Ravello exactly the same as shown above, obviously replacing the IP addresses of eth1 with those shown in the table earlier.  Once done with that we have completed our setup within Ravello and it’s now time to setup the tunnel.

Again, let’s start with EC2 and work our way over to google.  Take note of both your EC2 and Google elastic IP’s as we will need them for the configuration below.

First is our network configuration (/etc./network/interfaces).  Below is a shot of mine – the key here is that even though we specified an IP in the Ravello interface for eth0 we are not doing it within Ubuntu – we will be using this IP on our bridge instead, however we still want eth0 to be active so we set it as manual.  So as far as network interfaces your setup should be similar to below – of course with address being on the Google cloud.

# The primary network interface
auto eth0
iface eth0 inet manual
auto eth1
iface eth1 inet static

Once we have our network interfaces set up we can continue with the setup of the vxlan and bridge.  First we will walk through the EC2 side and then move to google.

We begin by adding the vxlan device with “ip link add mtu <MTUSIZE> <VXLAN DEVICE NAME> type vxlan id <VXLAN ID>”.  For example on the EC2 side I ran..

ip link add mtu 65000 vxlan1 type vxlan id 1

Then let’s create a new forwarding database entry on our vxlan device to allow all traffic through using “bridge fdb append <MAC> dev <VXLAN DEVICE NAME> dst <DESTINATION ADDRESS>”.  For example, I used the following on my EC2 instance – ensuring I use the public IP of the Google Cloud.

bridge fdb append 00:00:00:00:00:00 dev vxlan1 dst

From here we can add our bridge using “brctl addbr <BRIDGE NAME>” – again, I ran…

brctl addbr br0

Then add both our vxlan and our internal interface to the newly created bridge with “brctl addif <BRIDGE NAME> <INTERFACE> <INTERFACE>”

brctl addif br0 vxlan1 eth0

Now we can assign our internal IP that we wish to use for the internal lans gateway to our bridge, and then simply bring up our bridge and vxlan interfaces.  Lastly, I ran these commands…

ifconfig br0
ifconfig br0 up
ifconfig vxlan1 up

That does it for the Amazon configuration, the Google configuration is exactly the same, except substituting the Amazon public IP as our destination when adding the forwarding entry.  Once we do this you should be able to ping back and forth between the test client VMs, each located in their separate cloud.  Now think of the possibilities – nested ESXi in each cloud, with a layer 2 VM Network that stretches through the vxlan tunnel – pretty sweet stuff!

One thing that could be a PIA is that this config won’t persist across reboots.  I’ve tried in many ways to get the interfaces added to the persistent rules, but found the quickest way to get this to run on boot is to simply create a small bash script containing the commands; both Google and Amazons are shown below.

ec2script googlescript

Save each bash file in their respective /etc./init.d/ directories and make them executable.  I called the configVXLAN Then run the following on both Amazon and Google to ensure it gets ran on startup

update-rc.d /etc/init.d/configVXLAN defaults 99

And that is it!  A functioning stretched Layer 2 network between Google Cloud and Amazon EC2 using Ravello!  The possibilities are endless…  Again, I’m not a big networking guy, so if you know of any way I can improve this (use NSX?) just let me know…  Thanks for reading!

A Google Cloud to Amazon vMotion – The Ravello Way!

v2Ravello_Logo_largeToday Ravello Systems, a company based out of Palo Alto and Israel announced a new beta, a beta that I think is going to go over very well within the VMware community – one that will allow us to spin up vSphere labs, complete with vCenter Server, ESXi hosts, Domain Controllers, Storage and Network services and all the VMs that go with the punch inside of Google and Amazon’s cloud.  To be honest I was kind of skeptical when I first started working with Ravello?  I mean, come on, an ESXi host in Amazon, let alone and ESXi host running VMs inside of Amazon, an ESXi host running VMs with little to no performance penalty, all running within Amazon – you can see why I might of cringed a bit.  But Ravello gave me a shot to try it for myself – and during the introductory chat as they were showing me how things worked I thought, hey, what a use case for the new cross vCenter vMotion capabilities in vSphere 6!  A lab in Amazon, a lab in Google Cloud, and VMs migrating between them – how cool is that?

Who and what is Ravello Systems?

Now, before I get into the details of the vMotion itself I want to take a step back and explain a little bit about Ravello Systems themselves, and what they have to offer.  Ravello was founded in 2011 with the sole purpose of supporting and driving nested virtualization to the next frontier and did so when they launched their product globally in August of 2013 (You had to of seen the scooters at VMworld 🙂 )  They didn’t just want to simply provide an environment for nested virtualization though, they wanted to make it simple and easy for companies to replicate their data center infrastructure into the public cloud.  The core technology behind all of this is their HVX hypervisor – essentially acting as a Cloud VM, sitting in either Amazon or Google and providing overlay networking and storage to the VMs that are placed on top of it.


As per the diagram above the VMs present can be built from scratch or imported via an OVA within Ravello’s very easy to use intuitive interface – but perhaps more interestingly you can utilize the Ravello Import Tool(??), point it to your ESXi host or vCenter, and import VMs directly from your environment into the cloud!  But they don’t stop there, Ravello can also detect and create every network your VM is attached to, deploying an exact duplicate of your network infrastructure!  Now if this wasn’t good enough for you the beta today announces the ability to support Intel VT through HVX – which means we can now run VMs on top of ESXi on top of HVX on top of Amazon or Google!  True inception leaving us with a setup shown in the diagram below.


A great place to break things!

There is a reason why Ravello dubs their technology as having the ability to create “Smart Labs”!  Throughout my early access to the solution I broke and fixed so many things within my applications – and Ravello always gave me a way to rebuild or reconstruct my labs in a very efficient manner.

RavelloSaveToLibraryFirst up we are able to save our VMs to the library – which is essentially a personal set of VMs and images that we can re-use in all of our applications.  For example I only had to build my ESXi 6.0 image once – after saving this to the library I was able to simply drag and drop this VM as many times as needed to as many applications as needed, simply re-ip and re-naming after I was done.

RavelloSaveToBlueprintHaving the ability to re-use VMs is cool but the blueprint functionality that Ravello provides is really where I see value!  We are able to take a complete application, in my instance an ESXi host, domain controller, vCenter Server, etc and save the entire application as a blueprint.  Blueprints are then available to be used as starting points for new applications – meaning I can build a complete lab on Amazon, save as a blueprint, and then publish a new application to Google which is an exact identical copy, networks and all.  Blueprints are an excellent way to test out the different public clouds as well as version or snapshot your entire lab before making any major changes – if things go awry you can simply republish your saved blueprint to a new application.


Enough talk – Let’s see the vMotion!

Alright!  Let’s get to it!  Let me first warn you, the environment I built to do this was quick and dirty – not a lot of polishing going on here.

The two applications we will be using are Google-vxlan and EC2-vxlan – I’ll let you guess which public clouds each is published to.


As shown above these applications are pretty similar; each containing an Ubuntu server (used to establish the vxlan tunnel between EC2 and Google), a pfSense appliance that provides a VPN for my vMotion networks, a vCenter Server (the Windows version), and an ESXi host (just one for now).  The EC2 application also contains a jumpbox VM which provides entry into the local network as well as DNS services.


As far as networking goes the setup at both Amazon and Google is almost identical with the exception of the jumpbox.  The network is available at both EC2 and Google.  The network is the only network that is routed to the internet, therefore my only access into the labs outside of the Ravello GUI – this is why the jumpbox also has a connection to this network – to act as an RDP gateway of sorts.  The two Ubuntu servers have an elastic public IP attached to them in order to ensure the public IP doesn’t change and mess up my vxlan config.  The free trial of Ravello gives you two elastic IPs, and four other DCHP public IPs (subject to changing every now and then).  The vxlan tunnel is established between the two elastic IPs in order to provide Layer 2 connectivity between Amazon and Google.  The pfSense boxes each have a dynamic public IP attached to them with an IPSEC tunnel established between the and the networks.

vsphereshotOn the VMware side of things I have two vCenters with embedded PSCs (i know – bad practice) – one in Amazon and one in Google, which are attached to the same SSO domain and configured in Enhanced Linked Mode.  Therefore whatever is at Google can be seen at Amazon and vice versa.  As far as vMotion goes I’ve simply enabled this one my existing management interfaces (more bad practice – but hey, it’s a lab).  There is local storage attached to the ESXi hosts and one VM named EC2-VM1 present.

So my goal was to migrate this VM from Amazon to Google and back again, taking both the compute and storage with it.  Now just writing about a vMotion is not that exciting so I included a video below so you too can see it move 🙂  It’s my first attempt at a video and had some screaming kids while I made it so yeah, no narration – I’ll try and update with a little tour of the Ravello environment later 🙂

So there you have it – a VM moving from Amazon to Google and back, all while maintaining its’ ping response – pretty cool!

Is Ravello worth it?

esxi-home-labSo, with all this the question now remains is Ravello worth the cost?  Well, considering as how Ravello estimates the cost of a two ESXi Node, vCenter and Storage lab to be on average $0.81 – $1.71 per hour (usage based, no up front costs) I would certainly say it is!  The ability to run nested ESXi hosts on top of the public cloud provides a multitude of use-cases for businesses – but honestly I see this being a valuable tools for the community.  I plan on using Ravello solely for my home lab usage over the next year or so – it’s just so much nicer to break things and simply re-publish an application than it is to try and rebuild my lab at home.  If you want to give Ravello a shot you can sign up for the beta here – Even after the beta expires you simply swipe your credit card and pay Ravello directly – no Amazon accounts, no Google bills – just Ravello!  You will be limited during the beta’s and free trials in the amount of CPU, RAM and concurrent powered on VMs but they definitely give you enough resources to get a decent lab setup.

Ravello has a great solution and certainly expect more from me in regards to my lab adventures in the public cloud.

Disclaimer: Ravello gave me early access to their ESXi beta in order to evaluate their solution – I first signed up for a free trial and did get the amount of RAM and number of VMs available to me increased.  They didn’t however require that I write this post nor write anything for that matter, or provide any $$$ for anything that was written – these are all my words!