Friday Shorts – #vDM30in30, AWS, Reading, Writing, AWS and more…

Would it kill you not to be so funny all the time? That’s all I’m askin’. This woman thinks I’m very funny and now you’re gonna be funny, so what am I gonna be? I’m gonna be a short bald guy with glasses who suddenly doesn’t seem so funny. – George Costanza

VMworld is over – vSphere 6.5 is near

VMware LogoWith VMworld EMEA sweeping up the remains of the solutions exchange the dawn of vSphere 6.5 is here!  And with that comes a multitude – and I mean a crap load of multitudes of blogs talking about all of their favourite features of the release to come!  With that said I can’t possible link to all of them that I have read – but what I will link to is to the linkmaster himself Eric Siebert!  He’s got a great roundup post here of all of the features within vSphere 6.5.  Now this post doesn’t have links, just quickly lists the features – but if you want to go deeper on a certain feature be sure to check out his ‘vSphere 6.5 Link-O-Rama’ page as well!

Renting vSphere on AWS

vmware_awsVMware and AWS announced a partnership – one which would allow vSphere to run on the Amazon cloud!  What this means for customers is that they will get the familiarity of vSphere and all it has to offer, but on a cloud-like scale, running on bare-metal within Amazon’s datacenters.  What’s in it for VMware – well, they get that global cloud they have been looking for, without having to actually build the datacenters behind it!  And Amazon – well, who knows where they will take it but I can’t see them complaining too much about being able to pitch certain services and products to those who chose to run VMware on AWS!  Anyways, Frank Denneman has a great article here taking a deeper look into everything – as well, a little bit of a preview on how the new service enables one pretty cool new feature – Elastic DRS.

Alastair teaches us to read!

Although this post is a couple months old I’m just getting around to sharing it now!  I wanted to be sure as I’m always interested in how everyone in this community stays current with so much going on all the time!  The amount of content getting created for and by the tech community is huge, and keeping track of it all is certainly becoming a skill!  I read a lot, a lot of blogs so any way of figuring out ways to streamline my content consumption always sparks my interest!  Alastair has already taken us through is process of creating content, but with this post he takes us through his process of consuming content!  This is a great post with excellent tips on how to stay current!

More than simply writing a file

There’s a guest blog post on Michael White’s blog written by Michael Cade outlining the various backup methods used within Veeam Backup & Replication and how each one of them affects the capacity and performance on your backup repositories.   If you are using VBR and you haven’t had an in-depth look at these backup methods I would suggest starting with this post – it does a great job at outlining just how each backup method works, what is actually contained within each VBR backup file type.  It’s not just simply copying data here 🙂

#VDM30in30 30 posts, 30 days!

Think you have what it takes to pump out 30 posts in 30 days – I can say that I certainly thought I did at one point in time, but quickly realized after just a few days that I most certainly did not!  But hey, kudos for trying right – and even though I have a lot going on this November I will most likely try again!  Don’t have a clue what I’m talking about – well, the folks from the Virtual Design Master competition are once again challenging the blogging community to crank out 30 posts in 30 days throughout the month of November!  Have a look at Eric Wright’s blog for a great description of what this is like!  Now I know some people may think that the quality of posts may not be there, or that the content won’t be relevant, but honestly, my blogging isn’t about perfecting every single thing I do (I’m sure that’s apparent).  It’s about sharing things I do, problems I run into, opinions I have – sometimes it may be relevant, sometimes it won’t – sometimes you may agree, sometimes you won’t – sometimes you may just not even understand why a post ends up on this blog!   To me writing is about passion, and whether that passion be virtualization, technology, canadian folk music, maple syrup, hunting, whatever, it doesn’t matter!   The point here is that when I write, I think –  and when I think I learn – and nobody can argue that learning is invaluable!  So heads up, you might see some odd posts here come this November!

Tech Field Day 12 – Goin’ back to Cali!

kangolYes Mr Cool J you heard that right – Although you don’t ‘think’ you are heading back, this guy is indeed ‘Goin back to Cali!’  While Mr Cool J would rather stay in New York I’m heading to Silicon Valley to partake in Tech Field Day 12 with a slew of great delegates and sponsors alike!  This will be my first time in the Valley – so I’m pretty pumped to say the least!  I’m excited to finally be in the heart of all of the companies and technologies that I’ve been using my whole life, and writing about here for the past 5 years or so!

TFD what?

So if you haven’t heard of Tech Field Day then you have most certainly been missing out!  TFD is the brainchild of Stephen Foskett and his company Gestalt IT and is essentially a learning resource for the community.  Now I know, I know, there is already many many resources out there for us to find out about certain technologies or companies – we have white papers, books, blogs, videos, training, etc – but the problem is most of this stuff usually stems from strong marketing roots, and at times, can be a bit overwhelming trying to weed out the message from the technology!  TFD solves this by deep diving into the technology, and by placing a dozen or so tech minded folks in a room with a vendor it helps to keep the presentations and messages on point – it’s about the technology, not the marketing!  You know when you are sitting through a webinar or a presentation and someone poses a question – and said question is responded to with a “I’ll connect you with an SE or with someone afterwards to talk” – this kind of stuff doesn’t really happen at TFD – most the time, vendors and companies presenting have the knowledge and the resources in the room to leave no question unanswered – that’s what I like to think TFD is!

logo-cohesity-dark-100x12 dellemc_logo_prm_blue_gry_rgb-100x18 docker-logo-300-71x60 drivescale_logo-100x27
igneousiologo-100x34 1000px-intel-logo-svg_-91x60 logo-large-gray storage-os-logo-100x21

Anyways, so yeah, the Valley – so excited for this!!  Tech Field Day 12 has a number of great sponsors and vendors lined up to present at the event (you can see them above).  Some of these companies are giants (Dell EMC, Intel), some fairly new to the market (Rubrik, Cohesity), some are all the rage right now (Docker), and honestly some I’ve never dealt with or even really heard of (StorageOS, DriveScale, Igneous).  It’s normally the latter that really impress me at these events!  So heads up, the time is near – TFD12 airs November 15th and 16th with two jam packed days!  To learn more about the event, certainly check out the official landing page!

As I have with the other TFD events I’ve participated in I’ll try to consolidate all of my content surrounding the event on a single page, which you can find here!  A huge thanks to Gestalt IT for having me back!  I can’t wait!  Oh, and sorry for the 90’s hip hop references – it was as witty as I could get at the moment 🙂  Either way, I can almost hear those scratching records and that crazy jazz music which kicked of the song right now 🙂

VMCE v9 Study Guide – Module 10 -Veeam ONE Auto Discovery, Business View and Alarms

VMCE LogoVeeam ONE is essentially a management program, providing monitoring, reporting and alerting for your virtual infrastructure, whether that be Hyper-V or VMware.  The module for this course is broken down into 9 different sub modules, so lets look at each one in turn.


Auto Discovery of Backup and Virtual Infrastructure

During the initial configuration of Veeam ONE users are prompted to chose and add their virtual infrastructure type – the options presented are

  • VMware vCenter Server
  • Hyper-V Host, Failover Cluster or SCVMM server
  • Skip Virtual infrastructure configuration

After selecting your desired virtual infrastructure you will be prompted for it’s name, either IP or FQDN, and some credentials to connect – along with the required port to connect on.  After installation we have the ability add other types of servers such as a standalone ESXi host and a vCloud Director server on the inventory pane inside the Infrastructure view.

Once completed the initial configuration, the servers, both vSphere, Hyper-V or Veeam that you have setup during the installation will be propagated to all Veeam ONE components, such as monitor, reporter and business view and data will begin being collected  immediately as well as during its default scheduled time (weekdays @ 3 am).

Once a connection has been established Veeam ONE will start pulling information and data on the top level scope you’ve added, plus all children.  Individual hosts/clusters can be excluded after the face from the Monitored Hosts tab.  Subsequently, datastores can be excluded as well on the Monitored Datastores tab.  If you want to exclude certain VMs it is done a little differently.  Instead we use rules to exclude VMs.  By default an inclusion rule that adds all VMs is configured.  If you want to establish another inclusion rule, then you must disable this default rule.  You do not need to do this when creating exclusion rules.  You can create rules based on the following criteria.

  • By object name – add/exclude VMs based on VM name
  • By infrastructure location – Apply rule that applies to VMs only within a certain hierarchy of the infrastructure.

When creating our rules we can use the * and/or the ? wildcard.  The * stands for zero or more characters where the ? stands for just a single character.  If adding multiple conditions to a VM you need to specify whether you would like to apply the rule if any condition evaluates to true or if all conditions evaluate to true.

When prompted to add your Backup Infrastructure you simply need to pass the FQDN Name of either your Veeam Backup & Replication server or your Veeam Backup Enterprise Manager server and pass the required credentials.  You also have the ability to skip this step and add Veeam later.   If we connect a Backup & Replication server, data for the job sessions for the previous week is collected.  If we add an Enterprise Manager instance, Veeam ONE first builds a hierarchy of Veeam B&R servers, then collects the data for the job sessions for the previous week from each one of them.

Business Categorization

Categorization within Veeam ONE is done by a the Business View component which is installed alongside everything.  Business view allows you to create categorization models and values to help organize your infrastructure and display your infrastructure from a business standpoint.  This completely integrates into Veeam ONE Monitor and Reporter, so any groups you create within Business View will be available in the other Veeam ONE components.

Some examples of categories one might create in Veeam ONE Business View are…

  • Business Unit,
  • Department
  • Location
  • Purpose
  • SLA

Business View also supports reading and writing vSphere tags as well, meaning you can map business view categories and groups to vSphere tags and vice versa.  That said there are some default out of the box categories that Business View comes with..

  • Datastore – groups VMs by their datastore
  • Last Backup Date – dynamically groups VMs by the age of the latest backup or replica restore point created
  • Sample Business View Category – dynamically groups VMs by name
  • SLA – static groups for all types of virtual infrastructure objects.  Includes two groups; Mission Critical and Other
  • Storage Type – dynamically groups storage objects by type.
  • VM Network – dynamically groups VMs by connected network
  • VMs with Snapshots – dynamically groups VMs with snapshots by snapshot age.

You can add more custom categories, but keep in mind the maximum number of categories Veeam ONE supports is 25.

Pre-Defined Alerting

Veeam ONE Monitor comes with over 150 predefined alarms that can alert you on almost every aspect of your virtual and backup infrastructures.  In terms of data protection there are alarms already setup to…

  • connectivity issues an inability of backup infrastructure components to communicate with each other.
  • state of the VBR software installed on infrastructure components
  • Failing of jobs or jobs completed with warnings
  • Configuration issues, such as repositories running out of space
  • Long running jobs that exceed the backup window
  • License and pre-paid support expirations.

Alarms within Veeam ONE work in the following way

  • When Veeam ONE monitor detects that the behavior or state of an object meets an alarm criteria, it triggers the alarm with the corresponding severity level
  • Once triggered, Monitor console will display the alarm details and associated information in regards to the alarm.  At this point you can view, acknowledge, or resolve the alarm
  • After the alarm has fired, Monitor performs a responsive action; email, SNMP, and/or running a custom script.
  • Once the alarm has been resolved, Monitor updates the alarm status within the console.
  • If the state or condition returns to normal, Monitor will send a notification with the updated status.

Each alarm has rules associated with it that are used to trigger the alarm.  Each alarm can have up to 8 rules which are linked together either by AND or OR operators.  The rules can be setup as the following types of triggers.

  • Event-Based Rules – alert when specific events that occur in the backup or virtual infrastructure.  These can be events issued by the hypervisor, or by Veeam Backup & Replication.
  • Rules for a specific condition or state – these are rules that trigger when a condition is met, or a state has changed on your infrastructure objects.

Alarms also have a severity level attached to them of one of the following

  • Error (red) – indicates a critical situation or major problem
  • Warning (yellow) – indicates a potential problem or non critical issue.  Has the potential to move to an Error (red) if left unresolved.
  • Resolved (green) – indicates that the issue or alarm has been eliminated because of the changed conditions.
  • Information (blue) – indicates general information about the specific condition.

Alarms can be associated to objects by applying them directly to the object, on a group level using groups from Business View, or on the Infrastructure level by applying an alarm to all of a certain object type within the environment.

Interesting and testable tidbits about Veeam ONE Auto Discovery, Business View and Alarms

  • Cannot add a single ESXi host during the initial install, only vCenter Server.  ESXi and vCloud Directory are available to be added only after the initial install.
  • Ability to skip adding the infrastructure configuration during the install.
  • Backup Infrastructure can be added by either the Veeam Backup & Replication Server or the Veeam Backup Enterprise Manager.
  • The default data collection period for reporter and business view is weekdays @ 3am.  If at the end of an installation data will begin being collected immediately.
  • You cannot add the Backup Infrastructure inside of the Free edition of Veeam ONE UNLESS your VBR is licensed as a cloud connect only server.
  • When adding a VBR server, data is collected for the previous week only on all job sessions.
  • The maximum number of categories that Veeam ONE Business View supports is 25.
  • Each monitor alarm can have up to 8 rules associated with it.

VMCE v9 Study Guide – Module 5 – Creating Backup Jobs

VMCE LogoFinally, we get into the heart and soul of Veeam Backup & Replication; creating backup jobs. In this section we will kick off Module 5 and take a look at creating backup jobs from within Veeam Backup & Replication.

Before we go through the process of creating a backup job however there are a number of options and settings that we should completely understand!

Veeam Backup File types

.vbk – full backup file

.vib – forward incremental changes

.vrb – reverse incremental changes

.vbm – backup metadata

.vlb – transaction log backup

.vsb – synthetic backup used when creating virtual synthesized full backups for tape.

.bco – configuration backup

Backup Methods

Veeam Backup provides 3 methods for storing backups; Reverse Incremental Backup, Forward  Incremental Backup, Forever Forward incremental backup.

Reverse Incremental Backup

  • Consists of the last full backup and a set of reverse incremental backups to roll back in time.
  • Allows you to immediately restore a VM using the most recent state without having to chain together incremental.  When restoring from a later restore point, Veeam applies the required reverse incremental backup into the full backup and restores the VM from the full backup.
  • Performs as follows
    • During the first run Veeam creates a full backup
    • During subsequent runs, Veeam copies only blocks that have changed and injects these into the full backup file, resulting in the full backup always containing the most recent state.  During this, Veeam takes the blocks that are being replaced and builds a reverse incremental file with them and adds this to the backup chain.
    • Veeam checks the retention policy of the job.  If there is an outdated restore point it removes this from the backup chain

Forever Forward Incremental Backup

  • Consists of the first full backup, and subsequent incrementals following it.
  • Processed as follows
    • During the first run, Veeam creates a full backup
    • During subsequent runs, Veeam copies only changed VM data and saves these as an incremental backup file.
    • After adding the new restore point Veeam checks the retention period for the job.  If an outdated restore point is detected it transforms the backup chain as follows
      • Takes the restore point immediately following the full backup and injects those blocks into the full backup, thus moving the full backup ahead one restore point.
      • Deletes the restore point that was injected into the full backup

Forward Incremental Backup

  • produces a backup chain consisting of a full backup and set of incrementals following it.  In addition to this, synthetic or active full backups are created that split the backup chain into smaller chunks of chains.
  • Processed as follows
    • During the first run, Veeam creates a full backup file
    • During subsequent runs, Veeam copies only changed blocks and saves them as an incremental backup.
    • On days where a synthetic or active full backup is scheduled Veeam does the following
      • Active Full Backups
        • Veeam completes another full backup using data from the production datastore
      • Synthetic Full Backups
        • Veeam takes the incremental that was created in step 2, along with all other incrementals in the chain and merges them with the original full backup, creating a second full backup on disk.
        • Veeam then deletes the incremental that was created within step 2 as it is redundant data and already injected into the synthetic full.
        • This new synthetic full is now used as a starting point for future incrementals
    • The retention policy is then checked for the job and processed as follows
      • Veeam looks for outdated restore points and then looks to see if it is possible to delete them
      • Veeam needs to maintain a full backup and subsequent incrementals in order to restore to a certain point in time, therefore if a full backup has expired but some of its incrementals have not, Veeam will wait till the next full backup has expired in order to delete the first chain of backups
      • At times you may have more restore points on disk then expected.

Transforming incrementals into reverse incrementals

If we are creating synthetic full backups we can additionally chose to transform our forward incrementals to reverse incrementals.  In this case, Veeam will take the existing chain full and incrementals and transform it to reverse incremental restore points, allowing us to only keep one full backup on disk at a time, with reverse incremental restore points before it, and incremental restore points after.

Retention Policy for Deleted VMs

  • Applied only to reverse incremental, forever forward incremental, and forward incremental with synthetic full and transform
  • By default, VMs backups are set to remain on disk.
  • Space is not actually removed, it is flagged to be overwritten by other backups.

Storage Level Corruption Guard (Health Check for Backup Files)

VBR can periodically perform health checks for the latest restore points within backup files.  During this process VBR performs a CRC and hash check for both the metadata and blocks within the backup file to verify integrity.  This process is performed once a day when the health check is scheduled – if the backup session runs twice on the same day, it will not be run a second time.  The process of a health check is as follows

  • If corrupt blocks are detected in the metadata for a full backup file, Veeam marks the chain starting from this full as corrupted and triggers a health check retry
    • During the rety Veeam will transfer data blocks of the complete VM from the source datastore, creating a new full backup file.
  • If corrupt blocks are found in the meta data for an incremental file, Veeam removes the information about this incremental in the restore point configuration.
    • During the retry, Veeam transports incremental data relatively from the latest valid restore point – again, obtaining its data from the source datastore, creating a new incremental backup file on the repository.
  • If corrupt blocks are found in the actual backup file itself, full or incremental, Veeam marks these blocks as corrupt.
    • During the retry data blocks that were corrupt are transferred from the source datastore directly into the latest restore point.

Compression and Deduplication

Veeam provides 5 options in terms of compression

  • None – no compression – recommended when storing backups on storage devices that already have hardware compression
  • Dedupe-Friendly – optimized compression level for low CPU usage
  • Optimal – recommended compression level.  Best ratio between size and time.
  • High – Provides additional 10% over Optimal, but has 10x higher CPU usage
  • Extreme – smallest size of backup file but most demanding performance wise – recommended to have a proxy with at least 6 caores.

Deduplication allows us to save space as well in VBR and provides us with 4 options

  • Local Target (16TB) – recommended for backup jobs that may produce large (16TB+) backup files.  Uses 4MB block size in order to lower metadata size.
  • Local Target – recommended for backup to SAN, DAS, or local storage – uses 1MB block size.
  • LAN Target – recommended for NAS – uses 512 k block size.
  • WAN Target – recommended for backup over slower WAN – uses 256K block size

When we change compression settings for existing jobs any new settings will not be applied to already existing backups.  Compression will only be applied to new backup files created

When we change deduplication settings previously created files will not be affected.  New files created will not be deduplicated until we create an active full backup.

How do we do it?

The process of creating a backup job is as follows

  • There are a number of ways to kick off the Job wizard.
    • From the Jobs node in the Backup & Replication view right-click on jobs and select Backup->VMware (or Hyper-V)
    • From the Home tab, click Backup Job and then VMware
    • From the Virtual Machines view select the VM and right-click selecting either Add to Backup Job->Name of existing job or Add to Backup Job->New job.
  • Give the job a name and a description
  • In the Virtual Machines step of the wizard we need to select the VMs and VM containers we would like to backup.  If we select a contain, all child objects belonging to that container will be backed up – any change in our environment, like a new VM being added to that container will automatically be picked up by the job and processed.  To add our VMs select ‘Add’
    • We will now have a number of different views; Hosts and Clusters, VMs and Templates, Datastores and VMs, and Tags.
    • Select the desired VM/VM Containers and click ‘Add’
  • Here we can also exclude different VMs or disks…  Note, VM log files are automatically excluded to help reduce the size of the backup file and increase efficiency.
    • VMs from VM containers
      • From the VMs tab, click Add and select which VMs you wish to exclude from the job.  You will be presented with same multiple views as you were when you were choosing VMs to be backed up
    • Specific VM disks
      • On the disks tab select the VM in the list and click ‘Edit’.  If you are backing up a container or by tags you may need to Add the VM first by clicking ‘Add’ to manually place it in the list.
      • Chose which disks you would like to exclude from the VM – You can exclude all disks, 0:0 disks (system disks) or browse through a list of custom IDE/SCSI/Sata disks.
      • If you wish you Veeam can remove these disks from the VMs configuration file so you are able to power on the VM when restoring.  To do so, select the ‘Remove excluded disks from VM configuration’ checkbox.
    • VM templates
      • On the templates tab clear the Backup VM templates checkbox
      • If you wish so, you can alternatively clear the Exclude templates from incremental backup checkbox to only process templates in the full backup file.
  • Still on the Virtual Machines section we have the option to re-order the processing of VMs by selecting the VM and using the Up/Down buttons.  That said, if you are backing up a VM from a container or using tags you will not be able to do this as they are processed randomly.  To do so, you would need to add them as standalone VMs.  Also take note, with Parallel processing enabled you may find unexpected results with the processing order.  In cases where resources for a higher priority VM are not fully available but enough resources for a lower priority VM are, the lower VM may be processed first.
  • On the storage tab we will specify which backup proxy we would like to use, which repository to backup up to, the retention policy, any secondary destinations for the backups as well as any advanced settings we’d like to apply…
    • Backup Proxy has a couple different options
      • Automatic – Veeam will detect backup proxies that have access to the source datastore and automatically assign one to process VMs within the job.  This is done on a per VM basis.  Before processing a new VM, Veeam analyzes the available proxies, looking at their transport modes and current workload to select the optimal proxy for that VM
      • Use selected – This allows you to explicitly state which backup proxies can be used within the job.  At least two selections are recommended for HA or network loss.
    • Backup Repository – Select a destination repository to store the backups on.
      • If you already have a backup stored on the repository you can map to those already existing files.  To do so select the Map Backup link below the repository selection.
    • Retention Policy – Specify the number of restore points that you wish to store on the repository, keeping in mind all of the information we went through in the Retention sections above.
    • Secondary Destination – Allows for us to archive our backups to a secondary destination, either a backup repository or tape.  When this option is selected we will see an additional step appear.   In the additional step we link this backup job to another backup copy job or backup to tape job.
    • Advanced.  There are a number of advanced settings we can set on the storage step of the backup job as well.  Note, after configuring all of these settings you can save them as the default settings for future jobs.
      • Backup Settings
        • Select the desired backup method  to store the backup chain; Reverse Incremental, Incremental w/ Synthetic or active full or Forever forward incremental.  See earlier in the post on each backup type.
        • Select whether or not to create a synthetic full – synthetic full will build a full backup out of the restore points already located on the backup repository.  Also select the Days to create the synthetic full on.  If creating a synthetic full you can also specify whether to transform any previous backup chains to rollbacks
        • Select whether to create an active full backup (retrieves all source data from the datastore), and specify to create it monthly or weekly and specify days.
      • Maintenance Settings
        • Storage Level Corruption guard – check to periodically perform a health check on the latest restore point in the backup chain.  Helps to prevent a corrupted restore point.  If the health check discovers a corrupt block it starts the health check retry and transfers the data from the source datastore to a new backup file or the latest backup file, depending on the scenario – explained above.
        • Full Backup File Maintenance
          • Remove deleted VMs data after – specifies the number of days to keep backup data for VMs that have been deleted from the production environment
          • Defragement and compact full backup file – check to perform and schedule a compact operation.   Compact will create a new empty file and copy data blocks from the full backup to it.  If the backup files contains deleted VMs, they will be removed.  More info above.
      • Storage Settings
        • Data Reduction
          • Enable Inline data deduplication – clear the checkbox to disable deduplication.
          • Exclude swap file blocks – By default Veeam looks at the NTFS MFT file to identify the blocks that make up hiberfil.sys and pagefile.sys and excludes them from the backup.  Clear this checkbox if you prefer to back these files up
          • Exclude delete file blocks – By default Veeam does not copy file blocks that were deleted ormarked as dirty.  If you would rather copy these, clear this checkbox
        • Compression Level – Select the desired compression level for the job (None, Dedup-Friendly, Optimal, High, or Extreme).
        • Storage Optimization  – Select the type of backup target you plan to use; Local Target, LAN target, or WAN target.  Veeam will use different data block size to optimize backup performance.
        • Encryption
          • Check and provide a password to encrypt the backups on disk.
          • If you enable this on an existing job a full backup will be created on the next run and it and all subsequent incrementals will be encrypted.
      • Notification Settings
        • Can be used to send SNMP notifications for the job, to customize notifications on a per job basis rather than utilizing the global settings.
        • Can also set backup details to a VM attribute of choosing, either overwriting or appending.
      • vSphere Settings
        • Choose whether to enable VMware tools quiescence.
        • Choose whether to use Change block tracking and whether to enable CBT on existing VMs.
      • Integration Settings
        • Chose whether to enable backup from storage snapshots and whether to limit processed VM count per storage snapshot to a certain number.  You can also chose to failover to a standard backup if Veeam fails to create the storage snapshot.  If using NetApp you will also have the option to failover to primary storage snapshots.
      • Script Settings
        • Job Scripts
          • Chose to run a script before and/or after the job and specify a path to the script.
          • Chose when to run them – every X backup sessions or on selected days only.
  • Guest Processing handles the following options
    • Application Aware Processing
      • Enable/Disable application aware processing.
      • Clicking Applications gives you the following options on a per-VM basis.
        • General
          • Chose whether to require successful VSS application processing, try application processing but ignore failures or disable application processing
          • Chose how to handle transaction logs if this is a SQL server, Oracle, or Exchange VM, either process the logs (additional settings will be required on the SQL tab) or perform a copy only of the logs (logs will just be copied in backup as a normal backup would copy files)
        • SQL – additional settings for logs if you chose process from last step.
          • How to handle transaction logs, either truncate, don’t truncate, or backup logs periodically
          • If last option is chosen we need to specify the interval to backup transaction logs (in minutes) and a retention policy, either in days or until the corresponding image level backup that the logs are attached to is deleted.
          • We can also specify which server to use as a log shipping server, the server that transports the logs.  We can let Veeam automatically figure this out or specify a specific set of Log Shipping servers.  Log shipping servers are just any Windows servers added to our Backup Infrastructure.
        • Oracle
          • Oracle Account – specify a user account that has SYSDBA rights on the database.  If you chose to use guest credentials, Veeam will use the account setup within the guest processing section to connect.
          • Log Action – here we chose what to do with our logs, we can set it to not truncate logs at all, truncate logs older than X number of hours, or truncate logs over X number of GB.
          • Log Backup inteval – set the transaction log backup interval in minutes – default is 15.
          • Retention  – specify how long to keep the logs, either last X number of days or until the corresponding image level backup is deleted.
          • Log shipping server – just as with SQL we can either let Veeam automatically pick a log shippiong server or set this to a specific set.
        • File Exclusions – we are able to exclude certain files, folders or filetypes on a per VM basis.
          • First select to either disable file level exclusions, Exclude certain files and folders, or include only certain files and folders.
            • If excluding or including we can click add to add certain files or folders.  We have the option here to use environment variables, such as %windir% or file masks like * and ?
        • Scripts – we can use this section if we plan to back VMs up that do not support VSS in order to run scripts to obtain application consistent backups.
          • Script Processing Mode – select wehter to require successful script execution, ignore script execution failuer or disable script execution
          • Windows Scripts – path to windows scripts
          • Linux Scripts – path to Linux scripts
    • VM Guest OS file indexing – can select from the following options on a per-VM basis; Disable Indexing, Index everything, Index everything ecept and specify files/folders.
      • If indexing Linux OS, several components need to be installed on the VM.  mlocate, gzip, and tar.  Veeam will prompt to install these if they are not found.
    • Guest OS Credentials
      • Select default credentials to use to deploy runtime processes.  Any VMs that don’t have credentials explicitly assigned to them will use these credentials.
    • Guest Interaction Proxy
      • Specify a Guest Interaction Proxy to use or let Veeam Automatically choose one.
  • Schedule
    • Run job automatically
      • Daily – runs job daily at a selected time on selected days
      • Monthly – runs job monthly at a selected time on selected days
      • Periodically every – runs job periodically ever X number of hours/minutes or continuosly to continuosly run job
      • After this job – Allows you to chain this job to the ending of another.  This will only be ran if the first job is started by a schedule – manually running the first job will not run this job.
    • Automatic Retry
      • Select to retry VM X number of times with X number of minutes in between each try upon job failure
    • Backup Window
      • Can set the job to terminate if it exceeds a specified backup window
  • Summary – review settings and save job.  You do have the option to immediately execute the job here as well.

Nakivo Screenshot Verification – Getting to know your backups!

Picture yourself in this scenario – you walk into work on a Monday morning where you are promptly greeted by just about every IT staff member in the building.  They quickly tell you that certain mission critical services are down and the company is losing money as we speak.  But that’s OK, you are here, the infamous backup guy – the guy no one cares about unless things go down.  None the less you sit at your desk and begin the restore process.  “No problem”, you say, “services should be back up in 10 minutes or so…”.  The VM is fully restored and powered on – you sit, watching the console and the Windows spinning wheel and all of a sudden you see this!


Yikes!   Your backups, just like their production counterparts are corrupt – you try a different restore point, no go, they are all corrupt.  Believe it or not this is a common scenario that is played out inside of organizations everywhere.  Backups, just like any other production workload or service need to be tested in order to ensure that they are indeed restorable.  The best way of testing these backups is definitely done by performing full restores on the data – however doing so after each and every backup job can be quite time consuming and inefficient in terms of resource usage.

Enter Nakivo

nakivoNakivo does a great job at backing up and protection your VMware and Amazon environment and has a lot of features included within their product, Nakivo Backup and Replication.  I’ve previously reviewed the Nakivo product as a whole here if you’d like to check it out.  This post, will focus more on one feature – a feature that helps to prevent situations like the one above – Nakivo Screenshot verification.  There is no worse feeling than having terabytes of backups that prove to be unreliable and in turn, useless – which is the exact reason why Nakivo has developed Screenshot Verification inside of their Backup & Replication software – to give you piece of mind that when push comes to shove, your backups will indeed be restorable!

What is Screenshot Verification?

Screenshot verification is a simple concept with a lot of underlying technology at play – in its basic form, Nakivo will verify each VM backup after a new restore point is completed.  This is done by booting the VM directly from its’ corresponding deduplicated and compressed backup files located on a Nakivo backup repository, a process that Nakivo calls Flash VM Boot.  During a Flash VM boot Nakivo creates a new VM on a specified ESXi server.  It then takes the disks from within the backup files and exposes them as iSCSI targets, upon completion, the disks are mounted to the new VM as vRDMs.  A snapshot is created in order to provide disposal of any changes and the newly created VM is powered on, isolated from the production network.  Once booted, Nakivo utilizes VMware tools in order to take a screenshot of the booted OS.  After the screenshot is taken the newly created VM is discarded and backup files are brought back to a consistent state, and the screenshot is included within any job reports, either generated through the UI or emailed.

It’s this screenshot that gives you the “piece of mind” that when the time comes to restore your VMs, they will indeed be restorable!  A simple picture of the Windows login screen or Linux bash shell, or lack there of, certainly would’ve helped in the above scenario – alerting us that the next time we try and reboot our production VM or restore from a backup that problems may occur – giving us the leeway and time to fix the situation or restore to our last known good restore point on our own terms rather than doing so during an emergency.

How do we set this up?

As far as how to setup and configure Nakivo Backup and Replication as a whole I would recommend checking out my previous review here – but focusing solely on Screenshot Verification let’s go through the steps below…  **Note, we are setting this up for one of our backup jobs, however we can also enable screenshot verification for our replication jobs as well **

Screenshot verification, all be there a lot of moving parts underneath is actually a simply Enable/Disable feature within the backup job.  Nakivo has done a great job of abstracting away all of the complicated technology underneath and presenting us with some simple and easy to use configurable options.  On the last step of a job wizard, we see the Screenshot verification setting at the bottom of the first column (as shown below)…


Upon selecting ‘settings’ we are presented with some more options which we can configure.  The target container is the placeholder in which we will register the newly created VM that will be mounted to the backup files.  This can be your average vSphere object that VMs belong to such as a host or cluster.  Target datastore is where we would like to place the configuration files (vmx) of the VM that is created.  Verification Options allows us to do things such as limit the amount of VMs which will be verified simultaneously.  Running too many VM Screenshot Verification tests at once can produce a heavy load on your backup repositories, causing major delays in boot time depending on your hardware configuration – it’s best to tune this to your liking.    Also configurable here are things like RTO, which in this case defines the number of minutes that the VM has to fully boot and initialize VMware tools.  If this time is exceeded, the VM will be labeled as failed and the placeholder VM is discarded.  We can also set the delay between when the guest OS has booted, and the actual execution of the screenshot.


Honestly, this is all we need to do!  Simply save your job and on your next job run Screenshot verification should take place.  As shown below, we can see the events that take place within vCenter during a Screenshot verification test, along with the placeholder VM that is created in order to perform these tests, noting the creation and deletion of the VM, along with any required iSCSI setup.  This is all automated by Nakivo and requires no manual setup on your part.


So we have now seen that the Screenshot verification has been executed, but what does it look like in one of the reports/emails.  Right-clicking any job within Nakivo gives us the ability to run a few reports – the one we are most interested in now is the ‘Last run report’.  After generating and opening the ‘Last run report’ for our job with screenshot verification enabled we should see new information included in the report.  As shown below we see that we have a ‘Last verification’ row now, indicating whether or not that the screenshot verification was successful – in addition, we can also see the actual screenshot that was taken by Nakivo.  Below we see the actual login screen, giving us a pretty good indication that if we were to restore from this backup we would be successful.


Hey, Let’s have some fun!

As you can see, Screenshot verification is a very valuable tool giving us that piece of mind that our backups are actually restorable.  But where’s the fun in that right?  Let’s break some stuff and see how Screenshot verification reacts….

So, on my production VM let’s mimic some corruption and see if we can’t get Nakivo to detect it before we do!  In order to do this I’ve run the following commands on my production VM within an administrative console (***NOTE*** Don’t do this in production, please, please don’t do this in production Smile)

takeown /F C:\Windows\System32\WinLoad.exe

cacls C:\Windows\System32\WinLoad.exe /G administrator:F

del C:\Windows\System32\WinLoad.exe

bcdedit /set recoveryenabled No

The first three lines are pretty self explanatory, taking ownership, assigning rights, and deleting WinLoad.exe – the file that actually executes the loading of Windows upon boot.  The last line simply disables the automatic repair, Microsoft’s line of defense for preventing people from doing stupid things like this Smile  Anyways, we’ve essentially botched our server here, however we won’t notice until we do a reboot, something that probably doesn’t happen that frequently in a production environment – thus, it’s probably going to go unnoticed for quite some time – that is, unless we are utilizing Nakivo’s screenshot verification on our backup jobs.

nakivoss-log Let’s go ahead and run our backup job again on this same VM.  This time, we will see Nakivo report a failure on the backup job, specifying that screenshot verification has failed – upon further investigation, we can see below what appears on the console of our VM that used for the verification, and is exactly what would happen to our production VM if we were to reboot it!   Even though our newly created backup is not restorable, at least we now know and it won’t be a surprise to us in an emergency situation like the previous scenario.  This gives us time – time to come up with a plan, whether that be restoring from a known good backup, coming up with some sort of failover plan or even building a new server.


So in the end screenshot verification proves to be a very valuable tool in any backup administrators belt – whether that being knowing that your backups can be restored successfully, or sometimes even more important, knowing that they can’t – and in some cases, Screenshot verification can be leveraged to prevent production outages by getting a preview of things to come upon the next reboot!  The Flash VM Boot technology makes Screenshot verification a no-brainer in my opinion.  If you are using Nakivo, you should be enabling this on all of your mission critical VMs.  To learn more about Screenshot verification and other Nakivo features check out their help center here.  Fancy trying it for yourself?  You can get a full featured trial here, or if you are a VMUG member, VCP, or vExpert why not grab a free NFR license to tinker with!  If that isn’t enough options for you Nakivo also offers a fully featured free edition – yes, all of the same features of their premium paid versions, just limited to a couple VMs.  Thanks for reading!

Nakivo Instant Object Recovery for Microsoft Active Directory

nakivoNakivo, a backup company based out of Silicon Valley has been providing backup and replication software to the world since late 2012.  Today we will not focus so much and getting Nakivo up and running, we’ve already done that thoroughly here, but instead we will take a look at one individual feature; Instant Object Level Recovery for Microsoft Active Directory.  Let’s face it – mistakes happen – users get deleted, OU’s get wiped out, security groups get completely out of sync.  This is all stuff that happens, and happens more often than we know it.  Certainly performing a complete full restore of a domain controller can be a little bit over the top just to get one individual user back (depending on who it is I suppose Smile ), which is why Nakivo has been providing a means for restoring these individual Active Directory objects since their 5.5 release back in March of 2015.  Today we will take a more in-depth look at just how we perform these restorations.    Rather than simply showing how things are done I thought I’d have a little more fun with it this go around, put a little story behind it for all of our enjoyment 🙂  With that said, let’s dive in!

The Scenario

Let’s paint the scene – you are a sysadmin working for a pretty famous hockey club based out of Montreal.  You are using Nakivo to protect a couple of datacenters, one in Montreal and another in Brossard, with a fully virtualized Active Directory.    One morning for whatever reason your supervisor was a little off his game – maybe it was too much wine the night before, or perhaps he had a heaping of bad poutine at lunch, but when asked to disable and enable certain players directory accounts after a blockbuster trade, he had a slip up.  Sure, he disabled the “psubban” account of the outgoing player as he was asked to, however in the process of creating the new “swebber” account, somehow he ended up deleting Andrei Markov’s account (amarkov).

It wasn’t until Andrei showed up for practice that morning that anyone noticed – Andrei attempted to log in and quickly realized that something was up.   When the helpdesk ticket finally made its way to to your supervisors desk he knew immediately what had happened and quickly called upon you to help out.  “No worries”, you said, “We’re protecting that server with Nakivo!”

How can Nakivo help you?

Thankfully you had already setup a backup job which processes a domain controller belonging to the canadiens.local domain, the same domain the user was accidentally deleted from.  We won’t go into the nitty-gritty details of how to setup the backup job here, as this post focuses solely on the recovery, but we have covered it in detail in another post if you’d like to check it out.  But instead we’ll go through the steps for us to restore Andrei’s account – The first thing we need to do is fire up his browser and log into Nakivo Backup and Replication.  After logging into the application, simply selecting ‘Microsoft Active Directory objects’ under the ‘Recover’ menu kicks off the process (shown below).


The next step is quite simply and pretty self explanatory – we simply need to select the backup of our domain controller, in our case its named MSDC, and then select a desired restore point to restore from.  As shown below we also have the option to ‘Automatically locate application databases’, which is checked by default.  If we happened to know the exact location of the MS AD database then we could uncheck this an specify the path, and in turn maybe save a little time as Nakivo wouldn’t need to scan for the ntis.dit file.  Honestly though, the amount of time it takes Nakivo to locate the Active Directory database is trivial, so let’s leave this checked, and click ‘Next’.


Nakivo will now take a moment to load the desired restore point and display it to us.  The amount of time this takes greatly depends on the size of your Active Directory infrastructure.  Canadiens.local is relatively small, and took only a few seconds to load – but before we move on to the next step it’s good to go over what is happening behind the scenes here.  Nakivo Backup & Replication is actually scanning and mounting the server directly from within the compressed and deduplicated backup file – at no time does it perform a full recovery of the VM itself, saving us valuable time as we only need to restore that one individual object.  As shown below we are presented with a screen on which we can browse through the entire Active Directory infrastructure and find the object we’d like to restore.  It should be noted here that Nakivo supports object-level recovery for not just users, but containers and groups as well – so if it was an Organization Unit or Security Group that was deleted we would be able to restore it in the same manner.   Next we select the object by simply clicking the checkbox beside it, and then click ‘Download Selected’.  Alternatively we could click ‘Forward Selected’ to have Nakivo email out the ldif files to be used for import.    At this point we will have a couple or Recovery settings we can specify; User will be disabled – will restore the user with the account disabled or User must change password at next logon – Nakivo automatically generates a new password for the restored user, and sets the ‘Change password on next logon’ flag in AD.  Any password Nakivo generates will be stored in an included ‘Passwords.txt’ file added to our download.


After downloading the recovery bundle (should come in a .zip format) we can now get started on restoring Andrei Markov’s account back into the canadiens.local domain.  We does this by first extracting the bundle and copying the extracted folder back to his domain controller.  Since we are importing a user object back into Active Directory we need to have ldaps, or certification services enabled and configured on the domain controller.  Thankfully the canadiens.local domain is already setup this way, however if we need to implement ldaps there is a great post here on how to go about it.  Once we are back on the domain controller console we can simply open up an administrative command prompt and run the following command…

ldifde –I –t 636 –f filename –j logfolder <- where filename is the path the the downloaded ldif from Nakivo and logfolder is a path for import logs to be placed.

We can see a screenshot below of the before and after shots of the canadiens.local domain, with the after showing that Andrei Markov’s account has indeed been restored.

nakivoad-before nakivoad-after

With that you can now breathe easy as Andrei’s account is fully restored back into Active Directory, including all of his user attributes, group memberships, etc.  Honestly, it’s as if it was never deleted!  This whole process moves very quickly within Nakivo, honestly, within minutes – and when the time comes where you need to do a restore, especially one revolving around user access, time is most certainly of the essence.  Nakivo could certainly shave even more time off this process by implementing some way to automate the ldif import, or import directly back into the production VM – but honestly, the simplicity of this whole process far outshines the fact that it needs to be manually imported.  For now, you and your supervisor can get back to what matters most; the quest for Lord Stanley.

If you would like to learn more about Nakivo’s Instant Object Recovery for Active Directory or any other feature they offer I highly recommend checking out their help center here, where you can find items such as their knowledge base, release notes, and a very well written user guide.  Also if you want to check it out for yourself you can get a full featured trial here, or if you are a VMUG member, VCP, or vExpert why not grab a free NFR license to tinker with!  If that isn’t enough options for you Nakivo also offers a fully featured free edition – yes, all of the same features of their premium paid versions, just limited to a couple VMs.  Thanks for reading!