Category Archives: Posts
Veeam ONE is essentially a management program, providing monitoring, reporting and alerting for your virtual infrastructure, whether that be Hyper-V or VMware. The module for this course is broken down into 9 different sub modules, so lets look at each one in turn.
Auto Discovery of Backup and Virtual Infrastructure
During the initial configuration of Veeam ONE users are prompted to chose and add their virtual infrastructure type – the options presented are
- VMware vCenter Server
- Hyper-V Host, Failover Cluster or SCVMM server
- Skip Virtual infrastructure configuration
After selecting your desired virtual infrastructure you will be prompted for it’s name, either IP or FQDN, and some credentials to connect – along with the required port to connect on. After installation we have the ability add other types of servers such as a standalone ESXi host and a vCloud Director server on the inventory pane inside the Infrastructure view.
Once completed the initial configuration, the servers, both vSphere, Hyper-V or Veeam that you have setup during the installation will be propagated to all Veeam ONE components, such as monitor, reporter and business view and data will begin being collected immediately as well as during its default scheduled time (weekdays @ 3 am).
Once a connection has been established Veeam ONE will start pulling information and data on the top level scope you’ve added, plus all children. Individual hosts/clusters can be excluded after the face from the Monitored Hosts tab. Subsequently, datastores can be excluded as well on the Monitored Datastores tab. If you want to exclude certain VMs it is done a little differently. Instead we use rules to exclude VMs. By default an inclusion rule that adds all VMs is configured. If you want to establish another inclusion rule, then you must disable this default rule. You do not need to do this when creating exclusion rules. You can create rules based on the following criteria.
- By object name – add/exclude VMs based on VM name
- By infrastructure location – Apply rule that applies to VMs only within a certain hierarchy of the infrastructure.
When creating our rules we can use the * and/or the ? wildcard. The * stands for zero or more characters where the ? stands for just a single character. If adding multiple conditions to a VM you need to specify whether you would like to apply the rule if any condition evaluates to true or if all conditions evaluate to true.
When prompted to add your Backup Infrastructure you simply need to pass the FQDN Name of either your Veeam Backup & Replication server or your Veeam Backup Enterprise Manager server and pass the required credentials. You also have the ability to skip this step and add Veeam later. If we connect a Backup & Replication server, data for the job sessions for the previous week is collected. If we add an Enterprise Manager instance, Veeam ONE first builds a hierarchy of Veeam B&R servers, then collects the data for the job sessions for the previous week from each one of them.
Categorization within Veeam ONE is done by a the Business View component which is installed alongside everything. Business view allows you to create categorization models and values to help organize your infrastructure and display your infrastructure from a business standpoint. This completely integrates into Veeam ONE Monitor and Reporter, so any groups you create within Business View will be available in the other Veeam ONE components.
Some examples of categories one might create in Veeam ONE Business View are…
- Business Unit,
Business View also supports reading and writing vSphere tags as well, meaning you can map business view categories and groups to vSphere tags and vice versa. That said there are some default out of the box categories that Business View comes with..
- Datastore – groups VMs by their datastore
- Last Backup Date – dynamically groups VMs by the age of the latest backup or replica restore point created
- Sample Business View Category – dynamically groups VMs by name
- SLA – static groups for all types of virtual infrastructure objects. Includes two groups; Mission Critical and Other
- Storage Type – dynamically groups storage objects by type.
- VM Network – dynamically groups VMs by connected network
- VMs with Snapshots – dynamically groups VMs with snapshots by snapshot age.
You can add more custom categories, but keep in mind the maximum number of categories Veeam ONE supports is 25.
Veeam ONE Monitor comes with over 150 predefined alarms that can alert you on almost every aspect of your virtual and backup infrastructures. In terms of data protection there are alarms already setup to…
- connectivity issues an inability of backup infrastructure components to communicate with each other.
- state of the VBR software installed on infrastructure components
- Failing of jobs or jobs completed with warnings
- Configuration issues, such as repositories running out of space
- Long running jobs that exceed the backup window
- License and pre-paid support expirations.
Alarms within Veeam ONE work in the following way
- When Veeam ONE monitor detects that the behavior or state of an object meets an alarm criteria, it triggers the alarm with the corresponding severity level
- Once triggered, Monitor console will display the alarm details and associated information in regards to the alarm. At this point you can view, acknowledge, or resolve the alarm
- After the alarm has fired, Monitor performs a responsive action; email, SNMP, and/or running a custom script.
- Once the alarm has been resolved, Monitor updates the alarm status within the console.
- If the state or condition returns to normal, Monitor will send a notification with the updated status.
Each alarm has rules associated with it that are used to trigger the alarm. Each alarm can have up to 8 rules which are linked together either by AND or OR operators. The rules can be setup as the following types of triggers.
- Event-Based Rules – alert when specific events that occur in the backup or virtual infrastructure. These can be events issued by the hypervisor, or by Veeam Backup & Replication.
- Rules for a specific condition or state – these are rules that trigger when a condition is met, or a state has changed on your infrastructure objects.
Alarms also have a severity level attached to them of one of the following
- Error (red) – indicates a critical situation or major problem
- Warning (yellow) – indicates a potential problem or non critical issue. Has the potential to move to an Error (red) if left unresolved.
- Resolved (green) – indicates that the issue or alarm has been eliminated because of the changed conditions.
- Information (blue) – indicates general information about the specific condition.
Alarms can be associated to objects by applying them directly to the object, on a group level using groups from Business View, or on the Infrastructure level by applying an alarm to all of a certain object type within the environment.
Interesting and testable tidbits about Veeam ONE Auto Discovery, Business View and Alarms
- Cannot add a single ESXi host during the initial install, only vCenter Server. ESXi and vCloud Directory are available to be added only after the initial install.
- Ability to skip adding the infrastructure configuration during the install.
- Backup Infrastructure can be added by either the Veeam Backup & Replication Server or the Veeam Backup Enterprise Manager.
- The default data collection period for reporter and business view is weekdays @ 3am. If at the end of an installation data will begin being collected immediately.
- You cannot add the Backup Infrastructure inside of the Free edition of Veeam ONE UNLESS your VBR is licensed as a cloud connect only server.
- When adding a VBR server, data is collected for the previous week only on all job sessions.
- The maximum number of categories that Veeam ONE Business View supports is 25.
- Each monitor alarm can have up to 8 rules associated with it.
Finally, we get into the heart and soul of Veeam Backup & Replication; creating backup jobs. In this section we will kick off Module 5 and take a look at creating backup jobs from within Veeam Backup & Replication.
Before we go through the process of creating a backup job however there are a number of options and settings that we should completely understand!
Veeam Backup File types
.vbk – full backup file
.vib – forward incremental changes
.vrb – reverse incremental changes
.vbm – backup metadata
.vlb – transaction log backup
.vsb – synthetic backup used when creating virtual synthesized full backups for tape.
.bco – configuration backup
Veeam Backup provides 3 methods for storing backups; Reverse Incremental Backup, Forward Incremental Backup, Forever Forward incremental backup.
Reverse Incremental Backup
- Consists of the last full backup and a set of reverse incremental backups to roll back in time.
- Allows you to immediately restore a VM using the most recent state without having to chain together incremental. When restoring from a later restore point, Veeam applies the required reverse incremental backup into the full backup and restores the VM from the full backup.
- Performs as follows
- During the first run Veeam creates a full backup
- During subsequent runs, Veeam copies only blocks that have changed and injects these into the full backup file, resulting in the full backup always containing the most recent state. During this, Veeam takes the blocks that are being replaced and builds a reverse incremental file with them and adds this to the backup chain.
- Veeam checks the retention policy of the job. If there is an outdated restore point it removes this from the backup chain
Forever Forward Incremental Backup
- Consists of the first full backup, and subsequent incrementals following it.
- Processed as follows
- During the first run, Veeam creates a full backup
- During subsequent runs, Veeam copies only changed VM data and saves these as an incremental backup file.
- After adding the new restore point Veeam checks the retention period for the job. If an outdated restore point is detected it transforms the backup chain as follows
- Takes the restore point immediately following the full backup and injects those blocks into the full backup, thus moving the full backup ahead one restore point.
- Deletes the restore point that was injected into the full backup
Forward Incremental Backup
- produces a backup chain consisting of a full backup and set of incrementals following it. In addition to this, synthetic or active full backups are created that split the backup chain into smaller chunks of chains.
- Processed as follows
- During the first run, Veeam creates a full backup file
- During subsequent runs, Veeam copies only changed blocks and saves them as an incremental backup.
- On days where a synthetic or active full backup is scheduled Veeam does the following
- Active Full Backups
- Veeam completes another full backup using data from the production datastore
- Synthetic Full Backups
- Veeam takes the incremental that was created in step 2, along with all other incrementals in the chain and merges them with the original full backup, creating a second full backup on disk.
- Veeam then deletes the incremental that was created within step 2 as it is redundant data and already injected into the synthetic full.
- This new synthetic full is now used as a starting point for future incrementals
- Active Full Backups
- The retention policy is then checked for the job and processed as follows
- Veeam looks for outdated restore points and then looks to see if it is possible to delete them
- Veeam needs to maintain a full backup and subsequent incrementals in order to restore to a certain point in time, therefore if a full backup has expired but some of its incrementals have not, Veeam will wait till the next full backup has expired in order to delete the first chain of backups
- At times you may have more restore points on disk then expected.
Transforming incrementals into reverse incrementals
If we are creating synthetic full backups we can additionally chose to transform our forward incrementals to reverse incrementals. In this case, Veeam will take the existing chain full and incrementals and transform it to reverse incremental restore points, allowing us to only keep one full backup on disk at a time, with reverse incremental restore points before it, and incremental restore points after.
Retention Policy for Deleted VMs
- Applied only to reverse incremental, forever forward incremental, and forward incremental with synthetic full and transform
- By default, VMs backups are set to remain on disk.
- Space is not actually removed, it is flagged to be overwritten by other backups.
Storage Level Corruption Guard (Health Check for Backup Files)
VBR can periodically perform health checks for the latest restore points within backup files. During this process VBR performs a CRC and hash check for both the metadata and blocks within the backup file to verify integrity. This process is performed once a day when the health check is scheduled – if the backup session runs twice on the same day, it will not be run a second time. The process of a health check is as follows
- If corrupt blocks are detected in the metadata for a full backup file, Veeam marks the chain starting from this full as corrupted and triggers a health check retry
- During the rety Veeam will transfer data blocks of the complete VM from the source datastore, creating a new full backup file.
- If corrupt blocks are found in the meta data for an incremental file, Veeam removes the information about this incremental in the restore point configuration.
- During the retry, Veeam transports incremental data relatively from the latest valid restore point – again, obtaining its data from the source datastore, creating a new incremental backup file on the repository.
- If corrupt blocks are found in the actual backup file itself, full or incremental, Veeam marks these blocks as corrupt.
- During the retry data blocks that were corrupt are transferred from the source datastore directly into the latest restore point.
Compression and Deduplication
Veeam provides 5 options in terms of compression
- None – no compression – recommended when storing backups on storage devices that already have hardware compression
- Dedupe-Friendly – optimized compression level for low CPU usage
- Optimal – recommended compression level. Best ratio between size and time.
- High – Provides additional 10% over Optimal, but has 10x higher CPU usage
- Extreme – smallest size of backup file but most demanding performance wise – recommended to have a proxy with at least 6 caores.
Deduplication allows us to save space as well in VBR and provides us with 4 options
- Local Target (16TB) – recommended for backup jobs that may produce large (16TB+) backup files. Uses 4MB block size in order to lower metadata size.
- Local Target – recommended for backup to SAN, DAS, or local storage – uses 1MB block size.
- LAN Target – recommended for NAS – uses 512 k block size.
- WAN Target – recommended for backup over slower WAN – uses 256K block size
When we change compression settings for existing jobs any new settings will not be applied to already existing backups. Compression will only be applied to new backup files created
When we change deduplication settings previously created files will not be affected. New files created will not be deduplicated until we create an active full backup.
How do we do it?
The process of creating a backup job is as follows
- There are a number of ways to kick off the Job wizard.
- From the Jobs node in the Backup & Replication view right-click on jobs and select Backup->VMware (or Hyper-V)
- From the Home tab, click Backup Job and then VMware
- From the Virtual Machines view select the VM and right-click selecting either Add to Backup Job->Name of existing job or Add to Backup Job->New job.
- Give the job a name and a description
- In the Virtual Machines step of the wizard we need to select the VMs and VM containers we would like to backup. If we select a contain, all child objects belonging to that container will be backed up – any change in our environment, like a new VM being added to that container will automatically be picked up by the job and processed. To add our VMs select ‘Add’
- We will now have a number of different views; Hosts and Clusters, VMs and Templates, Datastores and VMs, and Tags.
- Select the desired VM/VM Containers and click ‘Add’
- Here we can also exclude different VMs or disks… Note, VM log files are automatically excluded to help reduce the size of the backup file and increase efficiency.
- VMs from VM containers
- From the VMs tab, click Add and select which VMs you wish to exclude from the job. You will be presented with same multiple views as you were when you were choosing VMs to be backed up
- Specific VM disks
- On the disks tab select the VM in the list and click ‘Edit’. If you are backing up a container or by tags you may need to Add the VM first by clicking ‘Add’ to manually place it in the list.
- Chose which disks you would like to exclude from the VM – You can exclude all disks, 0:0 disks (system disks) or browse through a list of custom IDE/SCSI/Sata disks.
- If you wish you Veeam can remove these disks from the VMs configuration file so you are able to power on the VM when restoring. To do so, select the ‘Remove excluded disks from VM configuration’ checkbox.
- VM templates
- On the templates tab clear the Backup VM templates checkbox
- If you wish so, you can alternatively clear the Exclude templates from incremental backup checkbox to only process templates in the full backup file.
- VMs from VM containers
- Still on the Virtual Machines section we have the option to re-order the processing of VMs by selecting the VM and using the Up/Down buttons. That said, if you are backing up a VM from a container or using tags you will not be able to do this as they are processed randomly. To do so, you would need to add them as standalone VMs. Also take note, with Parallel processing enabled you may find unexpected results with the processing order. In cases where resources for a higher priority VM are not fully available but enough resources for a lower priority VM are, the lower VM may be processed first.
- On the storage tab we will specify which backup proxy we would like to use, which repository to backup up to, the retention policy, any secondary destinations for the backups as well as any advanced settings we’d like to apply…
- Backup Proxy has a couple different options
- Automatic – Veeam will detect backup proxies that have access to the source datastore and automatically assign one to process VMs within the job. This is done on a per VM basis. Before processing a new VM, Veeam analyzes the available proxies, looking at their transport modes and current workload to select the optimal proxy for that VM
- Use selected – This allows you to explicitly state which backup proxies can be used within the job. At least two selections are recommended for HA or network loss.
- Backup Repository – Select a destination repository to store the backups on.
- If you already have a backup stored on the repository you can map to those already existing files. To do so select the Map Backup link below the repository selection.
- Retention Policy – Specify the number of restore points that you wish to store on the repository, keeping in mind all of the information we went through in the Retention sections above.
- Secondary Destination – Allows for us to archive our backups to a secondary destination, either a backup repository or tape. When this option is selected we will see an additional step appear. In the additional step we link this backup job to another backup copy job or backup to tape job.
- Advanced. There are a number of advanced settings we can set on the storage step of the backup job as well. Note, after configuring all of these settings you can save them as the default settings for future jobs.
- Backup Settings
- Select the desired backup method to store the backup chain; Reverse Incremental, Incremental w/ Synthetic or active full or Forever forward incremental. See earlier in the post on each backup type.
- Select whether or not to create a synthetic full – synthetic full will build a full backup out of the restore points already located on the backup repository. Also select the Days to create the synthetic full on. If creating a synthetic full you can also specify whether to transform any previous backup chains to rollbacks
- Select whether to create an active full backup (retrieves all source data from the datastore), and specify to create it monthly or weekly and specify days.
- Maintenance Settings
- Storage Level Corruption guard – check to periodically perform a health check on the latest restore point in the backup chain. Helps to prevent a corrupted restore point. If the health check discovers a corrupt block it starts the health check retry and transfers the data from the source datastore to a new backup file or the latest backup file, depending on the scenario – explained above.
- Full Backup File Maintenance
- Remove deleted VMs data after – specifies the number of days to keep backup data for VMs that have been deleted from the production environment
- Defragement and compact full backup file – check to perform and schedule a compact operation. Compact will create a new empty file and copy data blocks from the full backup to it. If the backup files contains deleted VMs, they will be removed. More info above.
- Storage Settings
- Data Reduction
- Enable Inline data deduplication – clear the checkbox to disable deduplication.
- Exclude swap file blocks – By default Veeam looks at the NTFS MFT file to identify the blocks that make up hiberfil.sys and pagefile.sys and excludes them from the backup. Clear this checkbox if you prefer to back these files up
- Exclude delete file blocks – By default Veeam does not copy file blocks that were deleted ormarked as dirty. If you would rather copy these, clear this checkbox
- Compression Level – Select the desired compression level for the job (None, Dedup-Friendly, Optimal, High, or Extreme).
- Storage Optimization – Select the type of backup target you plan to use; Local Target, LAN target, or WAN target. Veeam will use different data block size to optimize backup performance.
- Check and provide a password to encrypt the backups on disk.
- If you enable this on an existing job a full backup will be created on the next run and it and all subsequent incrementals will be encrypted.
- Data Reduction
- Notification Settings
- Can be used to send SNMP notifications for the job, to customize notifications on a per job basis rather than utilizing the global settings.
- Can also set backup details to a VM attribute of choosing, either overwriting or appending.
- vSphere Settings
- Choose whether to enable VMware tools quiescence.
- Choose whether to use Change block tracking and whether to enable CBT on existing VMs.
- Integration Settings
- Chose whether to enable backup from storage snapshots and whether to limit processed VM count per storage snapshot to a certain number. You can also chose to failover to a standard backup if Veeam fails to create the storage snapshot. If using NetApp you will also have the option to failover to primary storage snapshots.
- Script Settings
- Job Scripts
- Chose to run a script before and/or after the job and specify a path to the script.
- Chose when to run them – every X backup sessions or on selected days only.
- Job Scripts
- Backup Settings
- Backup Proxy has a couple different options
- Guest Processing handles the following options
- Application Aware Processing
- Enable/Disable application aware processing.
- Clicking Applications gives you the following options on a per-VM basis.
- Chose whether to require successful VSS application processing, try application processing but ignore failures or disable application processing
- Chose how to handle transaction logs if this is a SQL server, Oracle, or Exchange VM, either process the logs (additional settings will be required on the SQL tab) or perform a copy only of the logs (logs will just be copied in backup as a normal backup would copy files)
- SQL – additional settings for logs if you chose process from last step.
- How to handle transaction logs, either truncate, don’t truncate, or backup logs periodically
- If last option is chosen we need to specify the interval to backup transaction logs (in minutes) and a retention policy, either in days or until the corresponding image level backup that the logs are attached to is deleted.
- We can also specify which server to use as a log shipping server, the server that transports the logs. We can let Veeam automatically figure this out or specify a specific set of Log Shipping servers. Log shipping servers are just any Windows servers added to our Backup Infrastructure.
- Oracle Account – specify a user account that has SYSDBA rights on the database. If you chose to use guest credentials, Veeam will use the account setup within the guest processing section to connect.
- Log Action – here we chose what to do with our logs, we can set it to not truncate logs at all, truncate logs older than X number of hours, or truncate logs over X number of GB.
- Log Backup inteval – set the transaction log backup interval in minutes – default is 15.
- Retention – specify how long to keep the logs, either last X number of days or until the corresponding image level backup is deleted.
- Log shipping server – just as with SQL we can either let Veeam automatically pick a log shippiong server or set this to a specific set.
- File Exclusions – we are able to exclude certain files, folders or filetypes on a per VM basis.
- First select to either disable file level exclusions, Exclude certain files and folders, or include only certain files and folders.
- If excluding or including we can click add to add certain files or folders. We have the option here to use environment variables, such as %windir% or file masks like * and ?
- First select to either disable file level exclusions, Exclude certain files and folders, or include only certain files and folders.
- Scripts – we can use this section if we plan to back VMs up that do not support VSS in order to run scripts to obtain application consistent backups.
- Script Processing Mode – select wehter to require successful script execution, ignore script execution failuer or disable script execution
- Windows Scripts – path to windows scripts
- Linux Scripts – path to Linux scripts
- VM Guest OS file indexing – can select from the following options on a per-VM basis; Disable Indexing, Index everything, Index everything ecept and specify files/folders.
- If indexing Linux OS, several components need to be installed on the VM. mlocate, gzip, and tar. Veeam will prompt to install these if they are not found.
- Guest OS Credentials
- Select default credentials to use to deploy runtime processes. Any VMs that don’t have credentials explicitly assigned to them will use these credentials.
- Guest Interaction Proxy
- Specify a Guest Interaction Proxy to use or let Veeam Automatically choose one.
- Application Aware Processing
- Run job automatically
- Daily – runs job daily at a selected time on selected days
- Monthly – runs job monthly at a selected time on selected days
- Periodically every – runs job periodically ever X number of hours/minutes or continuosly to continuosly run job
- After this job – Allows you to chain this job to the ending of another. This will only be ran if the first job is started by a schedule – manually running the first job will not run this job.
- Automatic Retry
- Select to retry VM X number of times with X number of minutes in between each try upon job failure
- Backup Window
- Can set the job to terminate if it exceeds a specified backup window
- Run job automatically
- Summary – review settings and save job. You do have the option to immediately execute the job here as well.
Picture yourself in this scenario – you walk into work on a Monday morning where you are promptly greeted by just about every IT staff member in the building. They quickly tell you that certain mission critical services are down and the company is losing money as we speak. But that’s OK, you are here, the infamous backup guy – the guy no one cares about unless things go down. None the less you sit at your desk and begin the restore process. “No problem”, you say, “services should be back up in 10 minutes or so…”. The VM is fully restored and powered on – you sit, watching the console and the Windows spinning wheel and all of a sudden you see this!
Yikes! Your backups, just like their production counterparts are corrupt – you try a different restore point, no go, they are all corrupt. Believe it or not this is a common scenario that is played out inside of organizations everywhere. Backups, just like any other production workload or service need to be tested in order to ensure that they are indeed restorable. The best way of testing these backups is definitely done by performing full restores on the data – however doing so after each and every backup job can be quite time consuming and inefficient in terms of resource usage.
Nakivo does a great job at backing up and protection your VMware and Amazon environment and has a lot of features included within their product, Nakivo Backup and Replication. I’ve previously reviewed the Nakivo product as a whole here if you’d like to check it out. This post, will focus more on one feature – a feature that helps to prevent situations like the one above – Nakivo Screenshot verification. There is no worse feeling than having terabytes of backups that prove to be unreliable and in turn, useless – which is the exact reason why Nakivo has developed Screenshot Verification inside of their Backup & Replication software – to give you piece of mind that when push comes to shove, your backups will indeed be restorable!
What is Screenshot Verification?
Screenshot verification is a simple concept with a lot of underlying technology at play – in its basic form, Nakivo will verify each VM backup after a new restore point is completed. This is done by booting the VM directly from its’ corresponding deduplicated and compressed backup files located on a Nakivo backup repository, a process that Nakivo calls Flash VM Boot. During a Flash VM boot Nakivo creates a new VM on a specified ESXi server. It then takes the disks from within the backup files and exposes them as iSCSI targets, upon completion, the disks are mounted to the new VM as vRDMs. A snapshot is created in order to provide disposal of any changes and the newly created VM is powered on, isolated from the production network. Once booted, Nakivo utilizes VMware tools in order to take a screenshot of the booted OS. After the screenshot is taken the newly created VM is discarded and backup files are brought back to a consistent state, and the screenshot is included within any job reports, either generated through the UI or emailed.
It’s this screenshot that gives you the “piece of mind” that when the time comes to restore your VMs, they will indeed be restorable! A simple picture of the Windows login screen or Linux bash shell, or lack there of, certainly would’ve helped in the above scenario – alerting us that the next time we try and reboot our production VM or restore from a backup that problems may occur – giving us the leeway and time to fix the situation or restore to our last known good restore point on our own terms rather than doing so during an emergency.
How do we set this up?
As far as how to setup and configure Nakivo Backup and Replication as a whole I would recommend checking out my previous review here – but focusing solely on Screenshot Verification let’s go through the steps below… **Note, we are setting this up for one of our backup jobs, however we can also enable screenshot verification for our replication jobs as well **
Screenshot verification, all be there a lot of moving parts underneath is actually a simply Enable/Disable feature within the backup job. Nakivo has done a great job of abstracting away all of the complicated technology underneath and presenting us with some simple and easy to use configurable options. On the last step of a job wizard, we see the Screenshot verification setting at the bottom of the first column (as shown below)…
Upon selecting ‘settings’ we are presented with some more options which we can configure. The target container is the placeholder in which we will register the newly created VM that will be mounted to the backup files. This can be your average vSphere object that VMs belong to such as a host or cluster. Target datastore is where we would like to place the configuration files (vmx) of the VM that is created. Verification Options allows us to do things such as limit the amount of VMs which will be verified simultaneously. Running too many VM Screenshot Verification tests at once can produce a heavy load on your backup repositories, causing major delays in boot time depending on your hardware configuration – it’s best to tune this to your liking. Also configurable here are things like RTO, which in this case defines the number of minutes that the VM has to fully boot and initialize VMware tools. If this time is exceeded, the VM will be labeled as failed and the placeholder VM is discarded. We can also set the delay between when the guest OS has booted, and the actual execution of the screenshot.
Honestly, this is all we need to do! Simply save your job and on your next job run Screenshot verification should take place. As shown below, we can see the events that take place within vCenter during a Screenshot verification test, along with the placeholder VM that is created in order to perform these tests, noting the creation and deletion of the VM, along with any required iSCSI setup. This is all automated by Nakivo and requires no manual setup on your part.
So we have now seen that the Screenshot verification has been executed, but what does it look like in one of the reports/emails. Right-clicking any job within Nakivo gives us the ability to run a few reports – the one we are most interested in now is the ‘Last run report’. After generating and opening the ‘Last run report’ for our job with screenshot verification enabled we should see new information included in the report. As shown below we see that we have a ‘Last verification’ row now, indicating whether or not that the screenshot verification was successful – in addition, we can also see the actual screenshot that was taken by Nakivo. Below we see the actual login screen, giving us a pretty good indication that if we were to restore from this backup we would be successful.
Hey, Let’s have some fun!
As you can see, Screenshot verification is a very valuable tool giving us that piece of mind that our backups are actually restorable. But where’s the fun in that right? Let’s break some stuff and see how Screenshot verification reacts….
So, on my production VM let’s mimic some corruption and see if we can’t get Nakivo to detect it before we do! In order to do this I’ve run the following commands on my production VM within an administrative console (***NOTE*** Don’t do this in production, please, please don’t do this in production )
takeown /F C:\Windows\System32\WinLoad.exe
cacls C:\Windows\System32\WinLoad.exe /G administrator:F
bcdedit /set recoveryenabled No
The first three lines are pretty self explanatory, taking ownership, assigning rights, and deleting WinLoad.exe – the file that actually executes the loading of Windows upon boot. The last line simply disables the automatic repair, Microsoft’s line of defense for preventing people from doing stupid things like this Anyways, we’ve essentially botched our server here, however we won’t notice until we do a reboot, something that probably doesn’t happen that frequently in a production environment – thus, it’s probably going to go unnoticed for quite some time – that is, unless we are utilizing Nakivo’s screenshot verification on our backup jobs.
Let’s go ahead and run our backup job again on this same VM. This time, we will see Nakivo report a failure on the backup job, specifying that screenshot verification has failed – upon further investigation, we can see below what appears on the console of our VM that used for the verification, and is exactly what would happen to our production VM if we were to reboot it! Even though our newly created backup is not restorable, at least we now know and it won’t be a surprise to us in an emergency situation like the previous scenario. This gives us time – time to come up with a plan, whether that be restoring from a known good backup, coming up with some sort of failover plan or even building a new server.
So in the end screenshot verification proves to be a very valuable tool in any backup administrators belt – whether that being knowing that your backups can be restored successfully, or sometimes even more important, knowing that they can’t – and in some cases, Screenshot verification can be leveraged to prevent production outages by getting a preview of things to come upon the next reboot! The Flash VM Boot technology makes Screenshot verification a no-brainer in my opinion. If you are using Nakivo, you should be enabling this on all of your mission critical VMs. To learn more about Screenshot verification and other Nakivo features check out their help center here. Fancy trying it for yourself? You can get a full featured trial here, or if you are a VMUG member, VCP, or vExpert why not grab a free NFR license to tinker with! If that isn’t enough options for you Nakivo also offers a fully featured free edition – yes, all of the same features of their premium paid versions, just limited to a couple VMs. Thanks for reading!
Nakivo, a backup company based out of Silicon Valley has been providing backup and replication software to the world since late 2012. Today we will not focus so much and getting Nakivo up and running, we’ve already done that thoroughly here, but instead we will take a look at one individual feature; Instant Object Level Recovery for Microsoft Active Directory. Let’s face it – mistakes happen – users get deleted, OU’s get wiped out, security groups get completely out of sync. This is all stuff that happens, and happens more often than we know it. Certainly performing a complete full restore of a domain controller can be a little bit over the top just to get one individual user back (depending on who it is I suppose ), which is why Nakivo has been providing a means for restoring these individual Active Directory objects since their 5.5 release back in March of 2015. Today we will take a more in-depth look at just how we perform these restorations. Rather than simply showing how things are done I thought I’d have a little more fun with it this go around, put a little story behind it for all of our enjoyment 🙂 With that said, let’s dive in!
Let’s paint the scene – you are a sysadmin working for a pretty famous hockey club based out of Montreal. You are using Nakivo to protect a couple of datacenters, one in Montreal and another in Brossard, with a fully virtualized Active Directory. One morning for whatever reason your supervisor was a little off his game – maybe it was too much wine the night before, or perhaps he had a heaping of bad poutine at lunch, but when asked to disable and enable certain players directory accounts after a blockbuster trade, he had a slip up. Sure, he disabled the “psubban” account of the outgoing player as he was asked to, however in the process of creating the new “swebber” account, somehow he ended up deleting Andrei Markov’s account (amarkov).
It wasn’t until Andrei showed up for practice that morning that anyone noticed – Andrei attempted to log in and quickly realized that something was up. When the helpdesk ticket finally made its way to to your supervisors desk he knew immediately what had happened and quickly called upon you to help out. “No worries”, you said, “We’re protecting that server with Nakivo!”
How can Nakivo help you?
Thankfully you had already setup a backup job which processes a domain controller belonging to the canadiens.local domain, the same domain the user was accidentally deleted from. We won’t go into the nitty-gritty details of how to setup the backup job here, as this post focuses solely on the recovery, but we have covered it in detail in another post if you’d like to check it out. But instead we’ll go through the steps for us to restore Andrei’s account – The first thing we need to do is fire up his browser and log into Nakivo Backup and Replication. After logging into the application, simply selecting ‘Microsoft Active Directory objects’ under the ‘Recover’ menu kicks off the process (shown below).
The next step is quite simply and pretty self explanatory – we simply need to select the backup of our domain controller, in our case its named MSDC, and then select a desired restore point to restore from. As shown below we also have the option to ‘Automatically locate application databases’, which is checked by default. If we happened to know the exact location of the MS AD database then we could uncheck this an specify the path, and in turn maybe save a little time as Nakivo wouldn’t need to scan for the ntis.dit file. Honestly though, the amount of time it takes Nakivo to locate the Active Directory database is trivial, so let’s leave this checked, and click ‘Next’.
Nakivo will now take a moment to load the desired restore point and display it to us. The amount of time this takes greatly depends on the size of your Active Directory infrastructure. Canadiens.local is relatively small, and took only a few seconds to load – but before we move on to the next step it’s good to go over what is happening behind the scenes here. Nakivo Backup & Replication is actually scanning and mounting the server directly from within the compressed and deduplicated backup file – at no time does it perform a full recovery of the VM itself, saving us valuable time as we only need to restore that one individual object. As shown below we are presented with a screen on which we can browse through the entire Active Directory infrastructure and find the object we’d like to restore. It should be noted here that Nakivo supports object-level recovery for not just users, but containers and groups as well – so if it was an Organization Unit or Security Group that was deleted we would be able to restore it in the same manner. Next we select the object by simply clicking the checkbox beside it, and then click ‘Download Selected’. Alternatively we could click ‘Forward Selected’ to have Nakivo email out the ldif files to be used for import. At this point we will have a couple or Recovery settings we can specify; User will be disabled – will restore the user with the account disabled or User must change password at next logon – Nakivo automatically generates a new password for the restored user, and sets the ‘Change password on next logon’ flag in AD. Any password Nakivo generates will be stored in an included ‘Passwords.txt’ file added to our download.
After downloading the recovery bundle (should come in a .zip format) we can now get started on restoring Andrei Markov’s account back into the canadiens.local domain. We does this by first extracting the bundle and copying the extracted folder back to his domain controller. Since we are importing a user object back into Active Directory we need to have ldaps, or certification services enabled and configured on the domain controller. Thankfully the canadiens.local domain is already setup this way, however if we need to implement ldaps there is a great post here on how to go about it. Once we are back on the domain controller console we can simply open up an administrative command prompt and run the following command…
ldifde –I –t 636 –f filename –j logfolder <- where filename is the path the the downloaded ldif from Nakivo and logfolder is a path for import logs to be placed.
We can see a screenshot below of the before and after shots of the canadiens.local domain, with the after showing that Andrei Markov’s account has indeed been restored.
With that you can now breathe easy as Andrei’s account is fully restored back into Active Directory, including all of his user attributes, group memberships, etc. Honestly, it’s as if it was never deleted! This whole process moves very quickly within Nakivo, honestly, within minutes – and when the time comes where you need to do a restore, especially one revolving around user access, time is most certainly of the essence. Nakivo could certainly shave even more time off this process by implementing some way to automate the ldif import, or import directly back into the production VM – but honestly, the simplicity of this whole process far outshines the fact that it needs to be manually imported. For now, you and your supervisor can get back to what matters most; the quest for Lord Stanley.
If you would like to learn more about Nakivo’s Instant Object Recovery for Active Directory or any other feature they offer I highly recommend checking out their help center here, where you can find items such as their knowledge base, release notes, and a very well written user guide. Also if you want to check it out for yourself you can get a full featured trial here, or if you are a VMUG member, VCP, or vExpert why not grab a free NFR license to tinker with! If that isn’t enough options for you Nakivo also offers a fully featured free edition – yes, all of the same features of their premium paid versions, just limited to a couple VMs. Thanks for reading!
Finishing off Module 4 of the Veeam VMCE v9 Study Guide we will take a look at configuration backups, along with what can be set in terms of global notifications within Veeam Backup & Replication version 9.
Configuration Backup and Restore
A configuration backup essentially takes our Veeam Backup & Replication database and saves it to a backup file on the repository. The database data is then written to a set of xml files and archived into a (.bco) format. If for any reason our backup server experiences a failure we can simply reinstall a new backup server and quickly restore the old configuration. We could also use this backup to deploy another Veeam Backup server in the same environment. If you plan on migrating configuration data to another server be sure to stop and disable all running jobs before creating the backup or sessions may fail after restoring.
A configuration backup contains the following information
- Backup Infrastructure Components and objects – all hosts, servers, proxies, repositories, Wan accelerators, jobs, global configuration settings, etc..
- Backups – Backups, replicas and backup copies (information regarding the backups, not the backups themselves)
- Sessions – historical session information
- Tapes – libraries connected to the server
By default Veeam will create a configuration backup daily and store it in the default backup repository. That said, it’s best to redirect this to a different repository that doesn’t reside on the backup server itself. When you create a new repo, Veeam will offer to store the config backup on it, clicking yes will redirect NEW configuration backups to this repository. Old configuration backups REMAIN on the default repository.
If you have created a password within the password manager on the backup server Veeam will enforce that you encrypt the configuration backup. If you do not encrypt the configuration backup and there is a password present, Veeam will disable the configuration backup job. Also, without encryption the credentials will not be backed up with passwords within the configuration backup – you would need to enter all of the passwords again upon restore.
There are a couple of options when it comes to restoring
- Data Restore
- useful if the database gets corrupted, the SQL server hosting the database becomes corrupt or you deploy the database on a new SQL server, rolling back to a point in time or restoring data to a new database on the same SQL server.
- Data Migration
- used when you want to move the backup server and the configuration database to another location.
- If you forget your encryption password need for the restore you have the following option
- If the backup server is connected to Enterprise Manager you will be presented with a I forgot the password link.
- need to have enterprise or enterprise plus, and enterprise manager connected to the backup server
- Veeam will launch the encryption key restore wizard, at the request step a key will be generated, this can be copied or emailed.
- Within Enterprise Manager go to Configuration-> Key Management and click Password Recovery, and paste the key that was generated.
- Once the response is generated, copy or email that key.
- Back in the Encryption key restore wizard enter the copied response, upon completion VBR will apply them to the encrypted backup file and unlock all content within it.
- Backup and replica catalogs along with session history are optional when restoring a configuration backup
- Veeam can automatically setup your powershell policy for you during restore
- Veeam can back up existing databases before restoring over top of them.
- You can specify new passwords for the backed up credentials if they have changed between the backup and restore times.
- After a restore has completed a components upgrade will be checked and ran.
- After a restore has completed VBR can perform a sync operation for backup/replicas created on the server and tape libraries connected to it. This is ran if
- you restored a database from a backup created on 7.0 in restore mode
- you restored a database created with 8.0 in restore mode and selected to restore data from the backup and replica catalog.
You should also follow the below pre-reqs before restoring a configuration backup
- Stop all running jobs
- Check version of backup server. For instance v9 can restore configuration backups from 7 update 4, 8, and 9
Global Notification Settings
Veeam Backup & Replication can be setup to send out some alerts and notifications globally – some of which can be overridden on a per job status, but this section will just focus on global notifications.
Setting up notifications settings within Veeam is done through the Options option of the main menu on the email tab. From here we can specify things such as the smtp server to use, it’s port and authentication methods. We can also customize what our notification settings in terms of jobs look like for instance
- to – who the email goes to, anyone setup in this global area will receive notifications about every job ran on the system. Can be left empty if you wish as we can define additional emails to get notifications on a per job basis
- Subject – contains the following variables for use %time% (completion time), %jobName%, %jobResult%, %VMCount% and %issues% (number of VMs with warning or failed status).
- We can choose whether to notify on success, warning, and/or failure.
- Suppress notifications till the last retry
Aside from job messages we can also setup other notifications from VBR on the notifications tab such as
- Low Disk Space – Veeam will check disk space on datastore and target repository and include a warning message if it is below a certain threshold (warning is in the job session details). The threshold is in terms of percent on the backup storage, and in terms of GB on the datastore details.
- Support Expiration – By default, Veeam will warn all email recipients about the support expiration up to 14 days before it expires. This is included in every email notification sent from Veeam. This can be disabled here.
- Update Notifications – When enabled Veeam will automatically check for new product version and patches from the Veeam website.
Veeam can also send SNMP traps with the status of the jobs performed on the backup server. SNMP traps can be sent to 5 different destinations. From the SNMP tab input your receiver and community information and setting up your service properties with the Windows SNMP service are requirements to make this happen. Then, from within your job you simply check the Send SNMP notifications for this job check box within the Notification tab of the Job Options.
I didn’t really see WAN Acceleration mentioned anywhere within the course description of the VMCE class, so I decided this might be the best place to fit it in since we will be talking about managing network traffic in Module 4. That said, I’m sure the topic will be brought up again in later modules, however let’s go over what we can here!
WAN Acceleration is Veeam’s answer to help optimize VM traffic that will be going over the WAN. It does this by deploying at least 2 WAN Accelerators on 64 bit Windows Servers, one located at the source, and one located at the target. If you remember back to Module 3 we spoke a bit about WAN Acceleration so some of this may be a repeat, however its good to know for the exam.
Configuring WAN Accelerations happens in the following way
- Configure Source side WAN Accelerator, then the target.
- Launch the New WAN Accelerator wizard from the Backup Infrastructure view
- From the Server step
- specify the Windows Server you wish to use for the accelerator
- provide a description
- Traffic Port – Specify network port used for source to target communication – defaults to 6165
- Streams – Number of connections that must be used to transmit data (defaults to 5). Keep in mind as this number increases so will the bandwidth and accelerators resources it requires. Applies only to the source WAN Accelerator.
- Cache – location of service files and global cache
- Folder – Path o location where service files (for source and target) or Global Cache (target only) must be stored. Defaults to c:\VeeamWAN. It’s also best not to nest these deep in the file system as service file names can be very long, no use in making them longer.
- Size – Specify a size for the target WAN Accelerator according to the sizing best practices – we will go over this below
- Review components to be installed (data mover service, WAN Accelerator service) and click ‘Next’ to finish.
Clearing/Populating Global Cache
These process can all be accomplished by right clicking on the WAN Accelerator within the Wan Accelerators node in the Backup Infrastructure view and selecting the desired operation (process explained below)
WAN Accelerator Sizing
As mentioned above there are some best practices we need to take when correctly sizing how much space we need for WAN Accelerators, both source and target.
Source WAN Accelerator
- Veeam analyzes data blocks that will go to target and digests them, these are stored in our source accelerator.
- Size of cache on source accelerator depends on the capacity of all our source VM disks.
- Every 1TB of data requires 20GB of cache space. IE if you have 4TB of VM disks you are backing up, you should provide 80GB of cache on the source accelerator.
- There is no global cache on the source, only the digest metadata is stored here. Global is just for target accelerators.
Target WAN Accelerator
- This is where our global cache is stored.
- Global Cache is basically a library that holds data blocks that go from source to target.
- Populated fully on the first cycle of a job.
- If a new data block is constantly sent across the WAN, it will be added to the global cache.
- If an already cached block is not sent over the WAN after a period of time, it will be removed from the global cache.
- If a periodic check deems a block in the global cache is corrupt, it will remove it.
- Global cache can copy blocks stored from one source accelerator folder to another source accelerator folder if they are the same, meaning if we have two locations each replicating a Windows 2012 server, we can simply copy blocks from the first cache to the second cache without having to send them across the WAN.
- The Global Cache can be pre-populated without actually running the job.
- Useful on the first run of a job so all data blocks don’t need to be copied
- Useful if the cache becomes corrupt to prevent all data blocks to be copied again. This requires you to clean the cache first
- Encrypted backups are not used for population
- You cannot start any jobs using the accelerator while the cache is being populated.
- Veeam uses data blocks stored in specified repositories to populate the cache – only OS blocks are copied.
- That said if there is other accelerator cache already located in the target, it will match OSs from the source repository and copy these blocks directly from the already existing cache folders if they exist.
- Copied to a default cache folder, when a remote job starts Veeam renames this to the source accelerator used in the job.
- Recommended to provide 10GB of cache per every type of OS utilized. (defaults to 100GB, so 10 OSes). IE – say we backup 10 VMs (1xWin7, 6xWin2008, 3xWin2010) we should provide at least 30GB (3 OS types x 10GB).
- If the Digests data on the source accelerator is missing or for some reason cannot be used, the target accelerator will have to re-calculate this, therefore, will require space to do so. Therefore the same rule of source sizing applies also to target, in addition to the OS type cache allocated. IE those 10 VMs also occupy 4TB of space we will need to add 80GB (20GB/TB * 4) more cache space in addition to our OS cache. So 80GB for digest calculation and 30GB per OS caching = 110GB total.
- All this said, Global Cache is calculated per source accelerator. Within Veeam we have the ability to apply a many to one situation, meaning many source accelerators running through 1 target accelerator. This changes our cache size exponentially depending on the number of source accelerators. The formula is as follows
- Total Cache Size = (number of source accelerators) * ( Size of target WAN accelerators properties [10gb/OS]) + 20GB/TB of source data.
- Let’s say we add a second source accelerator to our example we have been using. The second accelerator has 1TB of source data spread across 2 OS types (Linux, Server 2003). We would end up with the following for a global cache size
- Total Cache size = 2(we have two source accelerators) * 50GB (5 OS types [Linux, server 2003, server 2008, server 2012, win7) at 10GB per) + 100GB ( 5TB of source data spread across the 2 source locations)
- 2 * 50GB + 100GB = Total Cache Size of 200GB
- With all of this, if you have the space it’s best to add as much as you can in order to obtain more efficient acceleration as it would be able to hold more repeating data blocks.
Data Block Verification
Veeam calculates checksums on blocks being transferred between source and target to help ensure that no corrupt blocks are stored in the global cache. This works in the following way
- Before sending, Veeam calculates a checksum on the block
- When the target receives the block it re-calculates this checksum (before it is even written to cache).
- The checksums are compared, if there is a difference, the target sends a request for the source to resend, upon receiving the block again, it is written to the global cache.
WAN Acceleration works in the following way
- If using a backup copy job, Veeam uncompressed backup file to analyze content
- Source accelerators analyzes data blocks and creates file with digest for blocks.
- Veeam compresses data and sends it to the target
- Target populates global cache with blocks from the copied file
- On the next job cycle, source analyzes data blocks in the file that need transferred and creates digests just for these blocks
- Source compares new digests with old – if duplicate blocks are found the file is not copied over the WAN. Instead, the target will pull this file from the global cache
- Also, restore points already existing on the target side are analyzed – if there is a duplicate located in them, the target will take them directly from the restore points.
Managing Network Traffic
Before we get into some of the ways we can throttle and manage our network manually, let’s have a look at a couple different ways Veeam manages network disconnects automatically.
Data Transport on WAN Disconnect
This type of reconnection attempt exists only on jobs who utilize WAN accelerators. Basically if a connection drops while we are transferring VM data between accelerators VBR will pick up and resume the job from the point where the connection was lost when services are restored, rather than starting all over again. When the connection is restored, VBR will initiate a new transfer process, this time writing data to a new working snapshot. If the connection drops multiple times, veeam will only keep 2 working snapshots on the VM by merging previous ones together. Once all data has made its way to the target, all snapshots are merged and a new restore point is created.
Resume on Disconnect
This process handles network disconnects not applying to accelerators, and handles disconnects between backup server, proxies, and repositories (storing replica metadata). VBR will attempt to reestablish the connection every 15 seconds for 30 minutes, picking up right where it left off.
Network Traffic Throttling Rules
Network throttling rules are setup and enforced globally on the backup server. They essentially limit the maximum throughput of traffic going from source to target. They are set with a pair of IP addresses, source ip, and target ip. If a component within the backup infrastructure fall into the specified source and target IP range, the rule is applied to them. The steps to set them up are as follows…
- Select Network Traffic from the Main Menu and click ‘Add ‘ in the Global Network Traffic Rules section.
- In the source ip range, specify a range of IPs representing the source components
- In the target IP range, specify a range of IPs representing the target components.
- Select the box to Throttle Network traffic
- Specify the maximum speed that must be used to transfer VM data to in the Throttle to field
- In the Apply throttling we can set up a schedule in which this rule will apply, or have it apply all the time.
- If a rule has overlapping schedules, the rule with the lowest maximum speed will apply
- Network Data Encryption is also setup in this same manner with the Encrypt network traffic checkbox. More on network encryption below
Managing Data Transfer Connections
By default Veeam uses 5 TCP/IP connections to transfer data from source to target. This may cause network traffic to be heavy if multiple jobs run at the same time. This can also be changed in the Global Network Traffic Rules settings using the ‘Use multiple upload streams per job’ selection box.
Enabling Network Encryption
By default Veeam encrypts data with 256-AES flowing to/from public IPs, however you may want to have encryption between your local/remote source and targets. Again this is done in the Global Network Traffic Rules window by clicking add. It’s the same process as setting up throttling rules (above), however checking the ‘Use Network Encryption’ box.
Specifying priority networks for transfer
VBR gives you the ability to specify what networks you want to send your VM data on. This is useful if you have some sort of backup network or non-production network that is utilized for backup data. Again from the Global Network Traffic Rules section we set this up
- Click on Networks
- Select to ‘Prefer the following networks for backup and replication traffic’ and click ‘Add’
- Specify a network in a CIDR notation or mask
- VBR will failover to the production network if for some reason the preferred networks are unavailable.