Last week, I completely virtualized my life. Well, at least my server life is all virtualized. I’d been running my Team Foundation Server in a virtual machine, but took the plunge to get my domain controller virtualized as well. One of the things I’d been planning to do was revisit my server backup strategy, especially in regards to disaster recovery. I’d been using traditional backup software on my servers, and even purchased the disaster recover option. However, when I tested the disaster recovery on a server, it didn’t fill me with a great amount of trust. With all my current and future servers virtualized, I felt I could simply copy the configuration and hard disk files for a machine and have the ultimate in disaster recovery. With a little bit of searching on the web and some quick development, I came up with a way to get a perfect backup of my virtual servers with around five minutes or so of total downtime per server! I thought others could find this useful so I thought I’d post it.
For my server virtualization, I use VMware Server. While many of us focused on Microsoft development use Microsoft’s Virtual Server, I’ve been using VMware’s products since Workstation beta 1.0 back in the ’90’s and have had very good luck with them. Throw in the fact that I can’t live without the USB support for virtual machines in VMware Workstation, and I’m a happy customer. With both VMware Server and Virtual Server being free, it’s not like it’s hard to pick one. Even if you’re a Virtual Server user, you can use the same techniques I’ll show here. If you’re using the Microsoft System Center Virtual Machine Manager 2007 (SCVMM), you’ve already got this functionality. I’ve been quite impressed with what I’ve seen of SCVMM combined with Server 2008, and plan to give them a very hard look in the very near future.
When it comes to backing up a virtual machine (VM), you have to get the VM in a safe state to back it up. With the free Virtual Server, that means shutting down the machine. After that, it’s just a really long running file copy or archive operation. I first looked at a nice script that numerous folks have contributed to in the VMware Server forum. The problem is that the script took hours and hours to run because copying or zipping 10+ GB .VMDK files is not the quickest thing in the world. If you can live without various servers for long periods of time, that’s fine, but knew that a little bit of Volume Shadow Copy Service (VSS) mojo could work wonders. (No, I didn’t forget the ‘C’ in the VSS abbreviation, Microsoft calls it VSS.)
The Volume Shadow Copy Service is a very cool feature that’s been in Windows since Windows XP and was designed to help backup applications to get a stable version of files and disks on running systems. While it’s been in Windows for a while, Vista was the first version to make these shadow copies accessible to end users. Right clicking on a drive, folder, or file in Vista and select Restore previous versions from the context menu, and you see the Previous Versions tab.
Previous versions, also called a restore point, are created once a day on Vista and generally when there’s a change to your computer, like when installing a hot fix. You can create a restore point on your own by going into System, clicking on the System Protection link, and clicking on the Create button.
The VSS API is exposed to developers and you can read more about in at this link. My original goal was to write a program that created a shadow copy of the drive where my VM files resided and do the copy from that. As I looked at the VSS API, I realized that it wasn’t going to be as easy as I had hoped. Fortunately, the VSS SDK comes with a sample program, VSHADOW.EXE that not only does the copy, it will expose the shadow copy as a drive letter. That means standard tools can easily work with the shadow copy and not know that it’s dealing with something special. There are two hang ups with VSHADOW.EXE. The first is the VSHADOW.EXE program isn’t the easiest tool to use because it assumes you know everything about VSS. The second is that you have to use the version of the program that is compiled for the OS you’re using. As I can’t give you the VSHADOW.EXE program, you’ll have to download the VSS SDK yourself to get it for Windows XP or Server 2003. For XP, you’ll find the appropriate version in “<VSS SDK install dir>TestAppsvshadowbinrelease-xp”. For Server 2003, it’s in “<VSS SDK install dir>TestAppsvshadowbinrelease-server”. For Vista (and I would presume Windows 2008), you’ll have to download the Platform SDK and you’ll find it in “C:Program FilesMicrosoft SDKsWindowsv6.0ABinvsstools”. As for figuring out VSHADOW.EXE, reading Adi Oltean’s blog and a little experimentation got me in the VSS groove. Adi, a developer on VSS, showed how to use VSHADOW.EXE to Copy an Opened File, and, the one that really helped me out, How to Assign a Drive Letter or Directory to a Shadow Copy.
While I could have been cool and used the VMware COM API to control VMware Server, I wasn’t that ambitious so I used the VMware server command line tool, VMWARE-CMD.BAT. I also could have been a true L33T developer and done everything in PowerShell, but didn’t have the ambition for that either. I guess I took the easy way out and did a couple of batch files. They were quick and did what I wanted them to do. You can download the batch files here. Before you run them, make sure you read the following so you know how to use them.
The main file you’ll use is FastVMBackup.CMD and it has the following assumptions in it:
- VMWARE-CMD.BAT is in the path. The default installation path is C:Program FilesVMwareVMware Server for 32-bit systems and C:Program Files (x86)VMwareVMware Server for 64-bit systems.
- In the same directory as FastVMBackup.CMD is the appropriate VSHADOW.EXE program for your OS.
- The directory where FastBackup.CMD resides is writable for the account starting FastBackup.CMD.
- The support batch file, MapShadowCopy.CMD, is in the same directory as FastBackup.CMD.
- Your custom batch file that does the actual copying for your environment is in the same directory as FastBackup.CMD. (More on this batch file in a moment.)
FastVMBackup.CMD has three required and three optional parameters.
The required parameters are in the following order:
- The full drive, path, and name of the .VMX file for the VM you want to back up. For example: F:TeamSystemTeamSystem.vmx.
- The drive to map the VSS copy. You must include the colon on the name. This drive letter is checked that it doesn’t already exist. This drive letter is passed to the copy batch file in the third parameter.
- The batch file you want to do the copy or archiving for your VM. By not having the copying hard coded in FastVMBackup.CMD, the script is far more flexible and useful for many of you.
The optional parameters are any additional items you want to pass directly to your copying batch file.
Here’s the command I use to backup my VM named Marx: FastVMBackup.CMD e:MarxMarx.vmx X: ZipVM.cmd Marx
FastVMBackup.CMD is relatively simple. The basic operation is:
- Shut down the VM with a call to ‘VMWARE-CMD <.VMX> stop soft’. This is the same as pressing the machine power button so you get a clean machine shutdown.
- MapShadowCopy.CMD is a slightly tweaked version of Adi Oltean’s batch file.
- It first creates the VSS copy of the drive and calls itself recursively to do the drive mapping to the newly created snapshot.
- It writes out to another batch file, FVMB-DeleteVSS.CMD, the call to VSHADOW.EXE to delete the drive mapping.
- Restarts the VM so you have minimal downtime.
If FVMB-DeleteVSS.CMD was created, which means the VSS snapshot worked
- Calls the batch file you passed in on the command line to do the actual work of copying or archiving. The first parameter to this batch file is the drive letter mapped to the VSS copy. Parameters two, three, and four to this batch file are the optional parameters passed to FastVMBackup.CMD.
- Calls FVMB-DeleteVSS.CMD to delete the shadow copy previously made.
- Deletes FVMB-DeleteVSS.CMD.
On my server running VMware Server, I have each VM running on their own dedicated hard drive as I, and
others, have found it much faster. Disks are dirt cheap, don’t fight it! If you want to back up multiple virtual machines that are running on the same disk, you may want to tweak FastVMBackup to take multiple .VMX files on the command line. VSS does whole drives at a time so you would just need to shut down all of your machines before letting MapShadowCopy.CMD do the snapshot. Obviously, that will increase your downtime on all but the first machine passed in.
While FastVMBackup.CMD and friends isn’t too exciting on a technical level, I couldn’t find any similar implementations out there when I looked around. If you’ve got the cash, definitely get something like SCVMM or VMware Consolidated Backup. If you’re a small shop, at least you’ve now got something that should keep your downtime to a minimum. Let me know if the comments for this blog if you find any problems or add enhancements.