Recently a colleague asked if it would be possible to provide 'secured' File Level Recovery to some vSphere 5.1 VMs in production. She suspected a Windows 2008 R2 Domain Controller may have become infected with a trojan through the recent Windows DirectPlay vulnerability (CVE-2012-1537) and she wanted to safely inspect its contents and allow for File Level Recovery while minimizing risk of further contamination.
One non-invasive way to do this is to use vmPRO 3.0.1, Debian 6.0.6 and ClamFS 1.0.1 to create a live VM A/V scanning File Level Recovery proxy. This can block access to harmful infected files, as well as notify you in the event harmful files are detected in your VMs or backups. In this example I used my main vmPRO virtual appliance which runs daily VM backups in the lab, and a Debian Linux VM I use for some light operations support tasks.
The first thing to do was to mount one of the vmPRO File Level Recovery CIFS shares to the Debian box so we could see the VM file volumes. There are two shares available on the vmPRO depending on how you'd like to access the file data: \\vmPRO\files will read from the active ESX VM disks, \\vmPRO\recover\files will read from your backups stored on a NAS. Reading from \\vmPRO\recover avoids any activity against the ESX hosts if you don't want them involved.
To do that, first I recommend installing the Debian cifs-utils package; this ensures that you can use/resolve host names in the CIFS UNC paths you try to mount:
sudo apt-get install cifs-utils
Then I make some directories under /mnt where we'll mount the vmPRO shares: mkdir /mnt/vmpro ; mkdir /vmnt/vmpro/files ; mkdir /mnt/vmpro/recover ; mkdir /mnt/vmpro/recover/files
Next I updated /etc/fstab with the path and share settings for the production vmPRO appliance; updating this file will make the mounts persistent after any reboot. Substitute your own hostname or IP address for the hostnames you see in the example here; once you have the file updated, you can ask Linux to mount all specified filesystems at any time with the command: /bin/mount -a
These are example lines for /etc/fstab -- the default CIFS login and password for the vmPRO shares are sysadmin/sysadmin. They should each be entered as one long line with no breaks:
//vmpro-gpn00.infra.n7.quantum.com/files /mnt/vmpro/files cifs defaults,ro,user=sysadmin,pass=sysadmin,uid=root,gid=root 1 3
//vmpro-gpn00.infra.n7.quantum.com/recover/files /mnt/vmpro/recover/files cifs defaults,ro,user=sysadmin,pass=sysadmin,uid=root,gid=root 1 3
You can also use a line like this to mount it manually any time you like outside of when /etc/fstab is read:
sudo mount -t cifs //vmpro-gpn00.infra.n7.quantum.com/files -o user=sysadmin,password=sysadmin /mnt/vmpro/files
Once that is done, you would see all of the vSphere virtual machines and their respective filesystems in the /mnt/vmpro/files folder, it looks something like this depending on how your Folders are laid out and the names of the VMs:
Any time you enter in to a .volume folder under the /mnt/vmpro/files mount point, it will trigger the vmPRO to request a snapshot of the VM on its corresponding ESX host, and its contents will be mounted automatically into the filesystem tree. After an idle period with no access against a .volume, the vmPRO appliance will dismount the disk and remove the snapshot from the VM on the ESX host.
In the same fashion any time you enter a .volume under /mnt/vmpro/recover/files it will mount the corresponding backup from the NAS storage rather than reaching out to the ESX host directly.
Now that we have the vmPRO file share mounted to our Debian box, the next thing we want to do is add the ClamFS packages and provide a configuration file for it to use. You can add the ClamFS packages to a Debian box like this:
sudo apt-get install clamfs
mkdir /etc/clamfs
vi /etc/clamfs/home.xml
In this example I have it configured to deliver directly to my local mailbox whenever ClamFS detects a virus/signature
Here we will use /usr/bin/clamfs with the home.xml to start ClamFS with the following command: sudo clamfs /etc/clamfs/home.xml
Now we can browse all of the contents of our VMs at /tmp/clamfs, and any files we attempt to open will first be passed through a scan in ClamFS before allowing us to open them. If ClamFS detects a malicious signature, it blocks access to the file and logs and audit event. In this case I have pointed it to the EICAR test virus example file:
At this point she could safely browse the filesystem on suspect Domain Controller, either the current state or from a backup on disk, and when the infected contents are located they will immediately generate a log alert and the file content will be blocked upon opening. Once the files are isolated and identified, she could then get direct access to the data through the /mnt/vmpro folders, bypassing ClamFS as needed for further analysis or post-mortem.
home.xml, 6 KB Permalink Reply by Casey Burns, PMM QTM on
