Have you ever been in a situation where you forgot VM in snapshot mode, snapshot that has grown up to fill out the datastore disk space, leaving just 10MB, putting down the VM itself but also compromising all other VMs located on that same datastore? Well that’s what happened to a customer of mine and hopefully we could get him out of trouble using VMKFSTOOL!
Most of the time this is an issue that usually happens in an ESX3.x/VC2.x environment because of the lack of proactive alerts, alarms and reporting tools especially for the storage. Let me tell you here a little trick that can save your life in such situation. Right after you have created your datastore (FC or iSCSI) you also create, on the same new datastore, a dummy file which is for instance 10% of the datastore original size. Free to you to adjust that percentage. The goal is to have a file large enough to allow later on a snapshot consolidation if necessary, or any other operation that requires disk space on the datastore.
So how it works?
Whenever you hit the situation above; no more disk space on the datastore, VM(s) down and won’t boot up anymore, boss screaming at you, … You ssh in any host that has access to that datastore and delete the dummy file, freeing up instantly 10% of disk space! Then you can eventually consolidate the snapshots and hopefully boot up that critical VM. Don’t forget, when everything has settled down, to recreate that dummy file, same size or less.
- Easy to setup,
- Easy to remove whenever you’re struggling with space (rm dummy.file),
- Avoid tedious, time consuming, manipulations with VMs to free up disk space,
- Minimum downtime for the affected VM(s),
- Helps you to stick with your VM’s SLA,
- Do not overload your storage infrastructure by moving around VMs,
- A certain amount of your storage is reserved for dummy files huh!
- An extra step when creating your datastore.
Make sure the CWD is set to the root of your datastore, something similar to /vmfs/volumes/datastore1, then type in the following command:
dd if=/dev/zero of=dummy.file bs=<10% of the datastore disk space in bytes> count=1
What about in vSphere?
Although this trick remains useful in a vSphere environment, vCenter has been updated with new management screens and capabilities such as raising alerts, alarms and improved datastore utilization reports, to enable management of over provisioned datastores. These enhancements provide administrators with proactive alerts and alarms to address issues before they interrupt the availability of applications running on those resources. There are other enhancements such:
- Quota limits on datastores,
- VMFS Volume Grow that complements the dynamic LUN expansion capability that exists in many storage array offerings today
Check out the What Is New in VMware vSphere™ 4: Storage for much more information about those new features.