Snapshots Filled Out The Datastore Disk Space


Have you ever been in a situation where you forgot VM in snapshot mode, snapshot that has grown up to fill out the datastore disk space, leaving just 10MB, putting down the VM itself but also compromising all other VMs located on that same datastore? Well that’s what happened to a customer of mine and hopefully we could get him out of trouble using VMKFSTOOL

Most of the time this is an issue that usually happens in an ESX3.x/VC2.x environment because of the lack of proactive alerts, alarms and reporting tools especially for the storage.  Let me tell you here a little trick that can save your life in such situation. Right after you have created your datastore (FC or iSCSI) you also create, on the same new datastore, a dummy file which is for instance 10% of the datastore original size. Free to you to adjust that percentage. The goal is to have a file large enough to allow later on a snapshot consolidation if necessary, or any other operation that requires disk space on the datastore.

So how it works?

Whenever you hit the situation above; no more disk space on the datastore, VM(s) down and won’t boot up anymore, boss screaming at you, … You ssh in any host that has access to that datastore and delete the dummy file, freeing up instantly 10% of disk space! Then you can eventually consolidate the snapshots and hopefully boot up that critical VM.  Don’t forget, when everything has settled down, to recreate that dummy file, same size or less.

Pro’s: 

  • Easy to setup,
  • Easy to remove whenever you’re struggling with space (rm dummy.file),
  • Avoid tedious, time consuming, manipulations with VMs to free up disk space,
  • Minimum downtime for the affected VM(s),
  • Helps you to stick with your VM’s SLA,
  • Do not overload your storage infrastructure by moving around VMs,
  • Etc..

Con’s: 

  • A certain amount of your storage is reserved for dummy files huh!
  • An extra step when creating your datastore.

How to: 

Make sure the CWD is set to the root of your datastore, something similar to /vmfs/volumes/datastore1, then type in the following command: 

dd if=/dev/zero of=dummy.file bs=<10% of the datastore disk space in bytes> count=1 

What about in vSphere?

Although this trick remains useful in a vSphere environment, vCenter has been updated with new management screens and capabilities such as raising alerts, alarms and improved datastore utilization reports, to enable management of over provisioned datastores. These enhancements provide administrators with proactive alerts and alarms to address issues before they interrupt the availability of applications running on those resources. There are other enhancements such:

  • Quota limits on datastores,
  • VMFS Volume Grow that complements the dynamic LUN expansion capability that exists in many storage array offerings today

Check out the What Is New in VMware vSphere™ 4: Storage for much more information about those new features.

Advertisements

About PiroNet

Didier Pironet is an independent blogger and freelancer with +15 years of IT industry experience. Didier is also a former VMware inc. employee where he specialised in Datacenter and Cloud Infrastructure products as well as Infrastructure, Operations and IT Business Management products. Didier is passionate about technologies and he is found to be a creative and a visionary thinker, expressing with passion and excitement, hopefully inspiring and enrolling people to innovation and change.
This entry was posted in Uncategorized. Bookmark the permalink.

5 Responses to Snapshots Filled Out The Datastore Disk Space

  1. Pingback: Tweets that mention Snapshots Filled Out The Datastore Disk Space « DeinosCloud -- Topsy.com

  2. Hussain says:

    hello,

    you are always hiting the right nails 🙂 Yesterday, i have just came back from 3 weeks vacation and found the same issue with some of my Datastores 2 of them has filled due to some heavy application development currently being tested and my workmate has taken the Snapshot Technology as a fun to play around with and forgot the actual size of the DataStore.

    Luckily, non of the VMs got hanged or stop responding, i have explained to him what went wrong and he is happy to learn new things from mistakes 🙂

    BR,
    Habibalby

    • deinoscloud says:

      Hi Hussain,

      Spot on, learn from mistakes is key leading to experience…

      Hope you enjoyed your vacation 😉

      Cheers,

  3. kopper says:

    nice the dummy trick

    what about the steps or commands you used for VMKFSTOOL you mention the tool but not the steps

    what are those?

    usually when you have more space in another datastore you just move the VM but when not?

    thanks a lot

    • deinoscloud says:

      Hi Kopper, we used the VMKFSTOOLS -i to clone the disk, and consolidate the snapshot, to another datastore. Then we recreated the VM and selected ‘Use an existing virtual disk’

      >usually when you have more space in another datastore
      >you just move the VM but when not?
      You could try to lower memory of the VM which will decrease the size of the swap file on the datastore, freeing up some disk space, maybe just enough to consolidate the snapshot, freeing even more disk space…

      Rgds,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s