Understanding VMFS Block Size And File Size

I was asked recently a question about the implications of the Block Size and the File Size when formatting a VMFS datastore and my answer was partially wrong. So I decided to deep dive into this matter over the week-end and came up with an article and a video which I hope will clarify the link between both Block Size and File Size.

But first the video:

From the vSphere 4.1 Configuration Maximums you can read the following information:
Item                                 Maximum
File size (1MB block size)    256GB
File size (2MB block size)    512GB
File size (4MB block size)    1TB
File size (8MB block size)    2TB minus 512B

What’s a Block Size also referred as a File Block Size?
The Block Size on a VMFS datastore defines two things:

• Maximum file size.
• The amount of space a file occupies.

How to determine the current Block Size of a datastore in an ESXi host?
SSH in your ESXi host and type in the following command:
vmkfstools -P /vmfs/volumes/ < Your VMFS datastore >

The output is similar to this:
VMFS-3.46 file system spanning 1 partitions.
File system label (if any): QNAPNAS02BIGLUN
Mode: public
Capacity 1239903371264 (147808 file blocks * 8388608), 1239274225664 (147733 blocks) avail <- File Block Size is 8MB
UUID: 4c4c0ff6-e78e2f7b-314c-000c29ff92b0
Partitions spanned (on “lvm”):
naa.60014058a60a1d5daadfd44bfd8d9cdf:1

N.B. You can also use vCenter Client to view the Block Size of any VMFS datastore.

File Block Size is 1MB, what does that mean regarding File Size?
Whatever the size of the VMFS datatsore (LUN), if you format it with a Block Size of 1MB, the maximum size of a file, a VMDK file for instance, is 256GB. With a 2MB Block Size, the File Size  is 512GB maximum, etc…

So you can format a 2TB VMFS datastore (LUN) with a 1MB Block Size, there is no problem doing that, but any single file such snapshot files, VMDK’s, ISO’s, etc… residing on the datastore cannot be bigger than 256GB in size. or 512GB maximum File Size for a 2MB Block Size, and up to 2TB maximum File Size for a 8MB Block Size.

N.B. There is no way to change the block size of a VMFS datastore once it is created. The data store must be deleted, recreated and reformatted with the required Block Size!

What about small files such the .VMX files, LOG files, etc…?
The VMware KB1003565 says:

VMFS3 uses sub blocks for directories and small files with size smaller than 1 MB. When the VMFS uses all the sub block (4096 sub blocks of 64 KB each), file blocks will be used. For files of 1 MB or higher, file blocks are used. The size of the file block depends on the block size you selected when the Datastore was created.

I had to read it few times to understand what VMware says here and still I’m not sure to get it completely right! So let me explain here what I’ve understood from the quote above and don’t hesitate to chime in with comments if I’m wrong.

First VMFS3 uses sub-blocks for directories and small files with size smaller than 1MB. Example of such small files are the .VMSD, .VMXF, .VMX, .NVRAM, and .LOG. files you can find in a guest folder for instance. Also you could have your own files, a bunch of scripts or config files for example. Note that directories use sub blocks as well.  
Sub Block is definitely a great thing but there is a drawback, you are limited to 4096 of those small files and directories per VMFS datastore! Once you ran out of sub blocks, VMFS uses the current Block Size which can be of 1MB up to 8MB in size.

N.B. Check the number of small files in your VMFS datastore with the following command: find -size -64k | wc -l
N.B. Check the number of directories in your VMFS datastore with the following command: find -type d | wc -l

What happens when the guest machine read/write small files from/to the VMFS datastore?
There is often a misunderstanding here and even myself I was wrong at some point. The virtual machine dictates the data block size. If the VM reads or writes 4KB, only 4KB will be read or written from/to the VMFS datastore and not a whole Block Size as you may think (and as I used to…). Another thing to pinpoint, sub blocks are irrelevant here, file systems on virtual machines’ VMDK’s don’t use sub blocks at all.

What about Block Size and performance?
There is a situation where using a large Block Size is better than small one performance wise and that is for thin provisioning. Let me refer you to Duncan Epping ‘s article where he demonstrates that with thin provisioning, when the VMDK extends in increment of xMB depending of the Block Size of the VMFS datastore, a SCSI locking occurs and if you use a small Block Size chances are greater that your VMFS datastore will ‘lock’ more frequently than if it was using a 8MB Block Size.

N.B. This might be not anymore an issue with vSphere 4.1 and the new VAAI’s Hardware-Assisted Locking mechanism which allow much more granularity in the way SCSI lockings happen. Basically the SCSI locking is off loaded to the array and doesn’t lock the entire LUN anymore but only the blocks that require it. Please read more about VAAI at VMware.

VMFS design, choice of a Block Size, role of sub blocks,  File Size maximums, VMDK placement and IOPS are always hot topics. Here I narrowed down to mainly Block Size and File Size features. I’m going to post several articles on these topics I hope, so stay tuned!

Sources: Block size limitations of a VMFS datastore, VMFS block size, Reformatting the local VMFS partition’s block size in ESX 4.0, Block sizes and growing your VMFS