Aloha ALUA

This is the second part of my deep dive into a bunch of storage acronyms, concepts and terminology we all have read at some points. Perhaps you’d prefer start by reading part one before going on with part two although this is not mandatory at all…

Note that ALUA is huge matter on its own. I’ve tried here to summarize what I think would be the best parts for my own understanding and for my day-to-day job as an vAdmin but also because I’m eager to learn more about this huge topic. This post contains a lot of references to different storage vendors and VMware documentation, white papers, blogs and reference guides. For sources, go to the post’s footnote.

What is ALUA?

ALUA stands for Asymmetric Logical Unit Access. It is part of the SPC-3 standard (2005) from the INCITS (InterNational Committee for Information Technology Standards).

Why you need ALUA?

The array’s active-active nature allows I/O requests to be serviced by either one of the controllers to a LUN. However, the asymmetric access nature of the array forces the optimal path to a LUN through one of the controllers. As opposed to Symmetric access nature where multiple controllers can own a LUN.

An optimal path is defined as an I/O path to the LUN with minimum processing overhead. This capability means that the controller with the optimal path to the LUN, also known as the managing controller (HP wording) or the storage processor (EMC wording), can issue I/O directly to the LUN. Whereas the non-managing controller, also known as the proxy controller, can receive I/O requests for a LUN it does not manage but has to proxy the request to the managing controller for processing to the LUN.

For example with a HP EVA A/A Asymmetric storage device, when a read I/O request is sent to the non-managing controller for a LUN, this controller has to transfer this request (proxy) to the managing controller for the LUN by copying the I/O request to the managing controller’s cache. The managing controller issues the I/O request to the LUN and caches the response data. The managing controller then transfers this response data to the non-managing controller’s cache so that the response data can be returned to the host through the controller/host ports to which the host initially sent the request.

This is why a proxy read (read through the non-managing controller) has additional processing overhead. Important to note that write requests are not affected by proxy processing overhead because write requests are automatically mirrored to both controllers’ caches for enhanced fault tolerance. Thus the managing controller always has a copy of the write request in its local cache and can process the request without the need of a proxy from the non-managing controller. I would add a note to this important note, some storage devices allow you to disable the cache mirroring feature leaving more memory for the controller’s own cache. In that case both reads and writes need to be proxied thus the overhead is even bigger. Diagram above from Chad Sakac.

Other ALUA key features

There is more with ALUA as it is defined in the SPC3 standard, that is the ability to identify and alter the LUN controller ownership. There are implicit and explicit ALUA modes.

• Implicit transition mode means that the array can assign and change the managing controller for the LUN.
• Explicit transition mode means that a host driver can set or change the LUN controller ownership.

VMware vSphere 4 is also ALUA-compliant and allows the hypervisor to detect that a storage system is ALUA-capable and to utilize the ALUA compliance to optimize I/O processing to the controllers and detect LUN failover between controllers. This is built into the PSP.

vSphere supports all four ALUA modes:

1. Not supported
2. Implicit
3. Explicit
4. Both implicit and explicit support

Additionally, vSphere 4 also supports all ALUA access types:

1. Active-optimized – The path to the LUN is through the managing controller.
2. Active-non-optimized – The path to the LUN is through the non-managing controller.
3. Standby – The path to the LUN is not an active path and must be activated before I/O can be issued.
4. Unavailable – The path to the LUN is unavailable through this controller.
5. Transitioning – The LUN is transitioning from and to any one of the types defined above

Note that Round Robin (RR) and MRU path policies are both ALUA-aware (more about RR and MRU in part one), meaning that both load balancing policies will first attempt to schedule I/O requests to a LUN through a path that is through the managing controller, that is active-optimized.

A word about TPGS

Now to detect these different ALUA mode and access types there is the Target Port Group Support. TPGS is a method to enable a new storage device to be automatically detected and to allow methodologies to understand different port characteristics and failover behaviors. TPGS also provides a method for determining the access characteristics of a path to a logical unit through a target port.

The access characteristics of a path are defined by the target port’s asymmetric access state (AAS), which is returned by the SCSI command REPORT TARGET PORT GROUPS (RTPG). Access states are set through the command SET TARGET PORT GROUPS (STPG) or by the target device.

To a better integration with vSphere

LUN follow-over goes hand in hand with ALUA. As discussed above, ALUA defines which controller in an asymmetric active-active array is the managing controller for a LUN. LUN follow-over, on the other end, ensures that when the optimal path to the LUN changes that all hosts accessing the LUN will also change their access path to the LUN to reflect this change. LUN follow-over in any vSphere 4 cluster is critical, as it will ensure that LUN path thrashing between the two array controllers never occurs and, as importantly, it will significantly reduce administrator configuration time. This is because all ESX servers accessing the LUN in question will update the LUN optimal access path to follow when the LUN is implicitly moved from one controller to the other.

ALUA is not always the best option

ALUA is definitely a good thing but there are designs where where it must be disabled. I have two examples using HP storage devices:

1. VMware SRM and HP Continuous Access EVA
2. Microsoft GeoCluster and HP StorageWorks Cluster Extension (CLX) which uses HP Continuous Access EVA as its foundation for replicating DR groups (LUNs) to another HP storage devices

The implicit LUN transition is automatically disabled for all members of an HP Continuous Access EVA DR group because Continuous Access requires that all members of a DR group be managed by the same controller, it would be necessary to move all members of the DR group if excessive proxy reads were detected on any member. This would impact performance and create a proxy read situation for the other virtual disks in the DR group which in turn may trigger another transition eventually leading to a path trashing scenario.

My ALUA compatible device is not showing up in vSphere

You’ve just purchased a new storage device, a HP P2000 for example, and when you run the esxcli nmp satp listrules | grep HP as shown on the screenshot below, you notice that vSphere doesn’t have a SATP plug-in for it:

If you have already attached the device to your vSphere you may have seen only one HBA with devices. Another HBA attached to the same storage may only show a “dead” path. No worries, let’s add a claiming rule to be used with the VMware default SATP plug-in to the list with the following command:

esxcli nmp satp addrule –satp=”VMW_SATP_ALUA” –vendor=”HP” –model=”P2000″ –description=”HP P2000 ALUA” –claim-option tpgs_on

Was it a successful added? Let’s find out:

Alternatively you can add a claiming rule to be more generic and to be used with all HP P2* models for instance:

esxcli nmp satp addrule –satp=”VMW_SATP_ALUA” –vendor=”HP” –model=”^P2*” –description=”HP P2000 Series ALUA” –claim-option tpgs_on

Agaion let’s check to see if it has been added successfully:

This will hold through a reboot, and a reboot can be used to put the rule into action. If you cannot reboot, then you can use the following command to trigger a reclaim action:

esxcli corestorage claiming reclaim -d

where <NAA_ID> is replaced with the NAA of the volume you are working against. For example: naa.600601604450650012ea2d38053cdf31
Once the reclaim is complete, rescan the storage again and the new pathing to the LUNs should be available.

It would not have been feasible to do this blog post without these widely available documents, white papers, VMware KB’s and blog posts. I have gone through them all to prepare these two articles. I have in many occasions copy/paste few lines and adding my thoughts or notes though. I’m listing them all here and would like to thank the authors for publishing such good and useful information to the wide.

Sources: Configuration best practices for HP StorageWorks Enterprise Virtual Array (EVA) family and VMware vSphere 4, Configuring HP StorageWorks D2D & VLS virtual tape libraries on ESX 4.x (Partner Support), Fibre Channel SAN Configuration Guide, What Is New in VMware vSphere™ 4:Storage, What is Pluggable Storage Architecture (PSA) and Native Multipathing (NMP)?, Vmware vSphere 4.1 PSA (Pluggable Storage Architecture) Understanding, Change the default pathing policy to round robin, What’s the point of setting “–IOPS=1″ ?, What’s that ALUA exactly?, Pluggable Storage Architecture, exploring the next version of ESX/vCenter, ESX4 ALUA, TPGS AND HP CA, Understanding more about NMP RR and iooperationslimit=1, See a list of Storage Array Type Plugins on ESX Server, Why do I have trespassed LUNs or all LUNs on one SP?, SCSI Primary Commands – 3 (SPC-3), EMC CLARiiON Asymmetric Active/Active Feature