Aloha ALUA


This is the second part of my deep dive into a bunch of storage acronyms, concepts and terminology we all have read at some points. Perhaps you’d prefer start by reading part one before going on with part two although this is not mandatory at all…

Note that ALUA is huge matter on its own. I’ve tried here to summarize what I think would be the best parts for my own understanding and for my day-to-day job as an vAdmin but also because I’m eager to learn more about this huge topic. This post contains a lot of references to different storage vendors and VMware documentation, white papers, blogs and reference guides. For sources, go to the post’s footnote.

What is ALUA?

ALUA stands for Asymmetric Logical Unit Access. It is part of the SPC-3 standard (2005) from the INCITS (InterNational Committee for Information Technology Standards).

Why you need ALUA?

The array’s active-active nature allows I/O requests to be serviced by either one of the controllers to a LUN. However, the asymmetric access nature of the array forces the optimal path to a LUN through one of the controllers. As opposed to Symmetric access nature where multiple controllers can own a LUN.

An optimal path is defined as an I/O path to the LUN with minimum processing overhead. This capability means that the controller with the optimal path to the LUN, also known as the managing controller (HP wording) or the storage processor (EMC wording), can issue I/O directly to the LUN. Whereas the non-managing controller, also known as the proxy controller, can receive I/O requests for a LUN it does not manage but has to proxy the request to the managing controller for processing to the LUN.

For example with a HP EVA A/A Asymmetric storage device, when a read I/O request is sent to the non-managing controller for a LUN, this controller has to transfer this request (proxy) to the managing controller for the LUN by copying the I/O request to the managing controller’s cache. The managing controller issues the I/O request to the LUN and caches the response data. The managing controller then transfers this response data to the non-managing controller’s cache so that the response data can be returned to the host through the controller/host ports to which the host initially sent the request.

This is why a proxy read (read through the non-managing controller) has additional processing overhead. Important to note that write requests are not affected by proxy processing overhead because write requests are automatically mirrored to both controllers’ caches for enhanced fault tolerance. Thus the managing controller always has a copy of the write request in its local cache and can process the request without the need of a proxy from the non-managing controller. I would add a note to this important note, some storage devices allow you to disable the cache mirroring feature leaving more memory for the controller’s own cache. In that case both reads and writes need to be proxied thus the overhead is even bigger. Diagram above from Chad Sakac.

Other ALUA key features

There is more with ALUA as it is defined in the SPC3 standard, that is the ability to identify and alter the LUN controller ownership. There are implicit and explicit ALUA modes.

  • Implicit transition mode means that the array can assign and change the managing controller for the LUN.
  • Explicit transition mode means that a host driver can set or change the LUN controller ownership.

VMware vSphere 4 is also ALUA-compliant and allows the hypervisor to detect that a storage system is ALUA-capable and to utilize the ALUA compliance to optimize I/O processing to the controllers and detect LUN failover between controllers. This is built into the PSP.

vSphere supports all four ALUA modes:

  1. Not supported
  2. Implicit
  3. Explicit
  4. Both implicit and explicit support

Additionally, vSphere 4 also supports all ALUA access types:

  1. Active-optimized – The path to the LUN is through the managing controller.
  2. Active-non-optimized – The path to the LUN is through the non-managing controller.
  3. Standby – The path to the LUN is not an active path and must be activated before I/O can be issued.
  4. Unavailable – The path to the LUN is unavailable through this controller.
  5. Transitioning – The LUN is transitioning from and to any one of the types defined above

Note that Round Robin (RR) and MRU path policies are both ALUA-aware (more about RR and MRU in part one), meaning that both load balancing policies will first attempt to schedule I/O requests to a LUN through a path that is through the managing controller, that is active-optimized.

A word about TPGS

Now to detect these different ALUA mode and access types there is the Target Port Group Support. TPGS is a method to enable a new storage device to be automatically detected and to allow methodologies to understand different port characteristics and failover behaviors. TPGS also provides a method for determining the access characteristics of a path to a logical unit through a target port.

The access characteristics of a path are defined by the target port’s asymmetric access state (AAS), which is returned by the SCSI command REPORT TARGET PORT GROUPS (RTPG). Access states are set through the command SET TARGET PORT GROUPS (STPG) or by the target device.

To a better integration with vSphere

LUN follow-over goes hand in hand with ALUA. As discussed above, ALUA defines which controller in an asymmetric active-active array is the managing controller for a LUN. LUN follow-over, on the other end, ensures that when the optimal path to the LUN changes that all hosts accessing the LUN will also change their access path to the LUN to reflect this change. LUN follow-over in any vSphere 4 cluster is critical, as it will ensure that LUN path thrashing between the two array controllers never occurs and, as importantly, it will significantly reduce administrator configuration time. This is because all ESX servers accessing the LUN in question will update the LUN optimal access path to follow when the LUN is implicitly moved from one controller to the other.

ALUA is not always the best option

ALUA is definitely a good thing but there are designs where where it must be disabled. I have two examples using HP storage devices:

  1. VMware SRM and HP Continuous Access EVA
  2. Microsoft GeoCluster and HP StorageWorks Cluster Extension (CLX) which uses HP Continuous Access EVA as its foundation for replicating DR groups (LUNs) to another HP storage devices

The implicit LUN transition is automatically disabled for all members of an HP Continuous Access EVA DR group because Continuous Access requires that all members of a DR group be managed by the same controller, it would be necessary to move all members of the DR group if excessive proxy reads were detected on any member. This would impact performance and create a proxy read situation for the other virtual disks in the DR group which in turn may trigger another transition eventually leading to a path trashing scenario.

My ALUA compatible device is not showing up in vSphere

You’ve just purchased a new storage device, a HP P2000 for example, and when you run the esxcli nmp satp listrules | grep HP as shown on the screenshot below, you notice that vSphere doesn’t have a SATP plug-in for it:

If you have already attached the device to your vSphere you may have seen only one HBA with devices. Another HBA attached to the same storage may only show a “dead” path. No worries, let’s add a claiming rule to be used with the VMware default SATP plug-in to the list with the following command:

esxcli nmp satp addrule –satp=”VMW_SATP_ALUA” –vendor=”HP” –model=”P2000″ –description=”HP P2000 ALUA” –claim-option tpgs_on

Was it a successful added? Let’s find out:

Alternatively you can add a claiming rule to be more generic and to be used with all HP P2* models for instance:

esxcli nmp satp addrule –satp=”VMW_SATP_ALUA” –vendor=”HP” –model=”^P2*” –description=”HP P2000 Series ALUA” –claim-option tpgs_on

Agaion let’s check to see if it has been added successfully:

This will hold through a reboot, and a reboot can be used to put the rule into action. If you cannot reboot, then you can use the following command to trigger a reclaim action:

esxcli corestorage claiming reclaim -d

where <NAA_ID> is replaced with the NAA of the volume you are working against. For example: naa.600601604450650012ea2d38053cdf31
Once the reclaim is complete, rescan the storage again and the new pathing to the LUNs should be available.

It would not have been feasible to do this blog post without these widely available documents, white papers, VMware KB’s and blog posts. I have gone through them all to prepare these two articles. I have in many occasions copy/paste few lines and adding my thoughts or notes though. I’m listing them all here and would like to thank the authors for publishing such good and useful information to the wide.

Sources: Configuration best practices for HP StorageWorks Enterprise Virtual Array (EVA) family and VMware vSphere 4Configuring HP StorageWorks D2D & VLS virtual tape libraries on ESX 4.x (Partner Support)Fibre Channel SAN Configuration GuideWhat Is New in VMware vSphere™ 4:StorageWhat is Pluggable Storage Architecture (PSA) and Native Multipathing (NMP)?Vmware vSphere 4.1 PSA (Pluggable Storage Architecture) UnderstandingChange the default pathing policy to round robinWhat’s the point of setting “–IOPS=1″ ?What’s that ALUA exactly?Pluggable Storage Architecture, exploring the next version of ESX/vCenterESX4 ALUA, TPGS AND HP CAUnderstanding more about NMP RR and iooperationslimit=1See a list of Storage Array Type Plugins on ESX ServerWhy do I have trespassed LUNs or all LUNs on one SP?SCSI Primary Commands – 3 (SPC-3), EMC CLARiiON Asymmetric Active/Active Feature

About PiroNet

Didier Pironet is an independent blogger and freelancer with +15 years of IT industry experience. Didier is also a former VMware inc. employee where he specialised in Datacenter and Cloud Infrastructure products as well as Infrastructure, Operations and IT Business Management products. Didier is passionate about technologies and he is found to be a creative and a visionary thinker, expressing with passion and excitement, hopefully inspiring and enrolling people to innovation and change.
This entry was posted in Uncategorized. Bookmark the permalink.

9 Responses to Aloha ALUA

  1. NiTRo says:

    Very nice post Didier, you must have dig a lot for all of that, thanks !

  2. Pingback: VMware PSA, MPP, NMP, PSP, MRU, … And Tutti Quanti! | DeinosCloud

  3. Pingback: Hu Yoshida » Blog Archive » VMware Alternate Pathing: The Importance of Being Active/Active

  4. Pingback: It All Started With This Question… | DeinosCloud

  5. Sascha says:

    Hi,

    there is a mistake in

    “The implicit LUN transition is automatically disabled for all members of an HP CA DR…”

    On HP EVA CA the explicit is automatically disabled because the host do not know which controller will handle the DR and can not set the controller ownership right because of the
    dependency in CA.

    !! ON HP EVA CA the array controller must work implicit and will try to ignore STPG by design !!

    Also it’s very important to know that a shared LUNs will not work in explicit ALUA because
    there is no distributed log between the hosts that they will send out the STPG request for the same controller at the same time. This will result in LUN trashes.

    Next is that I was not able to bring the ESX servers to a point where they identify an from HP EVA presented LUN as just implicit. Also if I use the latest XCS and tack the LUN before presenting. The ESX alway tells that the LUN is also explicit. This works until the ESX servers do not see a reason to send out STPG to the EVA. At the moment the servers see a reason (controller / zoning failure & CA will brake …) the trashes of the shared LUNs will start. This will also applies to a non CA EVA with no DR and get much more for example if you reach the storage fan out.

    In SPC-3 the implicit / explicit will never be disabled on storage site if the storage knowns both. The time where the host contacts the storage it ask for the storage characteristics. If he get within the ALUA Flags just a 0x10 he should not use explicit and if he gets (+)0x20 the host can use explicit. This works on other systems with shared LUNs but for me it looks like this flags will be ignored in the ESX ALUA implementation (= not SPC-3 conform).

    As far as I know at the moment this applies to all SPC-3 asymmetric storages that knows explicit ALUA from any vendor as long as the ESX ALUA will not be fixed and there will be no way to force the ESX to implicit on shared LUNs which is basic there. All best practice like “1 IO per path change” are just bad workarounds to keep the fan out fare away and stay near by fan in. If you do not reach it you do not have an issue with this at a much higher costs.

    thanks.

  6. PiroNet says:

    A very interesting Vmware blog: Configuration Settings for ALUA Devices available at http://blogs.vmware.com/vsphere/2012/02/configuration-settings-for-alua-devices.html

  7. Pingback: Why ALUA is a very cool acronym | Aussie Storage Blog

  8. Pingback: Why ALUA is a very cool acronym | ServerGround.net

  9. Pingback: Storage Administrator Primer | Matt Davis

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s