ESXi 4.1 – Major Security Issue – The Sequel And The Workaround

Twitter is good! Twitter is great! Today, late in the evening my time, I started to read tweets about a major security issue within ESX4.1.0. It all started with a post from nirvy in a VMware forum followed by an article and a tweet from lamw that I encourage you to read at virtuallyGhetto.com before you keep on reading my post.
I have tested the issue in my home lab and I can confirm the problem with ESXi4.1.0 fresh install or upgrade! I could  not resist to try to find out the root cause of the problem or at least a fix/workaround.  After a couple of hours here are what I have found:

• ESX4.1 doesn’t use MD5 but DES – cat /etc/shadow -> hash for root’s password doesn’t start with ‘$1$‘ and hash is very short.
• /etc/pam.d/common-password doesn’t exist but VMware KB1012033 tells you to edit that file if you want to change password restriction settings.
• sshd uses /etc/pam.d/system-auth file but that file has changed a lot since ESX4.0.

I leave to VMware to create a fix to address the issue. In the mean time here is a workaround to force MD5 encryption of root’s password but first the reminder!
This is an UNSUPPORTED hack that can lead to UNSTABLE system! Don’t try this on a production environment (don’t even think about it) and read the disclaimer below! I shall not be liable for any damages arising out! Now that I have scared you let’s move on with the fix.

1. vi /etc/pam.d/system-auth and change this line accordingly: password sufficient /lib/security/$ISA/pam_unix.so use_authtok nullok md5 shadow
2. save and exit the file with :wq!
3. change your password in DCUI,
4. verify that it has encrypted your root’s password using MD5 algorithm: cat /etc/shadow <- If root’s password hash starts with a ‘$1$‘ then MD5 algorithm was used.

[UPDATE1] Instead of using CHMOD, it is much easier and safer to  issue a :wq! to save and exit the file. Mike Horwath notified me my error. I spent too much years on Windows doh! Thx Mike!
Unfortunately the change doesn’t survive a reboot of the host and /etc/pam.d/system-auth file gets reverted back to default file. That means if you change again the root password from DCUI for instance, it will be again encrypted with a DES algorithm.
My feeling this issue is related somehow to Likewise Identity Service licensed by VMware in ESX(i)4.1.0 but that’s just a thought!
Now do we really have a major security problem here, something like a Microsoft zero-day virus or attack? It is likely that you’re not at risk but anyway VMware should come back with a fix to force MD5 by default and let me tell you why. At the moment the encryption uses Data Encryption Standard (DES) which was cracked in the late 1990’s. The problem with DES is that it is a 56bit algorithm, and thus can produce a maximum of 2^56 unique hashes. It is indeed a weak algorithm in terms of strength. The mitigating factor is that you need to have access to /etc/shadow to be able to crack the hashes.

[UPDATE2] I’ve mixed up two different issues! DES and LM HASH.  DES definitely truncates after 8 characters, there is no ‘second hash’. This is by design. On the other hand LM HASH does use a second hash for anything beyond 7 characters. I should have double checked instead relying solely on my memory… Thx Bert, I owe you one 🙂

[UPDATE3] VMware just published KB1024500 describing the behavior but so far no patch foreseen to encrypt passwords using MD5 instead of DES.

[UPDATE4] VMware just updated its KB1024500 with a solution to the issue. The steps are identical to those I detailed above. VMware goes further with an additional step for ESXi only to allow the change to persist through reboots. Now we are waiting for the patch… Great work VMware!

[UPDATE5] VMware updated again its KB! Now VMware is expecting to release a patch to this issue. Stay tuned!

[UPDATE6] November 15th 2010, VMware just released a patch for bug fixes and especially for the password truncated at eight characters, read more at VMware ESXi 4.1, Patch Release ESXi410-201010001

[UPDATE7] VMware just published a KB titled Changing the default password hashing algorithm from md5 to sha512 in ESX 4.x. The KB basically describes the same process, that is editing /etc/pam.d/system-auth to append a SHA512 instead of MD5. That’s an overall improvement of the security. Thx VMware.

DISCLAIMER. THIS INFORMATION IS PROVIDED TO YOU “AS IS” WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, WHETHER ORAL OR WRITTEN, EXPRESS OR IMPLIED. THE AUTHOR SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES OR CONDITIONS OF MERCHANTABILITY, SATISFACTORY QUALITY, NON-INFRINGEMENT AND FITNESS FOR A PARTICULAR PURPOSE AND SHALL NOT BE LIABLE FOR ANY DAMAGES ARISING OUT OF OR IN CONNECTION WITH THE USE OF THIS CONTENT, INCLUDING DIRECT, INDIRECT, CONSEQUENTIAL DAMAGES, LOSS OF BUSINESS PROFITS OR SPECIAL DAMAGES, EVEN IF THE AUTHOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.