Virtualization.com

News and insights from the vibrant world of virtualization and cloud computing

Infrastructure 3.5u2

Update On ESX 3.5 Issue: A Letter from Paul Maritz

by Leave a Comment

—

New VMware CEO Paul Maritz has issued an official statement about the major bug plaguing customers who had updated to ESX / ESXi 3.5 Update 2 and experienced a serious problem yesterday due to a mistake in the licensing code.

The letter in full:

“Last night, we became aware of a code issue with the recently released update to ESX 3.5 and ESXi 3.5 (Update 2).

When the time clock in a server running ESX 3.5 or ESXi 3.5 Update 2 hits 12:00AM on August 12th, 2008, the released code causes the product license to expire.  The problem has also occurred with a recent patch to ESX 3.5 or ESXi 3.5 Update 2.  When an ESX or ESXi 3.5 server thinks its license has expired, the following can happen:

  • Virtual machines that are powered off cannot be turned on;
  • Virtual machines that have been suspended fail to leave suspend mode; and,
  • Virtual machines cannot be migrated using VMotion.

The issue was caused by a piece of code that was mistakenly left enabled for the final release of Update 2.  This piece of code was left over from the pre-release versions of Update 2 and was designed to ensure that customers are running on the supported generally available version of Update 2.

In remedying the situation, we’ve already released an express patch for those customers that have installed/upgraded to ESX or ESXi 3.5 Update 2.  Within the next 24 hours, we also expect to issue a full replacement for Update 2, which should be used by customers who want to perform fresh installs of ESX or ESXi.

I am sure you’re wondering how this could happen.  We failed in two areas:

  • Not disabling the code in the final release of Update 2; and
  • Not catching it in our quality assurance process.

We are doing everything in our power to make sure this doesn’t happen again.  VMware prides itself on the quality and reliability of our products, and this incident has prompted a thorough self-examination of how we create and deliver products to our customers.  We have kicked off a comprehensive, in-depth review of our QA and release processes, and will quickly make the needed changes.

I want to apologize for the disruption and difficulty this issue may have caused to our customers and our partners.  Your confidence in VMware is extremely important to us, and we are committed to restoring that confidence fully and quickly.

Thank You,

Paul Maritz
President and CEO
VMware”

Major Bug Kills VMware Powered Virtual Servers

by 1 Comment

Today’s a black day for VMware, and also (and maybe especially) VMware customers who upgraded their virtual servers with the new Infrastructure 3.5 Update 2. As of this morning, many could not power on VMotion or any of their Virtual Machines. The VI Client threw the error “A general system error occurred: Internal Error”.

This was first reported by a customer in a thread on VMware Communities. You can find the Knowledge Base article on this problem here.

The problem apparently exists within the software licensing code, rather than the functional part of ESX software.The license code falsely identifies many implementations as being out of license, prevents new virtual machines from launching, or existing VMs from migrating to new hosts.

Big oops indeed.

“An issue has been uncovered with ESX 3.5 Update 2 and ESXi 3.5 that causes the product license to expire on August 12,” says a statement from VMware’s public relations company in response to press inquiries. “VMware is alerting customers and partners of this issue. Updated product bits with correct licensing will be made available for download as soon as possible. VMware regrets any inconvenience to customers. VMware is working on an immediate patch for customers in production. VMware expects to fix the issue in code in the next 36 hours once QA testing has been completed,” the statement concludes.

A work-around has been offered whereby customers should manually set the date of all ESX 3.5u2 hosts back to 10 August as a temporary fix. Brian Madden has more on the actual glitch and the workaround, as does Matthew Marlowe.

It looks like ESXi 3.5 and even some versions of Infrastructure 3.5 Update 1 with some patches have been affected. New downloads of all affected products has been disabled, and according to VMware patched products will be made available starting tomorrow.

This will hurt VMware in any event, even if we don’t really know how many customers have already downloaded Update 2, and how many of those were using it in a live environment.

VMware