VMWare Infrastructure 3.5

Update On ESX 3.5 Issue: A Letter from Paul Maritz

August 13, 2008 by Robin Wauters Leave a Comment

—

New VMware CEO Paul Maritz has issued an official statement about the major bug plaguing customers who had updated to ESX / ESXi 3.5 Update 2 and experienced a serious problem yesterday due to a mistake in the licensing code.

The letter in full:

“Last night, we became aware of a code issue with the recently released update to ESX 3.5 and ESXi 3.5 (Update 2).

When the time clock in a server running ESX 3.5 or ESXi 3.5 Update 2 hits 12:00AM on August 12th, 2008, the released code causes the product license to expire. The problem has also occurred with a recent patch to ESX 3.5 or ESXi 3.5 Update 2. When an ESX or ESXi 3.5 server thinks its license has expired, the following can happen:

Virtual machines that are powered off cannot be turned on;
Virtual machines that have been suspended fail to leave suspend mode; and,
Virtual machines cannot be migrated using VMotion.

The issue was caused by a piece of code that was mistakenly left enabled for the final release of Update 2. This piece of code was left over from the pre-release versions of Update 2 and was designed to ensure that customers are running on the supported generally available version of Update 2.

In remedying the situation, we’ve already released an express patch for those customers that have installed/upgraded to ESX or ESXi 3.5 Update 2. Within the next 24 hours, we also expect to issue a full replacement for Update 2, which should be used by customers who want to perform fresh installs of ESX or ESXi.

I am sure you’re wondering how this could happen. We failed in two areas:

Not disabling the code in the final release of Update 2; and
Not catching it in our quality assurance process.

We are doing everything in our power to make sure this doesn’t happen again. VMware prides itself on the quality and reliability of our products, and this incident has prompted a thorough self-examination of how we create and deliver products to our customers. We have kicked off a comprehensive, in-depth review of our QA and release processes, and will quickly make the needed changes.

I want to apologize for the disruption and difficulty this issue may have caused to our customers and our partners. Your confidence in VMware is extremely important to us, and we are committed to restoring that confidence fully and quickly.

Thank You,

Paul Maritz
President and CEO
VMware”

Major Bug Kills VMware Powered Virtual Servers

August 12, 2008 by Robin Wauters 1 Comment

Today’s a black day for VMware, and also (and maybe especially) VMware customers who upgraded their virtual servers with the new Infrastructure 3.5 Update 2. As of this morning, many could not power on VMotion or any of their Virtual Machines. The VI Client threw the error “A general system error occurred: Internal Error”.

This was first reported by a customer in a thread on VMware Communities. You can find the Knowledge Base article on this problem here.

The problem apparently exists within the software licensing code, rather than the functional part of ESX software.The license code falsely identifies many implementations as being out of license, prevents new virtual machines from launching, or existing VMs from migrating to new hosts.

Big oops indeed.

“An issue has been uncovered with ESX 3.5 Update 2 and ESXi 3.5 that causes the product license to expire on August 12,” says a statement from VMware’s public relations company in response to press inquiries. “VMware is alerting customers and partners of this issue. Updated product bits with correct licensing will be made available for download as soon as possible. VMware regrets any inconvenience to customers. VMware is working on an immediate patch for customers in production. VMware expects to fix the issue in code in the next 36 hours once QA testing has been completed,” the statement concludes.

A work-around has been offered whereby customers should manually set the date of all ESX 3.5u2 hosts back to 10 August as a temporary fix. Brian Madden has more on the actual glitch and the workaround, as does Matthew Marlowe.

It looks like ESXi 3.5 and even some versions of Infrastructure 3.5 Update 1 with some patches have been affected. New downloads of all affected products has been disabled, and according to VMware patched products will be made available starting tomorrow.

This will hurt VMware in any event, even if we don’t really know how many customers have already downloaded Update 2, and how many of those were using it in a live environment.

VMware

VMware Releases Infrastructure 3.5 Update 2, ESX and VirtualCenter Get Upgrades, ESXi Now 100% Free

July 28, 2008 by Robin Wauters 1 Comment

VMware has released Update 2 for Infrastructure 3.5, listing a whole set of updates and features. At the same time, the company has made the license for ESXi completely free, as announced last week during the earnings call.

These are the highlights of some of the enhancements available in this release of VMware Infrastructure 3 (free evaluation here):

Windows Server 2008 support – Windows Server 2008 (Standard, Enterprise, and Datacenter editions) is supported as a guest operating system. With VMware’s memory overcommit technology and the reliability of ESX, virtual machine density can be maximized with this new guest operating system to achieve the highest degree of ROI. Guest operating system customizations and Microsoft Cluster Server (MSCS) are not supported with Windows Server 2008.
Enhanced VMotion Compatibility – Enhanced VMotion compatibility (EVC) simplifies VMotion compatibility issues across CPU generations by automatically configuring server CPUs with Intel FlexMigration or AMD-V Extended Migration technologies to be compatible with older servers. Once EVC is enabled for a cluster in the VirtualCenter inventory, all hosts in that cluster are configured to ensure CPU compatibility for VMotion. VirtualCenter will not permit the addition of hosts which cannot be automatically configured to be compatible with those already in the EVC cluster.
Storage VMotion – Storage VMotion from a FC/iSCSI datastore to another FC/iSCSI datastore is supported. This support is extended on ESX/ESXi 3.5 Update 1 as well.
VSS quiescing support – When creating quiesced snapshot of Windows Server 2003 guests, both filesystem and application quiescing are supported. With Windows Server 2008 guests, only filesystem quiescing is supported. For more information, see the Virtual Machine Backup Guide and the VMware Consolidated Backup 1.5 Release Notes.
Hot Virtual Extend Support – The ability to extend a virtual disk while virtual machines are running is provided. Hot extend is supported for vmfs flat virtual disks without snapshots opened in persistent mode.
192 vCPUs per host – VMware now supports increasing the maximum number of vCPUs per host 192 given that the maximum number of Virtual Machines per host is 170 and that no more than 3 virtual floppy devices or virtual CDROM devices are configured on the host at any given time. This support is extended on ESX 3.5 Update 1 as well.

As for the hardware enablement and management:

8Gb Fiber Channel HBAs – Support is available for 8Gb fiber channel HBAs. See the I/O Compatibility Guide for ESX Server 3.5 and ESX Server 3i for details.
SAS arrays – more configurations are supported. See the Storage/SAN Compatibility Guide for ESX Server 3.5 and ESX Server 3i for details.
10 GbE iSCSI initiator – iSCSI over a 10GbE interface is supported. This support is extended on ESX Server 3.5 Update 1, ESX Server version 3.5 Update 1 Embedded and ESX Server version 3.5 Update 1 Installable as well.
10 GbE NFS support – NFS over a 10GbE interface is supported.
IBM System x3950 M2 – x3950 M2 in a 4-chassis configuration is supported, complete with hardware management capabilities through multi-node Intelligent Platform Management Interface (IPMI) driver and provider. Systems with up to 32 cores are fully supported. Systems with more than 32 cores are supported experimentally.
PMI OEM extension support – Execution of IPMI OEM extension commands is supported.
System health monitoring through CIM providers – More Common Information Model (CIM) providers are added for enhanced hardware monitoring, including storage management providers provided by QLogic and Emulex. LSI MegaRAID providers are also included and are supported experimentally.
CIM SMASH/Server Management API – The VMware CIM SMASH/Server Management API provides an interface for developers building CIM-compliant applications to monitor and manage the health of systems. CIM SMASH is now a fully supported interface on ESX Server 3.5 and VMware ESX Server 3i.
Display of system health information – More system health information is displayed in VI Client for both ESX Server 3.5 and VMware ESX Server 3i.
Remote CLI – Remote Command Line Interface (CLI) is now supported on ESX Server 3.5 as well as ESX Server 3i. See the Remote Command-Line Interface Installation and Reference Guide for more information.

VirtualCenter 2.5 update 2 adds full support for monitoring individual virtual machine failures based on VMware tools heartbeats. This release also extends support for clusters containing mixed combinations of ESX and ESXi hosts, and minimizes previous configuration dependencies on DNS. It also extends support for alarms on the overall health of the server by considering the health of each of the individual system components such as memory and power supplies. Alarms can now be configured to trigger when host health degrades.

VirtualCenter 2.5 Update 2 provides the ability of creating a clone of a powered-on virtual machine without any downtime to the running virtual machine. Therefore, administrators are no longer required to power off a virtual machine in order to create a clone of it. Also, you can now automatically authenticate to VirtualCenter using your current Windows domain login credentials on the local workstation, as long as the credentials are valid on the VirtualCenter server.

This capability also supports logging in to Windows using Certificates and Smartcards. It can be used with the VI Client or the VI Remote CLI to ensure that scripts written using the VI Toolkits can take advantage of the Windows credentials of your current session to automatically connect to VirtualCenter.

VMware