Online Training - 14

In this article I want to set the record straight regarding last week’s article about New in esxtop RESV/s (SCSI reservations per second). During VMware Communities Roundtable Podcast number 118 with Krishna Raj Raja. I have learned that I didn’t measure the right metric with the correct method so now I’ve used the right method and metric -> CONS/s. I hope you will enjoy it. You can also watch the HD version of this video on <YouTube> and <ScreenCast>.

I’ve received an email from Krishna Raj Raja with some additional details about SCSI RESV/s and CONS/s counters.

- A nonzero value for RESV/s counter indicates that the host is doing some metadata operation on the VMFS volume (like creating a file, allocating disk space, file rename, permission change, timestamp update etc) - this by itself does not cause any performance problem.

- SCSI reservation conflict counter - CONS/s will become non-zero  when a host tries to do any I/O on a LUN which has a SCSI reservation in progress.

- SCSI reservation is held for a very short period (few hundred microseconds) so the chances of a getting a conflict is very less on a small cluster. However as the number of hosts that shares the LUN increases conflicts could arise more frequently.

- SCSI reserve/release commands have higher priority and they are placed ahead in the host's SCSI I/O queue for immediate dispatch.

- When SCSI reservation is held on a LUN, I/O issued by a VM going to that LUN will fail with the BUSY status within the guest, and the guest will have to retry the I/O. This could result in an end to end increase in I/O latency for that particular I/O.

- The new vSphere 4.1 feature - VAAI - has support for ATS (Atomic test and set) functionality which is a replacement for SCSI reservation and it provides much faster/granular locking mechanism. Performance impact is very little if ATS is used instead of SCSI reserve/release mechanism.

At the VMworld 2010 Krishna Raj Raja and Haiping Yang have delivered a great session called Troubleshooting using ESXTOP for Advanced Users (TA6720). This session contains the new esxtop features which can be found in ESX 4.1. One of the new metrics is RESV/s (SCSI reservations per second). Duncan Epping has also written an article about it called Did you know? SCSI Reservations

VMFS is a clustered file system and uses SCSI reservations as part of its distributed locking algorithms. Administrative operations, such as creating or deleting a virtual disk, extending a VMFS volume, or creating or deleting snapshots, result in metadata updates to the file system using locks, and thus result in SCSI reservations.

Reservations are also generated when you expand a virtual disk for a virtual machine with a snapshot. A reservation causes the LUN to be available exclusively to a single ESX host for a brief period of time. Although it is acceptable practice to perform a limited number of administrative tasks during peak hours, it is preferable to postpone major maintenance or configuration tasks to off‐peak hours in order to minimize the impact on virtual machine performance.

In this video I’ll show you the new SCSI Reservation stats and the impact on the storage performance when the number if SCSI reservations becomes too high.

The information in this video is not completely correct, you can listen for an explanation in the VMware Communities Roundtable Podcast number 118 with Krishna Raj Raja.

In my previous post "Running the VMware vSphere Hypervisor stateless" I’ve explained how to setup an ESXi deployment server, but now we are going to take it one step further. To be able to save the ESXi configuration settings there’s a CCBoot feature available which is called “Write Back”. In this episode (HD) I’m going to show you how to enable “Write Back” for both the ESXi host and the CCBoot server. After rebooting the ESXI host the configuration will be preserved.

This video <- HD version - will show you how to create a stateless VMware vSphere Hypervisor formally known as ESXi and how to boot it trough PXE-iSCSI. Yesterday evening I finally succeeded in booting an ESXi 4.1 host with PXE and iSCSI. After firing the tweet, I’ve received a lot of responses asking me how I did it. In the video I’ll show you how to build your own staging server and the deployment of an ESXi 4.1 host. I’m not really a Linux guy so I had to create a Windows distribution server. In my search I’ve discovered a great little piece of software called CCBoot. This windows application enables a diskless boot of an OS with PXE and iSCSI. Diskless boot makes it possible for ESXi server to be operated without a local disk. The 'diskless' server is connected to a VMDK file over the network and boots up the hypervisor from the remotely located VMDK file which was created in VMware WorkStation. CCBoot is the convergence of the rapidly emerging iSCSI protocol with gPXE diskless boot technology. Remote boot over iSCSI, or CCBoot, pushes the iSCSI technology even further, opening the door to the exciting possibility of the diskless computer.

Here are the steps to set it up

The first thing to do is create a VMDK file with ESXi 4,1 in it, I’ve used VMware Workstation 7.1.
When ESXi 4.1 is installed we’re going to set-up the distribution server, go to http://ccboot.com and download CCBoot v1.9 Build 20100128 (5 users free version). When you’ve installed it, create an entry for you ESXi 4.1 host and you’re ready to run.

vSphere 4.1 introduces a load-based teaming (LBT) policy that ensures vDS dvUplink capacity is optimized. LBT avoids the situation of other teaming policies in which some of the dvUplinks in a DV Port Group’s team were idle while others were completely saturated just because the teaming policy used is statically determined. LBT reshuffles port binding dynamically based on load and dvUplinks usage to make an efficient use of the bandwidth available. LBT only moves ports to dvUplinks configured for the corresponding DV Port Group’s team. Note that LBT does not use shares or limits to make its judgment while rebinding ports from one dvUplink to another. LBT is not the default teaming policy in a DV Port Group so it is up to the user to configure it as the active policy.
LBT will only move a flow when the mean send or receive utilization on an uplink exceeds 75 percent of capacity over a 30-second period. LBT will not move flows more often than every 30 seconds. This video will show Load-Based Teaming (LBT) in action.

In this video I’ll show you how you use the VI Client user interface to deploy the CapacityIQ appliance. VMware distributes the appliance as a .zip file that includes an Open Virtualization Format (OVF) file. After the import the CapacityIQ OVF file, I’ll also show you how to configure the virtual appliance.

This is a maintenance release that adds support for vSphere 4.1. Only customers using vSphere 4.1 are encouraged to upgrade to this release; there are no additional enhancements in this version.

This release does not support virtual machine-based licensing for CapacityIQ. CapacityIQ will provide this type of licensing support in an upcoming release.

CapacityIQ 1.0.4 supports 200 hosts, 2000 powered on virtual machines, and a total of 3000 virtual machines. These scalability limits exist even though the limits of vCenter Server 4.1 are 1000 hosts, 10000 powered on virtual machines, and 15000 registered virtual machines.