vSphere

- Tuning ESX/ESXi for better storage performance by modifying the maximum I/O block size
- ESX has performance issues due to IRQ sharing
- Enabling IOAT and Jumbo frames
- Troubleshooting network performance issues
- Analyzing SCSI Reservation conflicts on VMware Infrastructure 3.x and vSphere 4.x
- Determining if multiple virtual CPUs are causing performance issues
- Slow performance caused by misconfigured local storage or SAN array
- Slow performance caused by out of date firmware on a RAID controller or HBA
- Testing virtual machine storage I/O performance
- iSCSI and Jumbo Frames configuration on ESX 3.x, ESX 4.x and ESXi 5.x
- Using esxtop to identify storage performance issues
- Configuring Jumbo Frames on a vNetwork Distributed Switch
- Verifying correct storage settings on ESX 4.x and ESXi 4.x
- Network I/O Resource Management in vSphere 4.1 with vDS
- Impact of virtual machine memory and CPU resource limits
- Checking your firmware and BIOS levels to ensure compatibility with ESX/ESXi
- Enabling Jumbo Frames for VMkernel ports in a virtual distributed switch
- Changing the Queue Depth for QLogic and Emulex HBAs
- High co-stop (%CSTP) values seen during virtual machine snapshot activities
- Balloon driver retains hold on memory causing virtual machine guest operating system performance issues

Source KB 2001003

This paper provides a deep look at the performance implications of the live storage migration of a VM running a large SQL Server database workload. Performance of the workload when migrating different components of the database (data, index, and log) was measured and compared against the steady state performance (when there was no migration). Experiments discussed in this paper also show the impact of I/O characteristics of the database components on the migration time of the virtual disk containing them. Based on the knowledge gained from the experiments, a few best practices are suggested for the live migration of the storage of the virtual machines (VMs) with minimal impact to the performance of applications running in them.

Live storage migration is the missing piece in liberating VMs and their associated files completely from the physical hardware on which they reside. Predictable migration times—with minimal impact on the performance of the application accessing the virtual disk that is migrated—are expected from the vSphere’s storage vMotion feature. This paper offers a deeper look at the interaction of svMotion with a large, active SQL database workload. The study includes application behavior when migrating individual virtual disks used by the database and the impact application I/O traffic had on the svMotion of a particular virtual disk. The study showed consistent and predictable disk migration time that largely depended on the capabilities of the source and the destination arrays. svMotion increased the CPU consumption of the VM running the test workload from 5% to 22% depending on the load conditions. The I/O patterns of the SQL database workload had noticeable impact on svMotion throughput (and the disk migration time).

 Storage vMotion of a Virtualized SQL Server Database

Ernst Cozijnsen over at Capgemini has found a way to install ESXi 4.1 unattended to SD-CARD / USB. Here are his findings: For a few weeks it was bugging me heavily that the ESXi 4.1 installer didn’t want to install to the internal SD-Card of our newly bought systems. Together with Max Daneri (creator of the PXE Manager) http://labs.vmware.com/flings/pxe-manager we searched for a possible cause because the debug installer was telling us “well I can’t install because somebody said I’m not allowed”.

With this in the back of my mind the urge to solve it grew by the minute and it almost became an obsession to find a working solution. A few days ago we cracked it! And it works like a charm.

In a nutshell:

- USB targets are not supported by default (we fixed this by disabling the check for it)
- The minimal target size should be 5.1GB (we scaled it down to 3.7GB so SD-Cards are supported)
- The SCRATH partition of 4GB (also scaled that one down to 2 GB)
- VMFS grow partition (VMFS of USB?? Neah….. isn’t going to fly so we removed it completely)

Replace you ienviron.vgz with this file (http://www.screencast.com/t/pusv1KGq3) and happy installing. If your to curious and want to know the exact modification: Unpack the file and search/grep on “ernst” in /usr/lib/vmware/weasel/* and /usr/lib/vmware/weasel/scripted/

If  you are using the “PXE manager”, modify the “kickstart” part so that you don’t use Local or Remote but instead use device driver.  As device driver you enter usb-storage. Oh…. just one more thing:  VMware support urged me to say that this hack is not supported by VMware! So this is a non-supported but very welcome hack.

Last week I’ve been interviewed by Edmond Varwijk, he works as a journalist for the well know Dutch ICT-News website - Webwereld. Edmond has asked me what the impact of the vSphere 5 ESXi only hypervisor will be on customers who are still using ESX.

Eric Sloof, in het dagelijks leven VMware-consultant en bekend van de NTPRO.NL-weblog, begrijpt keuze van VMware. "VMware wil een mean and lean hypervisor en die hebben ze met ESXi in handen." Volgens Eric Sloof is de impact van het verlies van de Service Console ook voor beheerders die nog een ESX-omgeving managen, bescheiden.

Uit ervaring weet hij dat het meeste in het dagelijks beheer nu al vanuit de vCenter-managementomgeving gebeurt. Bovendien heeft VMware de VMware Management Assistant (vMA) ontwikkeld. "Daarin is een vSphere Command Line (vCLI) beschikbaar die je in staat stelt om door middel van vicfg-commando's specifieke taken uit te voeren."

The article is in Dutch and you can get it here.

 http://webwereld.nl/nieuws/107172/gedwongen-afscheid-nemen-van-esx.html

VMware vShield is a suite of security virtual appliances built for VMware vSphere 4.1. It is a critical security component for protecting virtualized datacenters from attacks and misuse. vShield App and vShield Edge are the two products in the suite that address network security. The goal of this document is to provide details on the key security technologies implemented in the vShield App and vShield Edge products that enable administrators to build a multitenant virtualized datacenter environment that is flexible, agile, scalable and secure. The document first discusses the challenges in using physical security to protect virtual infrastructure and then describes in detail the key new technologies in vShield products that address those challenges.

 The Technology Foundations of VMware vShield

VMware vShield Edge, part of the VMware vShield family of virtualization security products, provides perimeter security and network services such as DHCP, NAT, Load balancing, and VPN service. vShield Edge is a virtual firewall appliance that can be provisioned on-demand and its services enabled on the fly to meet the flexibility requirement of cloud deployments. The goal of this document is to help customers understand where and how a vShield Edge firewall can be deployed to secure and isolate tenants/organizations, while providing some reference designs along the way. This document will also help VI administrators and network administrators understand the deployment of security and other network services in virtual datacenters using a vShield Edge firewall.

 vShield Edge Design Guide

VMware vShield App, part of the VMware vShield family of virtualization security products, protects applications in the virtual datacenter from network-based threats. vShield App gives organizations deep visibility into network communications between virtual machines and enables granular policy enforcement with security groups. This document helps VI administrators understand the deployment of security around the virtualized server infrastructure using VMware vShield App product. Two reference designs are provided to help customer understand the security deployment around the virtual infrastructure using vShield App product and advantages.

 vShield App Design Guide

In this demo from TrainSignal's vSphere Performance Monitoring course, David Davis will show you how to monitor CPU metrics in vSphere.

While I enjoyed a holiday and was drinking my piña colada cocktail at a sunny beach in the south of France, my virtualization friend Duncan Epping over at Yellow-Bricks.com has snagged away a real cool scoop regarding the release of a new technical paper about VMware vSphere 4.1 Network performance ;-) As Duncan points out is his article it’s real interesting to know that a single virtual machine actually can saturate the bandwidth of a 10Gbps physical network card.

In fact this newly released performance study will show that with using regular TCP/IP sockets, virtual machines on the same host can communicate at rates of 27Bbps. This is a 10 times increase in throughput from ESX 3.5. Even a single virtual machine with multiple virtual NICs running on vSphere can easily saturate the bandwidth of four 10Gbps NICs for a cumulative throughput of more than 35Gbps.

 http://www.vmware.com/resources/techresources/10181

This paper demonstrates that vSphere 4.1 is capable of meeting the performance demands of today's throughput-intensive networking applications. The paper presents the results of experiments that used standard benchmarks to measure the networking performance of different operating systems in various configurations. These experiments examine the performance of VMs by looking at VMs that are communicating with external hosts and are communicating among each other, demonstrate how varying the number of vCPUs and vNICs per VM influences performance, and show how the scalability results of overcommitting the number of physical cores on a system by adding four 1-vCPU VMs for every core.

Operations Management disciplines are converging in the cloud. Performance, capacity and configuration management are becoming inseparable due to the dynamic nature of converged infrastructure. Traditional tools and processes designed for silo-ed, static physical infrastructures don’t provide the automation and control needed to effectively manage highly virtualized and private cloud environments.  This guide will walk you through some key use cases for VMware vCenter Operations Standard to help you conduct a successful product evaluation. The content includes an overview, installation and configuration, and key use cases of VMware vCenter Operations Standard to demonstrate how to diagnose and correct performance problems in the virtual infrastructure.

http://www.vmware.com/resources/techresources/10178 (via @VMwareStorage)

Stephane Lalonde has been a Senior SE at VMware for 4 years. In previous roles he has worked for Citrix for over 4 years and have done extensive work on their SSL VPN solution. Stephane has also worked at one of Cisco's largest VARs where he implemented various solutions.

Stephane has created a presentation called "Networking From Physical to Virtual", the goal of this presentation is to cover networking in a VMware environment and showing the transition from a fully physical datacenter to a virtualized datacenter from a networking perspective.  The talking points are all in the speakers notes and all slides are screen captures or images to help describe the networking environment.

 

Mind mapping is a powerful technique that allows you to make the most of your brainpower by creating maps that use colour, lines, and images to capture and organize your ideas, information, and tasks.

VMware’s support team has created two new mind maps which will help you solving Management and Network issues. In order to use these minmaps, you have to use a recent version (9) of Adobe Acrobat.

Mindmap – vSphere Troubleshooting Network Issues

Mindmap – vSphere Troubleshooting Management Issues

 

 

When implementing virtualization technology, organizations must ensure that they can continue to maintain a secure environment and meet their compliance obligations. To do so, you will have to evaluate risks that might affect protected information and mitigate those risks through risk-appropriate standards, processes, and best practices.

This set of documents provides guidance on how to securely deploy VMware vSphere 4.1 (“vSphere”) in a production environment. The focus is on initial configuration of the virtualization infrastructure layer, which covers the following:

• The virtualization hosts (both VMware ESX® 4 and VMware ESXi™ 4)
  Configuration of the virtual machine container (NOT hardening of the guest operating system (OS) or any applications running within)
• Configuration of the virtual networking infrastructure, including the management and storage networks as well as the virtual switch (but NOT security of the virtual machine’s network)
• VMware vCenter™ Server, its database and client components
  VMware Update Manager (included because the regular update and patching of the ESX/ESXi hosts and the virtual machine containers are essential to maintaining the security of the environment)
  

 Download vSphere 4.1 Hardening Guide

This evening I had a cool GoToMeeting with Tintri’s David Friedlander and VP of Engineering Pratik Wadher. They showed me their new Tintri VMstore, it’s the first VM aware storage system. The VMstore T440 is an 8.5 TB hardware appliance that is designed from the ground up for hosting virtual machines through 10 GigE NFS.

Tintri uses virtual machine abstractions. VMs and virtual disks in place of conventional abstractions such as volumes, LUNs or files. This allows the VMstore to measure and control I/O performance individually for each virtual disk in the system. In effect, the only layer of abstraction in a Tintri device is the virtual disk itself. Other solutions rely on storage abstractions that create a fundamental mismatch between the storage and VM layers.

I’m very impressed by their appliance, hundreds of VMs in 4U - An intelligent hybrid flash and disk architecture provides enterprise flash performance at disk prices.

http://www.tintri.com/

Using Private VLANs doesn’t consolidate the number of VLANs used or does it? If you want to add security between virtual machines on the same subnet without exhausting VLAN number space, VMware advises to use Private VLANs. Private VLANs are an excellent way to provide layer 2 network isolation between servers in the same subnet. In this video you will learn that using Community Private VLANS doesn’t consolidate the number of VLANs used. On the other hand, when using Isolated Private VLANs you won’t exhaust the number of available VLAN IDs.

VMware’s Nathan Small who works as a Staff Engineer at Global Support Services has put together a great presentation about Advanced Root Cause Analysis. The presentation was designed to give you more insight into how a VMware Technical Support Engineer reviews logs, gathers data and performs in-depth analysis. Nathan is hoping to show you the skills they’re using every day to help determine the root cause for an issue in your environment. With this core knowledge, you will become more self-sufficient within your own environment and be able to diagnose an issue as it occurs rather than after the damage has been done.

Cody Bunch over at ProfessionalVMware has posted another great presentation by Nathan Small - vStorage Performance Troubleshooting