Posts tagged PVSCSI
VMware pvSCSI – When and when not to use it
Feb 5th
Introduction
Hopefully you have read my previous blog posts on pvSCSI. It describes what the driver is, how it works, and how it can positively impact your performance and workloads. Part two covers the process of installing the pvSCSI driver on an existing system and a new system. Both can be found here on the site and you might find them useful:
http://www.virtualinsanity.com/index.php/2009/11/21/more-bang-for-your-buck-with-pvscsi-part-1/
http://www.virtualinsanity.com/index.php/2009/12/01/more-bang-for-your-buck-with-pvscsi-part-2/
Interrupt Coalescing
VMware recently published a KB article that answers a question that has been floating around the community for a while. The pvSCSI driver sounds superior to the LSI driver with direct I/O access to the hypervisor so why not use it in all cases? The article states that you should only use the newer driver when driving higher workloads, those that are typically 2000 IOPS or greater. For those that don’t know 2000 IOPS is a pretty big workload. Consider this, a standard fiber channel 10,000 RPM drive averages around 125 IOPS per disk.
I didn’t really understand this and the knowledge base article is lacking any detail on the rational behind the statement. I reached out to VMware performance engineer Scott Drummonds to see if he had anything he could publish to help clarify the KB article. Scott was nice enough to research this and posted his findings here.
So it appears that the technical explanation is interrupt coalescing or buffering. The paravirtual SCSI driver was designed to handle receiving multiple requests at a high rate and then “batching” the requests together for better efficiencies in throughput. If you aren’t generating high enough workloads on the virtual machine, the I/O request could unnecessarily sit in the queue while the “batch" waits to be filled up for the next transaction. This could cause storage performance problems which would typically be seen as higher latency and would negatively impact the virtual machine.
Now and Then
The great news is the current release of the driver is optimized for heavy workloads. If you are starting to virtualize SQL/Oracle systems and need the performance, go for the pvSCSI driver and get better throughput. If your deploying standard virtual machines that are doing lower workloads, continue to embrace the existing LSI Logic driver.
If you are new to vSphere 4, or have just upgraded from 3.5 and are starting to rebuild your templates to embrace virtual hardware version 7, don’t use the pvSCSI driver as part of your standard template. VMware is working on the driver and will be introducing advanced coalescing functionality. When this is built into the driver stack pvSCSI will then be able to be utilized for all workloads as it will understand when it needs to ramp up for higher workloads.
Thanks again to Scott Drummonds for taking the time out of his busy schedule to track this one down.
More Bang for your Buck with PVSCSI (Part 2)
Dec 1st
Part 2 Doing the work
As you might have noticed, this blog post is a continuation to my first post about PVSCSI, you can access Part 1 here.
Hopefully now you have a better understanding of what the Paravirtual SCSI driver is all about, and we can prove there are tangible reasons to move in this direction. Let’s get on with the important part, the implementation phase.
(I need to finish off this blog post, I am running out of pictures of SCSI cables)
There are some caveats I need to start out with. In case you missed it, PVSCSI drivers on virtual machines aren’t supported on operating system disks unless you are running vSphere 4 update 1. You can use the driver on a secondary data disk if you so desire, but for this post I am going to assume you are running vSphere 4 update 1 (Virtual Center and ESX Hosts) and want to know how to get the driver working on all disks.
In most cases, it’s always easier to build new. You know you have a clean install, the drivers are updated, the configuration is solid. I would suggest updating your templates to include the new paravirtual scsi driver. Your existing virtual machines run fine with their existing configurations, and depending on your environment, it might be a lot of work to go back and target all of your virtual machines. For an upgrade path, my personal opinion would be to target your heavy I/O virtual machines. Upgrade the VM’s that will make a difference, and you will see some immediate benefits. Reducing the I/O on the disk subsystem will only benefit the other virtual machines that might share those same physical disk spindles.
Clean install
This section will walk you through the process of installing the driver with a Microsoft Windows 2003/2008 operating system. Currently these two operating systems are the only ones supported. Hopefully we will see some added support for the various Linux operating systems down the road.
Walk through the “New virtual machine Wizard” as you normally would. On step 9, ensure you select the “VMware Paravirtual” option as seen below.
Before powering your new VM up, you need to connect the virtual floppy image file that has the driver for your desired guest operating system. This is not on the VMware.com website under downloads, it already exists on your ESX host. You will need to browse to the following location on your ESX host. [Datastores]\vmimages\floppies I would wait to connect your floppy disk image after you boot off the Windows CD-ROM so it doesn’t try to boot off the floppy drive.
When you power up your new virtual machine, select the F6 option to tell the operating system you need to use a third party SCSI driver:
Now connect your floppy disk image to your virtual machine under the “edit settings” option. You should now be able to point to operating system to the driver as seen below:
Continue on with your normal installation, and you are complete. Your new virtual machine is now utilizing the Paravirtual SCSI drivers. I suggest now converting this image you created to a template for future deployments with this configuration.
Upgrading and Existing Virtual Machine
To upgrade an existing virtual machine, the process is pretty straight forward. Assuming you have already upgraded to the latest virtual hardware (Version 7), make sure your VMtools are upgraded post Update 1. Shut down the VM, and edit the settings “Change Type” as shown below:
You will get another window that will alllow you to change the type of controller as seen below:
Select the “VMware Paravirtual” and then select ok. Boot up your virtual machine and you are all set. Your system is now running with the updated drivers and you can take advantage of the newer drivers that provide better throughput and less latency!
Hope you found this post useful. Good luck!
Scott
More Bang for Your Buck with PVSCSI (Part 1)
Nov 21st
One of the new features that was added to the release of VMWare vSphere 4.0 was a new SCSI subsystem driver that allows more I/O and less latency per virtual machine. What the heck is PVSCSI? Here is the technical definition stripped right from the vSphere storage guide. (RTFM).
VMware Paravirtualized SCSI (PVSCSI) is a special purpose driver for high-performance storage adapters that offer greater throughput and lower CPU utilization for virtual machines. They are best suited for environments in which guest applications are very I/O intensive. VMware requires that you create a primary adapter for use with a disk that will host the system software (boot disk) and a separate PVSCSI adapter for the disk that will store user data, such as a database. The primary adapter will be the default for the guest operating system on the virtual machine. For example, a virtual machine with Microsoft Windows 2008 guest operating systems, LSI Logic is the default primary adapter. The PVSCSI driver is similar to vmxnet in that it is an enhanced and optimized special purpose driver for VM traffic and works with only certain Guest OS verision that currently include Windows Server 2003, 2008 and RHEl 5. It can also be shared by multiple VMs running on a single ESX, unlike the VMDirectPath I/O which will dedicate a single adaptor to a single VM.”
So what does all that mean for you? Better disk performance and less CPU cycles spent on processing these disk requests. I took some notes at VMWorld 2009 during a few different sessions that discussed PVSCSI. Here is my logical diagram of what PVSCSI is. Download the PDF version here so you can print it out and frame it on your cube wall!
With the release of vSphere 4.0 PVSCSI was only supported on disks other than the operating system (secondary data drives). For more information on this, reference KB Article: 1010398.
vSphere 4 update 1 is now released and it’s exciting news for those looking at utilizing PVSCSI. Support for boot disk devices attached to a Paravirtualized SCSI ( PVSCSI) adapter has been added for Windows 2003 and 2008 guest operating systems.
So let’s first find out if it’s all that. We need to do some testing to validate the hype. I created two virtual machines, one with the traditional LSI Logic SCSI driver, and one with the new PVSCSI driver. The host is the same for each VM, 4 socket Intel Xeon system with 64 GB of RAM, connected to EMC Clariion CX3-80 storage. The Raid configuration is a 4+1 RAID 5 set (10K spindles), with the default Clariion Active/Passive MRU setup (No PPVE). Each VM has 2 vCPU’s and 4 GB of RAM and both are running 32 bit Microsoft Windows 2003 R2. Both Virtual Machines data disks were formatted using diskpart and the tracks were correctly aligned. Anti-virus real time scanning was disabled on both systems. This test is meant to get as close as possible to a standard configuration that we can benchmark from.
I used IOMETER as my testing engine. I didn’t go too deep on the various settings. The first run is 32K 50%R 0%W.
Non-PVSCSI
With-PVSCSI
Quite the difference, no? To be honest, I was seeing a lot of fluctuation while doing my tests. I probably should have segregated things out a little more, but the screen captures were the average of the results. I was thinking maybe I should use the built-in random IOmeter combined results. So here you go.
Non-PVSCSI
With-PVSCSI
I believe the results speak for themselves. I need to do a little more testing for my own personal preferences. I want to get a more insight on what the differences are on the reads/writes and the various sizes. I am certain cache has a lot to do with the results, but I think IOmeter can bypass cache since you force the randomizer. I’m also curious about the sweet spot on the block sizes and how that plays out with read vs write.
Conclusion
PVSCSI is a technology worth moving towards. There is no cost involved, and it can deliver better disk performance across your ESX environment. It also can bring your host CPU utilization down, which can provide you with better consolidation ratios across your clusters. Stay tuned for part 2, when I am going to provide the “how to do it” aspect so you can begin to leverage this technology you are already paying for!
Hope you found this information helpful. Thanks goes out to Aaron Sweemer (@asweemer) for allowing me to abuse his website and not having to deal with bringing up my own site.
Thanks!
Scott Sauer