Storage
VMware pvSCSI – When and when not to use it
Feb 5th
Introduction
Hopefully you have read my previous blog posts on pvSCSI. It describes what the driver is, how it works, and how it can positively impact your performance and workloads. Part two covers the process of installing the pvSCSI driver on an existing system and a new system. Both can be found here on the site and you might find them useful:
http://www.virtualinsanity.com/index.php/2009/11/21/more-bang-for-your-buck-with-pvscsi-part-1/
http://www.virtualinsanity.com/index.php/2009/12/01/more-bang-for-your-buck-with-pvscsi-part-2/
Interrupt Coalescing
VMware recently published a KB article that answers a question that has been floating around the community for a while. The pvSCSI driver sounds superior to the LSI driver with direct I/O access to the hypervisor so why not use it in all cases? The article states that you should only use the newer driver when driving higher workloads, those that are typically 2000 IOPS or greater. For those that don’t know 2000 IOPS is a pretty big workload. Consider this, a standard fiber channel 10,000 RPM drive averages around 125 IOPS per disk.
I didn’t really understand this and the knowledge base article is lacking any detail on the rational behind the statement. I reached out to VMware performance engineer Scott Drummonds to see if he had anything he could publish to help clarify the KB article. Scott was nice enough to research this and posted his findings here.
So it appears that the technical explanation is interrupt coalescing or buffering. The paravirtual SCSI driver was designed to handle receiving multiple requests at a high rate and then “batching” the requests together for better efficiencies in throughput. If you aren’t generating high enough workloads on the virtual machine, the I/O request could unnecessarily sit in the queue while the “batch" waits to be filled up for the next transaction. This could cause storage performance problems which would typically be seen as higher latency and would negatively impact the virtual machine.
Now and Then
The great news is the current release of the driver is optimized for heavy workloads. If you are starting to virtualize SQL/Oracle systems and need the performance, go for the pvSCSI driver and get better throughput. If your deploying standard virtual machines that are doing lower workloads, continue to embrace the existing LSI Logic driver.
If you are new to vSphere 4, or have just upgraded from 3.5 and are starting to rebuild your templates to embrace virtual hardware version 7, don’t use the pvSCSI driver as part of your standard template. VMware is working on the driver and will be introducing advanced coalescing functionality. When this is built into the driver stack pvSCSI will then be able to be utilized for all workloads as it will understand when it needs to ramp up for higher workloads.
Thanks again to Scott Drummonds for taking the time out of his busy schedule to track this one down.
More Bang for your Buck with PVSCSI (Part 2)
Dec 1st
Part 2 Doing the work
As you might have noticed, this blog post is a continuation to my first post about PVSCSI, you can access Part 1 here.
Hopefully now you have a better understanding of what the Paravirtual SCSI driver is all about, and we can prove there are tangible reasons to move in this direction. Let’s get on with the important part, the implementation phase.
(I need to finish off this blog post, I am running out of pictures of SCSI cables)
There are some caveats I need to start out with. In case you missed it, PVSCSI drivers on virtual machines aren’t supported on operating system disks unless you are running vSphere 4 update 1. You can use the driver on a secondary data disk if you so desire, but for this post I am going to assume you are running vSphere 4 update 1 (Virtual Center and ESX Hosts) and want to know how to get the driver working on all disks.
In most cases, it’s always easier to build new. You know you have a clean install, the drivers are updated, the configuration is solid. I would suggest updating your templates to include the new paravirtual scsi driver. Your existing virtual machines run fine with their existing configurations, and depending on your environment, it might be a lot of work to go back and target all of your virtual machines. For an upgrade path, my personal opinion would be to target your heavy I/O virtual machines. Upgrade the VM’s that will make a difference, and you will see some immediate benefits. Reducing the I/O on the disk subsystem will only benefit the other virtual machines that might share those same physical disk spindles.
Clean install
This section will walk you through the process of installing the driver with a Microsoft Windows 2003/2008 operating system. Currently these two operating systems are the only ones supported. Hopefully we will see some added support for the various Linux operating systems down the road.
Walk through the “New virtual machine Wizard” as you normally would. On step 9, ensure you select the “VMware Paravirtual” option as seen below.
Before powering your new VM up, you need to connect the virtual floppy image file that has the driver for your desired guest operating system. This is not on the VMware.com website under downloads, it already exists on your ESX host. You will need to browse to the following location on your ESX host. [Datastores]\vmimages\floppies I would wait to connect your floppy disk image after you boot off the Windows CD-ROM so it doesn’t try to boot off the floppy drive.
When you power up your new virtual machine, select the F6 option to tell the operating system you need to use a third party SCSI driver:
Now connect your floppy disk image to your virtual machine under the “edit settings” option. You should now be able to point to operating system to the driver as seen below:
Continue on with your normal installation, and you are complete. Your new virtual machine is now utilizing the Paravirtual SCSI drivers. I suggest now converting this image you created to a template for future deployments with this configuration.
Upgrading and Existing Virtual Machine
To upgrade an existing virtual machine, the process is pretty straight forward. Assuming you have already upgraded to the latest virtual hardware (Version 7), make sure your VMtools are upgraded post Update 1. Shut down the VM, and edit the settings “Change Type” as shown below:
You will get another window that will alllow you to change the type of controller as seen below:
Select the “VMware Paravirtual” and then select ok. Boot up your virtual machine and you are all set. Your system is now running with the updated drivers and you can take advantage of the newer drivers that provide better throughput and less latency!
Hope you found this post useful. Good luck!
Scott
More Bang for Your Buck with PVSCSI (Part 1)
Nov 21st
One of the new features that was added to the release of VMWare vSphere 4.0 was a new SCSI subsystem driver that allows more I/O and less latency per virtual machine. What the heck is PVSCSI? Here is the technical definition stripped right from the vSphere storage guide. (RTFM).
VMware Paravirtualized SCSI (PVSCSI) is a special purpose driver for high-performance storage adapters that offer greater throughput and lower CPU utilization for virtual machines. They are best suited for environments in which guest applications are very I/O intensive. VMware requires that you create a primary adapter for use with a disk that will host the system software (boot disk) and a separate PVSCSI adapter for the disk that will store user data, such as a database. The primary adapter will be the default for the guest operating system on the virtual machine. For example, a virtual machine with Microsoft Windows 2008 guest operating systems, LSI Logic is the default primary adapter. The PVSCSI driver is similar to vmxnet in that it is an enhanced and optimized special purpose driver for VM traffic and works with only certain Guest OS verision that currently include Windows Server 2003, 2008 and RHEl 5. It can also be shared by multiple VMs running on a single ESX, unlike the VMDirectPath I/O which will dedicate a single adaptor to a single VM.”
So what does all that mean for you? Better disk performance and less CPU cycles spent on processing these disk requests. I took some notes at VMWorld 2009 during a few different sessions that discussed PVSCSI. Here is my logical diagram of what PVSCSI is. Download the PDF version here so you can print it out and frame it on your cube wall!
With the release of vSphere 4.0 PVSCSI was only supported on disks other than the operating system (secondary data drives). For more information on this, reference KB Article: 1010398.
vSphere 4 update 1 is now released and it’s exciting news for those looking at utilizing PVSCSI. Support for boot disk devices attached to a Paravirtualized SCSI ( PVSCSI) adapter has been added for Windows 2003 and 2008 guest operating systems.
So let’s first find out if it’s all that. We need to do some testing to validate the hype. I created two virtual machines, one with the traditional LSI Logic SCSI driver, and one with the new PVSCSI driver. The host is the same for each VM, 4 socket Intel Xeon system with 64 GB of RAM, connected to EMC Clariion CX3-80 storage. The Raid configuration is a 4+1 RAID 5 set (10K spindles), with the default Clariion Active/Passive MRU setup (No PPVE). Each VM has 2 vCPU’s and 4 GB of RAM and both are running 32 bit Microsoft Windows 2003 R2. Both Virtual Machines data disks were formatted using diskpart and the tracks were correctly aligned. Anti-virus real time scanning was disabled on both systems. This test is meant to get as close as possible to a standard configuration that we can benchmark from.
I used IOMETER as my testing engine. I didn’t go too deep on the various settings. The first run is 32K 50%R 0%W.
Non-PVSCSI
With-PVSCSI
Quite the difference, no? To be honest, I was seeing a lot of fluctuation while doing my tests. I probably should have segregated things out a little more, but the screen captures were the average of the results. I was thinking maybe I should use the built-in random IOmeter combined results. So here you go.
Non-PVSCSI
With-PVSCSI
I believe the results speak for themselves. I need to do a little more testing for my own personal preferences. I want to get a more insight on what the differences are on the reads/writes and the various sizes. I am certain cache has a lot to do with the results, but I think IOmeter can bypass cache since you force the randomizer. I’m also curious about the sweet spot on the block sizes and how that plays out with read vs write.
Conclusion
PVSCSI is a technology worth moving towards. There is no cost involved, and it can deliver better disk performance across your ESX environment. It also can bring your host CPU utilization down, which can provide you with better consolidation ratios across your clusters. Stay tuned for part 2, when I am going to provide the “how to do it” aspect so you can begin to leverage this technology you are already paying for!
Hope you found this information helpful. Thanks goes out to Aaron Sweemer (@asweemer) for allowing me to abuse his website and not having to deal with bringing up my own site.
Thanks!
Scott Sauer
Get Thin Provisioning working for you in vSphere
Oct 12th
Going Thin and not looking back.
Yes, I am slowly losing my hair like many other aging men out there, but it wouldn’t be virtual insanity if I were blogging about my personal male pattern baldness issues. With the latest release of VMware vSphere comes a lot of new features and functionality that can be leveraged to make our lives easier. One of these features, that I personally have been looking forward to for a while, is Thin Provisioning. If you aren’t familiar with this technology, jump over to Gestalt IT for a great explanation of what it is and how it works.
One of the exciting promises of thin provisioning, is getting more “bang for your buck” out of the expensive enterprise storage you have been investing in for your ESX environment. But, as Bret Michael’s once said, “Every rose has its thorn” and there are some things to look out for and considerations to make, before implementing thin disk technologies.
Efficiencies are great if they work right and don’t over
complicate the environment.
Do your homework and make sure you understand the characteristics of the virtual machine that you are considering migrating into a thin disk configuration. The last thing you want to do is convert every VM to thin disk, and four months down the road all of your data stores are filling up and you’re scrambling for a storage CAPEX. Some people are of the opinion to do thin provisioning either on the host side (VMware) or on the storage array side, but not both. Take a gander at Chad Sakac’s blog that discusses thin on thin and some thoughts around each of these approaches. I’m not going to go into all of the pluses and minuses of thin provisioning but rather focus on how to make it work for you.
Coffee Talk
So now that we have some of the basics out of the way, I wanted to share my thoughts on thin provisioning. Like many organizations, we get requests from our customers that err on the side of caution. They want to plan for the worse case and ensure that their project and/or application isn’t setup for failure. I don’t blame them really, I do it myself all the time when I make coffee at home. I always end up making more coffee than I typically drink, just in case I might need that extra charge. The best way to do that is pad it, request more than what you might really need, just in case something comes up down the road. Virtual machine disk storage in some cases fits this same profile. If my coffee maker granted me access to hot coffee on demand, I would stop making extra coffee. Thin disks can give your end users that capacity on demand so you can gain control of the padding effect that typically takes place in most corporate organizations.
Take it back…
So now you have done your research, you’re starting to get a feel for what this thin stuff is and how it might play out in your shop. It’s go time. If you’re a smaller VMware customer, you probably already have an idea of what are good target disks to convert. If you’re a larger environment, it might be a little more difficult to gauge where the bloated pigs are hiding.
I worked at GE for a couple of years and was exposed to some of the Six Sigma methodologies they preach as well as practice. Sounds boring, right? Not really. You can really leverage DMAIC for a lot of IT related problems/issues/projects. You don’t have to take it to the extreme, use the framework to help guide you on your quest:
DMAIC
The DMAIC project methodology has five phases:
- Define high-level project goals and the current process.
- Measure key aspects of the current process and collect relevant data.
- Analyze the data to verify cause-and-effect relationships. Determine what the relationships are and attempt to ensure that all factors have been considered.
- Improve or optimize the process based upon data analysis using techniques like Design of experiments.
- Control to ensure that any deviations from target are corrected before they result in defects. Set up pilot runs to establish process capability, move on to production, set up control mechanisms and continuously monitor the process.
We have already defined our project goals and what we are trying to accomplish. We need a good “Measure” tool to really find where we might benefit from thin provisioning. Powershell is a great tool that most VMware administrators use, or have at least heard of. So this was the first place I turned to for assistance.
Alan Renouf of “Virtu-AL” http://www.virtu-al.net/ gave me a hand in writing the powershell script needed. (Thanks again, Alan!). Alan already had a one liner script to produce a list of vm’s, their disks assigned, and how much data each disk was consuming. I needed the ability to see this data outside a powershell window and be able to analyze it in a better format. We have a decent-sized VMware environment and exporting this out to a .csv for analysis is extremely helpful. Here is the script!
************************************************************************
# Set the Filename for the exported data
$Filename = “C:\VMDisks.csv”
Connect-VIServer MYVIServer
$AllVMs = Get-View -ViewType VirtualMachine
$SortedVMs = $AllVMs | Select *, @{N=”NumDisks”;E={@($_.Guest.Disk.Length)}} | Sort NumDisks -Descending
$VMDisks = @()
ForEach ($VM in $SortedVMs){
$Details = New-object PSObject
$Details | Add-Member -Name Name -Value $VM.name -Membertype NoteProperty
$DiskNum = 0
Foreach ($disk in $VM.Guest.Disk){
$Details | Add-Member -Name “Disk$($DiskNum)path” -MemberType NoteProperty -Value $Disk.DiskPath
$Details | Add-Member -Name “Disk$($DiskNum)Capacity(MB)” -MemberType NoteProperty -Value ([math]::Round($disk.Capacity/ 1MB))
$Details | Add-Member -Name “Disk$($DiskNum)FreeSpace(MB)” -MemberType NoteProperty -Value ([math]::Round($disk.FreeSpace / 1MB))
$DiskNum++
}
$VMDisks += $Details
Remove-Variable Details
}
$VMDisks | Export-Csv -NoTypeInformation $Filename
***********************************************************************
So now that you have this great spreadsheet, you can do all sorts of crazy sorting and reporting, within Excel. Take some time on phase 3, “Analyze” what you’re seeing. Talk to your VM stakeholders to see how things might be changing from their perspective. Try to plan for the surprises and position yourself accordingly.
Next is the “Improve” phase of DMAIC (see it’s easy!). This is the part where you actually do the work. It’s time to start leveraging the storage VMotion API’s, and reclaim some of that unused disk.
- Select the target VM in the VC client.
- Right click on the VM and select the option “Migrate”.
- Select the option “Change Datastore”.
- Select the destination, or click advanced if you are targeting one particular disk.
- Select “Thin provisioned format”.
- Select Finish.
Rinse and Repeat for the rest of that spreadsheet you have worked so hard on.
The last phase of DMAIC is “Control”. This is one of the most important pieces to thin provisioning in my opinion. At the minimum you need to setup Virtual Center alerts to monitor when your datastores are approaching critical levels. You can’t implement thin disks in your vSphere environment and walk away. The smart people over at VMware have given us the ability to monitor datastore disk space usage and over-allocation with the latest release of Virtual Center. Setup your monitors so you are e-mailed when some of these thin disks begin to grow and you need to take some action.
Eric Gray of VMware takes this to the next level, check out his blog post on utilizing powershell to prevent datastore emergencies. My personal approach to this concept is to setup a “hotspare” datastore for your environment. A good practice to implement here would be to try reclaiming enough storage from your migrations to thin disks to free-up a “hot spare datastore”. Implementing an automated recovery solution like Eric’s will help you sleep easier at night. Worried about what might happen if your script doesn’t work or you do hit the perfect storm and end up with a full VMFS volume? Intelligence has been built into vSphere to automatically pause the virtual machines, impressive. Check out Eric’s video:
Wrapping it all up
Thin disk provisioning is a great feature that you should consider leveraging in your environment. With some forward thinking and best practices you can achieve higher ROI for your ESX storage. VMware vSphere offers the ability for you to migrate from thick to think with no downtime, so you can begin reclaiming storage on the fly. Keep it simple, start out with a high level analysis of your infrastructure. Identify the candidates that are a good fit and worth focusing on. Setup your alerts on the datastores as soon as you migrate your first virtual machine so you are protecting yourself from problems down the road. Consider taking automated actions if your datastores are reaching critical thresholds.
I hope you found this article helpful, good luck!
Scott Sauer
VCDX Admin Exam Notes – Section 1.4
May 19th
I’m trying to get these out faster. But I’m finding it’s a pain to compile, tweak and reformat my notes so they make sense and look right on the blog. Here’s the next in the series.
Objective 1.4 – Implement and manage Storage VMotion.
Knowledge
Describe Storage VMotion operation
I like to think of Storage VMotion as the “inverse” of VMotion. Instead of moving (live) the front end (i.e. CPU, memory, and network) from one physical device to another, Storage VMotion moves (live) the back end (i.e. disk) from one physical device to another.
The following was taken directly from the VMware.com website and acurately describes how a Storage VMotion works.
- Before moving disk files, Storage VMotion creates a new virtual machine home directory for the virtual machine in the destination datastore.
- Next, a new instance of the virtual machine is created. Its configuration is kept in the new datastore.
- Storage VMotion then creates a child disk for each virtual machine disk that is being moved to capture a copy of write activity, while the parent disk is in read only mode.
- The original parent disk is copied to the new storage location.
- The child disk is re-parented to the newly copied parent disk in the new location.
- When the transfer to the new copy of the virtual machine is completed, and the original instance is shut down. Then, the original virtual machine home is deleted from VMware vStorage VMFS at the source location.
Explain implementation process for Storage VMotion
For ESX 3.5, Storage VMotion is a command-line only implementation. There are third party GUI plugins to the VI client that I have used and work really well, but they are of course not supported by VMware. And for the purposes of this post, I would imagine the VCDX exam will stick to VMware only supported implementations.
To execute a Storage VMotion, you’ll need the RCLI (Remote Command Line Interface) from VMware.com. There is an RCLI for both Windows and Linux, and there’s an RCLI virtual appliance too. Take your pick, and then be sure to review the Remote Command-Line Interface Installation and Reference Guide for more info on RCLI. But specifically for Storage VMotion, the RCLI command comes in two flavors (from the guide):
To use the command in interactive mode, type svmotion –interactive. You are
prompted for all the information necessary to complete the storage migration.
When you invoke the command in interactive mode, all other parameters are
ignored.In noninteractive mode, the svmotion command uses the following syntax:
svmotion [standard Remote CLI options] –datacenter=<datacenter name>
–vm <VM config datastore path>:<new datastore>
[--disks <virtual disk datastore path>:<new datastore>,
<virtual disk datastore path>:<new datastore>]
- Identify Storage VMotion use cases
There are a number of reasons that I can think of where you’d want to use Storage VMotion. Some of the more obvious reasons (at least to me, anyway) would be:
- Array maintenance
- Migrating to newer or different (i.e. FC, iSCSI, NAS) hardware
- Adding new storage
- To achieve optimal distribution of storage consumption across LUNs
- To resolve storage bottlenecks and other performance issues
I’m sure if I thought about it some more, I could come up with a few more. But I think this is a decent list.
Understand performance implications for Storage VMotion
There are a couple things that occur during a Storage VMotion that could affect performance, and therefore need to be considered when planning to move storage.
- During the Storage VMotion, the VM actually does a “Self VMotion,” meaning the VM is VMotioned to the same host it’s already running on (review step #2 in the graphic above). And therefore during this time, there is temporarily twice the amount of memory consumed.
- During the move, extra disk space is required on the source volume while all disk writes are redirected to the snapshot disk.
- All disk I/0 for the copy (i.e. read from the source volume, write to the snapshot disk and write to the destination volume) is going through the VMkernel of the host.
Skills and Abilities
Use Remote CLI to perform Storage VMotion operations
- Interactive mode
This is pretty easy. Just enter svmotion –interactive at the command line and then follow the prompts. - Non-interactive mode
In my environment, the command to move all the virtual disks associated with aaron-corp-xp (the name of an actual VM I use) from vol1 to vol2, would look like this …
svmotion –url=https://10.10.8.60/sdk –username=aaron –password=<yeah right> –datacenter=cincylab –vm=’[vol1] aaron-corp-xp/aaron-corp-xp.vmx: vol2’
VCDX Admin Exam Notes – Section 1.3
May 18th
Ugh. My brain hurts. I’ve spent the past few hours reviewing scripted ESX installations and working on a PowerShell script for a customer that will reorder vmnics after a scripted installation is complete (because I can’t find any other way to force their order during the install). It’s been a few months since I’ve done a scripted installation, so I definitely needed a refresher. Plus, according to the VMware Enterprise Administration Exam Blueprint v3.5, section 8.1 is all about automating ESX deployments. The good news is that section 8.1 is the last section of the blueprint, so I believe I’m almost done preparing for the VCDX Admin Exam, which I’m scheduled to take in a few days.
Anyway, going back to the beginning of the Blueprint, and continuing from where I left off, here is the next section of my study notes.
Objective 1.3 – Troubleshoot Virtual Infrastructure storage components.
Knowledge
Identify storage related events and log entries. Analyze storage events to determine related issues.
All storage related events will be recorded in the /var/log/vmkernel log file. Most of the messages in this log file are fairly cryptic and can be difficult to interpret. Furthermore, this log file contains all messages from the vmkernel, not just storage related messages, so you’ll have to filter through it. An easy way to do this is simply to search for SCSI. For example, the command cat /var/log/vmkernel | grep SCSI on one of my servers produces the following output (only showing the last 10 lines) …
[root@cincylab-esx3 root]# cat /var/log/vmkernel | grep SCSI | tail -10
May 18 12:57:47 cincylab-esx3 vmkernel: 2:16:01:55.748 cpu2:1069)iSCSI: login phase for session 0×8603f90 (rx 1071, tx 1070) timed out at 23051576, timeout was set for 23051576
May 18 12:57:47 cincylab-esx3 vmkernel: 2:16:01:55.748 cpu2:1071)iSCSI: session 0×8603f90 connect timed out at 23051576
May 18 12:57:47 cincylab-esx3 vmkernel: 2:16:01:55.748 cpu2:1071)<5>iSCSI: session 0×8603f90 iSCSI: session 0×8603f90 retrying all the portals again, since the portal list got exhausted
May 18 12:57:47 cincylab-esx3 vmkernel: 2:16:01:55.748 cpu2:1071)iSCSI: session 0×8603f90 to iqn.2004-08.jp.buffalo:TS-IGLA68-001D7315AA68:vol1 waiting 1 seconds before next login attempt
May 18 12:57:48 cincylab-esx3 vmkernel: 2:16:01:56.748 cpu2:1071)iSCSI: bus 0 target 0 trying to establish session 0×8603f90 to portal 0, address 10.10.8.200 port 3260 group 1
May 18 12:58:00 cincylab-esx3 vmkernel: 2:16:02:09.355 cpu2:1071)iSCSI: bus 0 target 0 established session 0×8603f90 #3, portal 0, address 10.10.8.200 port 3260 group 1
May 18 12:58:01 cincylab-esx3 vmkernel: VMWARE SCSI Id: Supported VPD pages for vmhba35:C0:T0:L0 : 0×0 0×80 0×83
May 18 12:58:01 cincylab-esx3 vmkernel: VMWARE SCSI Id: Device id info for vmhba35:C0:T0:L0: 0×1 0×1 0×0 0×18 0×42 0×55 0×46 0×46 0×41 0×4c 0×4f 0×0 0×0 0×0 0×0 0×0 0×1 0×0 0×0 0×0 0×0 0×0 0×0 0×0 0×2 0×0 0×0 0×0
May 18 12:58:01 cincylab-esx3 vmkernel: VMWARE SCSI Id: Id for vmhba35:C0:T0:L0 0×20 0×20 0×20 0×20 0×56 0×49 0×52 0×54 0×55 0×41
[root@cincylab-esx3 root]#
If you look closely, I clearly had some issues with my iSCSI appliance a few hours ago. I decided make some configuration changes to the switch and then, all of a sudden, the ESX server lost connectivity to its storage. Weird! 
Anyway, what does all this mean? There’s an really good VMworld Europe 2008 presentation (which you can get from www.vmworld.com) titled VI3 Advanced Log Analysis, which goes into detail about how to interpret VMware log files. From that presentation, I found this diagram which describes the components of a message in the vmkernel log file.
Skills and Abilities
Verify storage configuration and troubleshoot storage connection issues using CLI , VI Client and logs
- Rescan events
A rescan event can be initiated with the esxcfg-rescan at the command line. The output should look like the following …
[root@cincylab-esx3 root]# esxcfg-rescan vmhba32
Rescanning vmhba32 …
On scsi1, removing: 0:0.
On scsi1, adding: 0:0.
Done.
[root@cincylab-esx3 root]# cat /var/log/vmkernel | grep SCSI | tail -3
May 18 19:13:09 cincylab-esx3 vmkernel: VMWARE SCSI Id: Supported VPD pages for vmhba32:C0:T0:L0 : 0×0 0×80 0×83
May 18 19:13:10 cincylab-esx3 vmkernel: VMWARE SCSI Id: Device id info for vmhba32:C0:T0:L0: 0×2 0×0 0×0 0×18 0×4c 0×69 0×6e 0×75 0×78 0×20 0×41 0×54 0×41 0×2d 0×53 0×43 0×53 0×49 0×20 0×73 0×69 0×6d 0×75 0x
May 18 19:13:10 cincylab-esx3 vmkernel: VMWARE SCSI Id: Id for vmhba32:C0:T0:L0 0×36 0×52 0×58 0×36 0×4a 0×39 0×39 0×58 0×20 0×20 0×20 0×20 0×20 0×20 0×20 0×20 0×20 0×20 0×20 0×20 0×47 0×42 0×30 0×31 0×36 0x
[root@cincylab-esx3 root]#
- Failover events
I don’t have redundant paths in my lab to simulate this. So, again from the VMworld presentation, VI3 Advanced Log Analysis, here is a screen shot from the slide that covers this topic.
There will obviously be a lot of different types of error and event messages in /var/log/vmkernel. And I’m certainly not going to try and list every possible combination here. I highly suggest you download the VMworld preso because it does a great job of explaining how to further decipher the log files (like defining SCSI error codes).
Well, that’s about it for this section. Back to PowerShell scripting for another hour or so.
VCDX Admin Exam Notes – Section 1.2
May 11th
Last week I was in San Francisco with most (maybe all) of the VMware field technical folks for a three day technical summit. One of the evenings we had an awards ceremony and dinner. And guess what? The first eight VCDX certifications ever to be awarded were announced.
Now, VMware is a pretty big company, so I didn’t recognize seven of the eight names. But I definitely recognized one of them. Well, that is, I should say I recognized his name. Having never officially met him fact to face, I couldn’t pick him out of a crowd of two. You might know him as the rock star blogger from Yellow Bricks. Congratulations Duncan Epping! I believe he said he is VCDX number seven and the first VCDX in Europe. Very cool. And well deserved, for sure.
I’m a little behind Duncan. I’m scheduled to take the Admin Exam later this month, which is the first of two exams. Then I’ll have to present and defend a successful design and deployment before a jury … er, I mean, panel of my peers.
Anyway, here are my notes for section 1.2 of the VMware Enterprise Administration Exam Blueprint v3.5. (Section 1.1 can be found here). Everything in Blue is a direct cut and paste from the exam blueprint.
Objective 1.2 – Implement and manage complex data security and replication
configurations.
Knowledge
Describe methods to secure access to virtual disks and related storage devices
- Distributed Lock Handling
In the graphic below, notice how each ESX server sees and has access to the same LUN? This is achieved via VMFS, a clustered file system which leverages distributed file locking to allow multiple hosts to access the same storage. When a Virtual Machine is powered on, VMFS places a lock on its files, ensuring no other ESX server can access them.
Identify tools and steps necessary to manage replicated VMFS volumes
- Resignaturing
First, there’s a really good article on VMFS resignaturing by Duncan (go figure). Also, Chad Sakac over at Virtual Geek has a great article too. I’m not going to reinvent the wheel, so make sure you read their posts. You’ll need to understand this. For the exam, you’ll certainly need to know the following …The following is from the Fibre Channel SAN Configuration Guide:
EnableResignature=0, DisallowSnapshotLUN=1 (default)
In this state:
- You cannot bring snapshots or replicas of VMFS volumes by the array into the ESX Server host regardless of whether or not the ESX Server has access to the original LUN.
- LUNs formatted with VMFS must have the same ID for each ESX Server host.
EnableResignature=1, (DisallowSnapshotLUN is not relevant)
In this state, you can safely bring snapshots or replicas of VMFS volumes into the same servers as the original and they are automatically resignatured.EnableResignature=0, DisallowSnapshotLUN=0
This is similar to ESX 2.x behavior. In this state, the ESX Server assumes that it sees only one replica or snapshot of a given LUN and never tries to resignature. This is ideal in a DR scenario where you are bringing a replica of a LUN to a new cluster of ESX Servers, possibly on another site that does not have access to the source LUN. In such a case, the ESX Server uses the replica as if it is the original.Do not use this setting if you are bringing snapshots or replicas of a LUN into a server
with access to the original LUN. This can have destructive results including:
- If you create snapshots of a VMFS volume one or more times and dynamically
bring one or more of those snapshots into an ESX Server, only the first copy is
usable. The usable copy is most likely the primary copy. After reboot, it is
impossible to determine which volume (the source or one of the snapshots) is
usable. This nondeterministic behavior is dangerous.- If you create a snapshot of a spanned VMFS volume, an ESX Server host might
reassemble the volume from fragments that belong to different snapshots. This can
corrupt your file system.
Skills and Abilities
Configure storage network segmentation
- FC Zoning
Zoning delivers access control in the SAN, restricting visibility to devices in the zone solely to other members of that zone. It is a common technique used to do things like group ESX servers into production/test/dev, increase security and decrease traffic, among other things.
- iSCSI/NFS VLAN
Storage segmentation for IP storage can be accomplished in one of two ways: VLANs or physical segmentation (i.e. separate layer 2 switches for storage).
Configure LUN masking
The Disk.MaskLUNs parameter should be used when you’re trying to mask specific LUNs to your ESX host. This is a useful option when you don’t want your ESX server to access a particular LUN, but are unwilling (or unable) to configure your FC switch.
To configure LUN masking in the VI Client go to Configuration –> Advanced Settings for the host you want to configure. You’ll find the Disk.MaskLUNs parameter under the section Disk. It looks like this in my VI Client.
Enter a value in the following format … <adapter>:<target>:<comma separated LUN range list>. Be sure to rescan when your done and verify the Mask has been properly applied.
Use esxcfg-advcfg
This one’s easy. Just use the man page (type “man esxcfg-advcfg” at the command prompt). It’ll tell you everything you need to know
Set Resignaturing and Snapshot LUN options
So, following along with the man page above, here is a cut and paste from my server …
[asweemer@cincylab-esx3 config]$ su -
Password:
[root@cincylab-esx3 root]# esxcfg-advcfg -s 0 /LVM/EnableResignature
Value of EnableResignature is 0
[root@cincylab-esx3 root]# esxcfg-advcfg -s 1 /LVM/EnableResignature
Value of EnableResignature is 1
[root@cincylab-esx3 root]#
[root@cincylab-esx3 root]# esxcfg-advcfg -s 0 /LVM/DisallowSnapshotLun
Value of DisallowSnapshotLun is 0
[root@cincylab-esx3 root]# esxcfg-advcfg -s 1 /LVM/DisallowSnapshotLun
Value of DisallowSnapshotLun is 1
[root@cincylab-esx3 root]#
Manage RDMs in a replicated environment
RDMs can be created via the CLI with the following command …
vmkfstools -r /vmfs/devices/disks/vmhbaX:Y:Z:0 my-vm.vmdk
By default, the RDM will be created in Virtual Compatibility Mode. But should you need and/or prefer Physical Compatibility Mode, you can change this by editing the VMDK file and changing the createType value to vmfsPassthroughRawDeviceMap.
Use proc nodes to identify driver configuration and options
The proc filesystem is a pseudo filesystem, it’s not “real.” It consumes no storage space and is used to access process information from the kernel. You’ll find quite a bit of valuable data and configuration options in the many subdirectories of /proc/vmware/config. Here’s a quick example from my ESX server …
[asweemer@cincylab-esx3 LVM]$ pwd
/proc/vmware/config/LVM
[asweemer@cincylab-esx3 LVM]$ ls
DisallowSnapshotLun EnableResignature
[asweemer@cincylab-esx3 LVM]$ cat EnableResignature
EnableResignature (Enable Volume Resignaturing) [0-1: default = 0]: 0
[asweemer@cincylab-esx3 LVM]$
Use esxcfg-module
Just like esxcfg-adv, use the man page.
VCDX Admin Exam Notes — Section 1.1
Apr 27th
I finally got a chance to sit down and reformat some of my notes for the VCDX Admin Exam. Below are my notes for Section 1.1 of the VMware Enterprise Administration Exam Blueprint v3.5. Everything in Blue is a direct cut and past from the exam blueprint.
Oh, and thanks to the Disqus comment from VirtualizationTeam (Blog), letting me know that Peter van den Bosch has a more recent version of his VMware Enterprise Administration Exam Study Guide 3.5.
Section 1 – Storage
Objective 1.1 – Create and Administer VMFS datastores using advanced techniques.
Knowledge
Describe how to identify iSCSI, Fibre channel, SATA and NFS configurations using CLI commands and log entries
Here are a few command line examples that I believe would work well …
1) esxcfg-mpath –l
This command produces the following output on my server:
[root@cincylab-esx3 root]# esxcfg-mpath -l
Disk vmhba0:0:0 /dev/sdb (152627MB) has 1 paths and policy of Fixed
Local 0:31.2 vmhba0:0:0 On active preferred
Disk vmhba32:0:0 /dev/sda (152627MB) has 1 paths and policy of Fixed
Local 0:31.2 vmhba32:0:0 On active preferred
Disk vmhba35:0:0 /dev/sdc (923172MB) has 1 paths and policy of Fixed
iScsi sw iqn.1998-01.com.vmware:cincylab-esx3-1d029e5f<->iqn.2004-08.jp.buffalo:TS-IGLA68-001D7315AA68:vol1 vmhba35:0:0 On active preferred
2) esxcfg-info –s
The –s flag will narrow the scope of the output to just storage and disk related info. But even with the narrowed scope, this command produces way too much output to be displayed here. You’ll likely want to pipe the output into grep, or at a minimum to a more/less to get what you’re looking for.
3) cat /var/log/vmkernel | grep vmhba | tail –10
This will search the vmkernel log file and display the last 10 lines containing the text vmhba. If you want more (or fewer lines) change the –10 to whatever suits your needs.
If found this one particularly useful when you’ve enabled the software iSCSI initiator at the command line, but don’t know yet number has been assigned to the vmhba (e.g. vmhba35).
4) esxcfg-vmhbadevs –m and ls –lah /vmfs/volumes
The command esxcfg-vmhbadevs –m will show the mapping between vmhba numbers, device files and their UUIDs. If you’d like a quick and easy way to see what UUIDs are mapped to their human readable name, you can follow that up with a ls –lah /vmfs/volumes. The two commands back to back produce the following output on my server:
[root@cincylab-esx3 root]# esxcfg-vmhbadevs -m
vmhba35:0:0:1 /dev/sdc1 4986310d-6525e5e6-ebbd-00237d0681e7
vmhba0:0:0:3 /dev/sdb3 49e115fb-3e22358c-c10a-00237d0681e7
vmhba32:0:0:1 /dev/sda1 4985c53e-e7b1904f-5042-00237d0681e7
[root@cincylab-esx3 root]# ls -lah /vmfs/volumes/
total 10M
drwxr-xr-x 1 root root 512 Apr 20 23:07 .
drwxrwxrwt 1 root root 512 Apr 11 18:12 ..
drwxr-xr-t 1 root root 1.2K Feb 1 21:34 4985c53e-e7b1904f-5042-00237d0681e7
drwxr-xr-t 1 root root 3.7K Apr 14 14:49 4986310d-6525e5e6-ebbd-00237d0681e7
drwxr-xr-t 1 root root 980 Apr 11 18:13 49e115fb-3e22358c-c10a-00237d0681e7
lrwxr-xr-x 1 root root 35 Apr 20 23:07 cincylab-esx3:storage1 -> 4985c53e-e7b1904f-5042-00237d0681e7
lrwxr-xr-x 1 root root 35 Apr 20 23:07 cincylab-esx3:storage2 -> 49e115fb-3e22358c-c10a-00237d0681e7
lrwxr-xr-x 1 root root 35 Apr 20 23:07 vol1 -> 4986310d-6525e5e6-ebbd-00237d0681e7
5) vmkiscsi-ls
This one only applies to iSCSI storage, of course, and produces the following output on my server:
[root@cincylab-esx3 root]# vmkiscsi-ls
*************************************************************
SFNet iSCSI Driver Version … 3.6.3 (27-Jun-2005 )
*************************************************************
TARGET NAME : iqn.2004-08.jp.buffalo:TS-IGLA68-001D7315AA68:vol1
TARGET ALIAS :
HOST NO : 4
BUS NO : 0
TARGET ID : 0
TARGET ADDRESS : 10.10.8.200:3260
SESSION STATUS : ESTABLISHED AT Sun Apr 12 11:35:09 2009
NO. OF PORTALS : 1
PORTAL ADDRESS 1 : 10.10.8.200:3260,1
SESSION ID : ISID 00023d000001 TSIH 1400
*************************************************************
Describe the VMFS file system
There are many subsections here and before digging into each one, check out the following three links …
- Advanced VMFS Configuration and Troubleshooting.
- Really advanced, but really good: Understanding VMFS Volumes
- An oldie but goodie: VMware Virtual Machine File System: Technical Overview and Best Practices
Metadata
The simple definition of Metadata is “data about data.” All file systems handle metadata differently. VMFS uses metadata, stored in a special area of each volume, to manage all the files, directories (in VMFS-3 only), and attributes about the volume. VMFS is a clustered file system, meaning more than one ESX server can access the same file system at the same time. Therefore an update to the metadata requires locking of the LUN using a SCSI reservation.
Multi-access and locking
The following was taking from Advanced VMFS Configuration and Troubleshooting.
Distributed Lock handling by VMFS3
- Done in-band
- Hosts mount a VMFS3 volume
- Hosts’ ids posted to heartbeat region
- Heartbeat records are updated at regular intervals by hosts
- Host X locks a file, the lock is associated with its ID
- If host X dies or loses access to volume the file lock is stale
- Host Z attempts to lock the same file which is locked
- Host Z check the heartbeat record of Host X (~5 times)
- If host X heartbeat record is not updated, Host Z will age the lock
- All other hosts yield to host Z and not attempt to lock the file
- Lock is broken and Host Z acquires the lock
- Journal is replayed by Host Z
Extents
Extents are logical extensions of a file system. They are typically used to grow a volume beyond the VMFS size limitations. Essentially, an extent is the “joining” of two or more volumes into a single, logical VMFS volume.
Tree structure and files
The vmfs partition is mounted to the directory with the corresponding UUID found in /vmfs/volumes. The human readable name of the volume is merely a symbolic link to that directory. By default, all VMs are given a directory at the root of the partition. So, for example, a VM with the name of AaronSweemer would have the directory /vmfs/volume/UUID/AaronSweemer. In this directory you will find all files specific and relevant to that VM. This is the default behavior as some (not all) of these files can be configured to reside elsewhere.
Here is a table of common files found on the VMFS file system.
| Extension | Usage |
| .dsk | VM disk file |
| .vmdk | VM disk file |
| .hlog | VMotion log file |
| .vswp | Virtual swap file |
| .vmss | VM suspend file |
| .vmtd | VM template disk file |
| .vmtx | VM Template configuration file |
| .REDO | Files used when VM is in REDO mode |
| .vmx | VM configuration file |
| .log | VM log file |
| .nvram | Nonvolatile RAM |
Journaling
From Wikipedia …
A journaling file system is a file system that logs changes to a journal (usually a circular log in a dedicated area) before committing them to the main file system. Such file systems are less likely to become corrupted in the event of power failure or system crash.
Explain the process used to align VMFS partitions
The following procedure was found in VMware Enterprise Administration Exam study guide 3.5 (page 5) and Advanced VMFS Configuration and Troubleshooting (slide 36).
Aligned partitions start at 128. If the Start value is 63 (the default), the partition is
not aligned. If you choose not to use the VI Client and create partitions with
vmkfstools, or if you want to align the default installation partition before use, take
the following steps to use fdisk to align a partition manually from the ESX Server
service console:
1. Enter fdisk /dev/sd<x> where <x> is the device suffix.
2. Determine if any VMware VMFS partitions already exist. VMware VMFS
partitions are identified by a partition system ID of fb. Type d to delete to
delete these partitions.
Note: This destroys all data currently residing on the VMware VMFS partitions you
delete.
3. Ensure you back up this data first if you need it.
4. Type n to create a new partition.
5. Type p to create a primary partition.
6. Type 1 to create partition No. 1.
Select the defaults to use the complete disk.
7. Type t to set the partition’s system ID.
8. Type fb to set the partition system ID to fb (VMware VMFS volume).
9. Type x to go into expert mode.
10. Type b to adjust the starting block number.
11. Type 1 to choose partition 1.
12. Type 128 to set it to 128 (the array’s stripe element size).
13. Type w to write label and partition information to disk.
Explain the use cases for round-robin load balancing
Multipathing is typically used for failover. Meaning, if one storage path becomes available the host can failover to an alternate path. However, multipathing can also be used in a round-robin fashion to achieve load balancing to achieve better utilization of the HBAs. There are a couple different configurable options that specify when an ESX server switches paths. From the Round-Robin Load Balancing technical note …
When to switch – Specify that the ESX Server host should attempt a path switch after a specified number of I/O blocks have been issued on a path or after a specified number of read or write commands have been issued on a path. If another path exists that meets the specified path policy for the target, the active path to the target is switched to the new path. The –custom-max-commands and –custom-max-blocks options specify when to switch.
Which target to use – Specify that the next path should be on the preferred target, the most recently used target, or any target. The –custom-target-policy option specifies which target to use.
Which HBA to use – Specify that the next path should be on the preferred HBA, the most recently used HBA, the HBA with the minimum outstanding I/O requests, or any HBA. The –custom-HBA-policy option specifies which HBA to use.
Skills and Abilities
Perform advanced multi-pathing configuration
- Configure multi-pathing policy
- Configure round-robin behavior using command-line tools
- Manage active and inactive paths
- Again, from the Round-Robin Load Balancing technical note …
Setting the Path Switching Policy
You can set the path?switching policy for failover and for load balancing by using the esxcfg-mpath command.You can set the path switching policy on a per?LUN basis by using the esxcfg-mpath command’s –policy custom option. If you specify –policy custom, you must also specify one of the custom policy options. Because the path switching policy is set on a per?LUN basis, you must always specify the LUN using the –lun option.
…
Notes
If you set the custom-max-blocks and custom-max-commands, options, the system attempts to switch paths as soon as one of the limits is reached.
If you set the target or the HBA policy to preferred, the system chooses the target or the HBA of the preferred path when possible. If a preferred policy is set on an active/passive SAN array, and the preferred target is not on the active SP (Storage Processor), the system does not select the preferred target but a target on the active SP.
Path switching is not performed if an outstanding SCSI reservation is on the target, or if a path failover is underway. Path switching is delayed until an I/O request is performed when no reservations or path failovers are pending.
Configure and use NPIV HBAs
<<I don’t have NPIV in my lab. Need to revisit this section>>
Manage VMFS file systems using command-line tools
The command line tool you’ll use for managing VMFS file systems in vmkfstools. It’s a very powerful tool and there are many options available, so I suggest you read the man page. The following examples (taken from the online documentation) are certainly not inclusive, just a quick sample of what the tool can do.
Example for Creating a VMFS File System
vmkfstools -C vmfs3 -b 1m -S my_vmfs /vmfs/devices/disks/vmhba1:3:0:1Example for Extending a VMFS-3 Volume
vmkfstools -Z /vmfs/devices/disks/vmhba0:1:2:1 /vmfs/devices/disks/vmhba1:3:0:1Upgrading a VMFS-2 to VMFS-3
-T –tovmfs3 -x –upgradetype [zeroedthick|eagerzeroedthick|thin]Example for Creating a Virtual Disk
vmkfstools -c 2048m /vmfs/volumes/myVMFS/rh6.2.vmdkExample for Cloning a Virtual Disk
vmkfstools -i /vmfs/volumes/templates/gold-master.vmdk /vmfs/volumes/myVMFS/myOS.vmdk
Configure NFS datastores using command-line tools
Assuming your NAS is configured properly, this is pretty easy. The following command will mount an NFS datastore on an ESX host …
esxcfg-nas –a –o 10.10.8.25 –s /nfs/share NAS
In this example, the –a adds a host with the IP address followed by the –o flag using the share configured after the –s flag. Upon successfully adding the datastore, the NFS mount will be found at /vmfs/volumes/NAS
The following command will remove the datastore
esxcfg-nas –d –o 10.10.8.25 NAS
Configure iSCSI hardware and software initiators using command-line tools
I don’t know if I’ve seen an official, formal example of how to do this (though I’m sure it exists somewhere). So, here’s how I do it …
Step 1: Add the portgroup to vSwitch0
esxcfg-vswitch –add-pg=VMkernel vSwitch0
Step 2: Add the IP to the VMkernel portgroup
esxcfg-vmknic -a -i 10.10.8.202 -n 255.255.255.0 VMkernel
Step 4: Enable iSCSI
esxcfg-swiscsi –e
Step 5: Add the target
vmkiscsi-tool -D -a 10.10.8.200 vmhba34
Step 6: Rescan the HBA
esxcfg-rescan vmhba34
That’s it for section 1.1 … time to go reformat my notes for section 1.2!
Eating the Dog Food (my Conversion to a Dedicated Virtual Desktop)
Jan 20th
I’m a big believer in “practice what you preach” and I try to do my best to live by this motto. Now, let me be the first to say that I also believe that by being human, almost by definition, makes me a hypocrite and I am certainly not saying that I’ve never failed at this. All I’m saying is that I try my best.
All day long, as a VMware Systems Engineer, I spend my time evangelizing the wonderful things that virtualization and VMware software can do for companies. I can do this because I’ve been working with VMware software almost since VMware began. And I’ve helped implement virtual infrastructures for many different clients, long before actually being employed at VMware. I even helped build VDI’s (Virtual Desktop Infrastructures) before VMware had a VDI product offering.
VDI is really starting to take off this year and most of my conversations these days are about giving corporate users a virtual desktop. And while I’ve helped build VDI solutions before, am I actually using a virtual desktop myself? No! Well technically I do, given that I run my corporate destkop in VMware Workstation on top of Ubuntu Linux. But this is not the same solution I’m preaching to my customers. A virtual desktop running locally is a very different beast than running a virtual desktop remotely on VI3 via a desktop broker. Normally, I would be able to rationalize this and say “I’m not a corporation that has the virtual infrastructure to run a virtual destktop.” But this is no longer true. I have a dedicated Internet connection and at least three good servers always running in my lab.
So, I’m about to start practicing what I preach and “eating my own dog food.” I’m going to begin a series of blog posts that document my conversion to a dedicated virtual desktop. I hope that this can also serve as inspiration, and possibly even a guide to a proof of concept, for those who are interested in virtual desktops. I’m sure this blog series evolve as I progress, but I envision the upcoming posts will look something like this …
- Planning the environment
- Setting up the network and dedicated remote access
- Getting the virtual infrastructure set up correctly
- Setting up VMware View (the broker)
- Preparing the desktops
- ThinApp’ing my applications
- Troubleshooting, tweaking and optimizing
Again, as I progress, this could all change. But for now, this is the blog series I’ll begin in the next day or two and hope to complete over the next few weeks. I hope you join me. If you plan to follow along and, better yet, get involved, please sign up for a Disqus account. It’s the system I use for commenting and discussions.
By the way, do you know who originally coined the term “eating our own dog food?” VMware’s CEO, Paul Maritz. But he didn’t coin the term while on duty at VMware. From Wikipedia …
In 1988, Microsoft manager Paul Maritz sent Brian Valentine, test manager for Microsoft LAN Manager, an email titled “Eating our own Dogfood” challenging him to increase internal usage of the product; from there, the usage of the term spread through Microsoft, as chronicled in the book Inside Out: Microsoft—In Our Own Words (ISBN 0446527394)
The irony is so thick, you can cut it with a knife!!
How Much Storage Will You Save With View Composer?
Jan 5th
It depends But to help you better guesstimate how much storage you’ll save with View Composer, let’s take a look at the actual storage savings in my home lab. Keep in mind that VMware View isn’t just something I’m playing around with. There are currenlty 5 other VMware Systems Engineers using my lab on a regular basis. And I use VMware View to give each lab user a dedicated, persistent desktop VM which they connect to remotely via the View Client. And from their dedicated lab desktop, they have full access to the lab environment I’ve built. So, while my home lab environment may be small and by no means a production enviornment, I’m still using VMware View as a “production” application, so to speak. It’s the one application in my lab that needs to be up, and it gets heavily used.
Let’s first look at how much storage my virtual desktop infrastructure would require without the use of View Composer. As I said, each lab user has a dedicated desktop, which has a 12G partition for OS and a 4G partition for user data. Also, each desktop has 1G of memory with zero memory reservation, which transltates into a 1G swap file. That’s a total of 17G per user. Right now there are 5 other remote users, plus I have a desktop for myself. And I have 2 desktops on standby for new users (so they don’t have to wait for a new VM to be built upon first login). So, that brings the total virtual desktops to 8. Therefore the total storage requirement for virtual desktops in my lab without View Composer would be 136G (17G * 8 desktops).
Now let’s look and see how much space is actually being used in my lab. Here’s a cut and paste showing the disk usage of the volume holding the desktop VMs …
[root@cincylab-esx1 cincylab-vol1]# du -sh *
3.1G labuser-01
2.9G labuser-02
3.6G labuser-03
2.6G labuser-04
3.1G labuser-05
1.9G labuser-06
1.5G labuser-07
1.5G labuser-08
2.0G replica-774c678b-e6b5-495a-b8ff-
448K source-lc-8ae5400e-4af3-4a09-8d8
13G parent_desktop
[root@cincylab-esx1 cincylab-vol1]#
The grand total here is 33.2G, which is about a 75% reduction in storage. Not bad. The storage reduction is achieved through two technologies, Linked Clones and Thin Disks.
Linked Clones
In my environment, the virtual disks for the desktops are located in the labuser-0* directories (this is done automatically for me during the deployment process). These virtual disks are not thick, monolithic disks like they used to be in the previous version. Rather, they are delta disks, which only store data differences between the desktop OS and the OS of the parent VM. In the cut and paste above, notice the 13G parent_desktop? That’s my starting point and contains my golden image. That direcotry also contains a snapshot, which I took when I was ready to being deploying desktops. The replica-774c678b-e6b5-495a-b8ff- VM is derived from this snapshot and ultimately serves as the parent VM (for now, this can change over time) of the labuser-0* desktops. Linked Clones have other benefits too, especially around patching and updateing. But that’s a topic for a later post.
Thin Disks
Remember how I said the users need a 17G? (12G for OS, 4G for user data and 1G for swap) But did you notice that the parent_desktop directory is only 13G in size (12G OS plus 1G swap)? From the administration guide “Thin provisioned disks (thin disks) are used by the linked clones to store user data, and are not linked to the Parent VM.” So I didn’t need to include the user partition in the parent VM. The user partition is handled for me when a desktop is deployed and included with each of the user desktops. User data disks are persistent and they are thin, meaning they occupy no more space than the data requires. In my environment, looking at the virtual disks for labuser-03:
[root@cincylab-esx1 labuser-03]# ls -lah
total 3.6G
drwxr-xr-x 1 root root 1.8K Dec 27 09:23 .
drwxr-xr-t 1 root root 3.7K Jan 5 06:10 ..
-rw——- 1 root root 1.0G Dec 27 09:23 labuser-03-a2263868.vswp
-rw——- 1 root root 8.5K Dec 27 09:23 labuser-03.nvram
-rw——- 1 root root 4.0G Jan 5 08:07 labuser-03-vdm-user-disk-D-flat.vmdk
-rw——- 1 root root 443 Dec 27 09:24 labuser-03-vdm-user-disk-D.vmdk
-rw——- 1 root root 75 Dec 27 09:21 labuser-03.vmsd
-rwxr-xr-x 1 root root 3.9K Dec 31 06:58 labuser-03.vmx
-rw——- 1 root root 265 Dec 31 06:58 labuser-03.vmxf
-rw——- 1 root root 2.5G Jan 5 08:07 replica-774c678b-e6b5-495a-b8ff–cl1-delta.vmdk
-rw——- 1 root root 379 Dec 27 09:24 replica-774c678b-e6b5-495a-b8ff–cl1.vmdk
-rw-r–r– 1 root root 62K Dec 27 09:21 vmware-1.log
-rw-r–r– 1 root root 36K Jan 5 06:21 vmware.log
[root@cincylab-esx1 labuser-03]#
Look at the size of labuser-03-vdm-user-disk-D-flat.vmdk, see how the file system “thinks” it’s 4.0G? It’s really not. A closer look at disk usage reaveals:
[root@cincylab-esx1 labuser-03]# du -sh *
1.0G labuser-03-a2263868.vswp
64K labuser-03.nvram
43M labuser-03-vdm-user-disk-D-flat.vmdk
64K labuser-03-vdm-user-disk-D.vmdk
64K labuser-03.vmsd
64K labuser-03.vmx
64K labuser-03.vmxf
2.6G replica-774c678b-e6b5-495a-b8ff–cl1-delta.vmdk
64K replica-774c678b-e6b5-495a-b8ff–cl1.vmdk
64K vmware-1.log
64K vmware.log
[root@cincylab-esx1 labuser-03]#
It’s actually only taking up 43M, not 4G. And a quick look inside the VM confirms it too “thinks” it has 4G:
So that’s my real world example of how much storage I’m saving with View Composer. How much will you save? Again, it depends There are things that I haven’t discussed in this post (e.g. desktop refresh, desktop recomposition and desktop rebalance), which will also affect storage savings. Also, using ThinApp’d applications that live on a file share (as opposed to putting them in the OS) could have a big positive impact on storage savings. In my lab, with only linked clones and thin disks, I’m getting a 75% reduction. With a little more effort and planning on my part, I’m confident I could achieve 85% or better.