VCDX Admin Exam Notes — Section 1.1

I finally got a chance to sit down and reformat some of my notes for the VCDX Admin Exam.  Below are my notes for Section 1.1 of the VMware Enterprise Administration Exam Blueprint v3.5.  Everything in Blue is a direct cut and past from the exam blueprint.

Oh, and thanks to the Disqus comment from VirtualizationTeam (Blog), letting me know that Peter van den Bosch has a more recent version of his VMware Enterprise Administration Exam Study Guide 3.5


Section 1 – Storage

Objective 1.1 – Create and Administer VMFS datastores using advanced techniques.


Describe how to identify iSCSI, Fibre channel, SATA and NFS configurations using CLI commands and log entries

Here are a few command line examples that I believe would work well …

1)  esxcfg-mpath –l
This command produces the following output on my server:


[root@cincylab-esx3 root]# esxcfg-mpath -l

Disk vmhba0:0:0 /dev/sdb (152627MB) has 1 paths and policy of Fixed

Local 0:31.2 vmhba0:0:0 On active preferred

Disk vmhba32:0:0 /dev/sda (152627MB) has 1 paths and policy of Fixed

Local 0:31.2 vmhba32:0:0 On active preferred

Disk vmhba35:0:0 /dev/sdc (923172MB) has 1 paths and policy of Fixed

iScsi sw<-> vmhba35:0:0 On active preferred

2)  esxcfg-info –s

The –s flag will narrow the scope of the output to just storage and disk related info.  But even with the narrowed scope, this command produces way too much output to be displayed here.  You’ll likely want to pipe the output into grep, or at a minimum to a more/less to get what you’re looking for.

3)  cat /var/log/vmkernel | grep vmhba | tail –10

This will search the vmkernel log file and display the last 10 lines containing the text vmhba.  If you want more (or fewer lines) change the –10 to whatever suits your needs.

If found this one particularly useful when you’ve enabled the software iSCSI initiator at the command line, but don’t know yet number has been assigned to the vmhba (e.g. vmhba35). 

4)  esxcfg-vmhbadevs –m  and  ls –lah /vmfs/volumes

The command esxcfg-vmhbadevs –m will show the mapping between vmhba numbers, device files and their UUIDs.  If you’d like a quick and easy way to see what UUIDs are mapped to their human readable name, you can follow that up with a ls –lah /vmfs/volumes.  The two commands back to back produce the following output on my server:

[root@cincylab-esx3 root]# esxcfg-vmhbadevs -m
vmhba35:0:0:1   /dev/sdc1                        4986310d-6525e5e6-ebbd-00237d0681e7
vmhba0:0:0:3    /dev/sdb3                        49e115fb-3e22358c-c10a-00237d0681e7
vmhba32:0:0:1   /dev/sda1                        4985c53e-e7b1904f-5042-00237d0681e7


[root@cincylab-esx3 root]# ls -lah /vmfs/volumes/
total 10M
drwxr-xr-x    1 root     root          512 Apr 20 23:07 .
drwxrwxrwt    1 root     root          512 Apr 11 18:12 ..
drwxr-xr-t    1 root     root         1.2K Feb  1 21:34 4985c53e-e7b1904f-5042-00237d0681e7
drwxr-xr-t    1 root     root         3.7K Apr 14 14:49 4986310d-6525e5e6-ebbd-00237d0681e7
drwxr-xr-t    1 root     root          980 Apr 11 18:13 49e115fb-3e22358c-c10a-00237d0681e7
lrwxr-xr-x    1 root     root           35 Apr 20 23:07 cincylab-esx3:storage1 -> 4985c53e-e7b1904f-5042-00237d0681e7
lrwxr-xr-x    1 root     root           35 Apr 20 23:07 cincylab-esx3:storage2 -> 49e115fb-3e22358c-c10a-00237d0681e7
lrwxr-xr-x    1 root     root           35 Apr 20 23:07 vol1 -> 4986310d-6525e5e6-ebbd-00237d0681e7

5)  vmkiscsi-ls

This one only applies to iSCSI storage, of course, and produces the following output on my server:

[root@cincylab-esx3 root]# vmkiscsi-ls

        SFNet iSCSI Driver Version … 3.6.3 (27-Jun-2005 )
TARGET NAME             :
TARGET ALIAS            :
HOST NO                 : 4
BUS NO                  : 0
TARGET ID               : 0
SESSION STATUS          : ESTABLISHED AT Sun Apr 12 11:35:09 2009
NO. OF PORTALS          : 1
PORTAL ADDRESS 1        :,1
SESSION ID              : ISID 00023d000001 TSIH 1400

Describe the VMFS file system

There are many subsections here and before digging into each one, check out the following three links …

The simple definition of Metadata is “data about data.”  All file systems handle metadata differently.  VMFS uses metadata, stored in a special area of each volume, to manage all the files, directories (in VMFS-3 only), and attributes about the volume.  VMFS is a clustered file system, meaning more than one ESX server can access the same file system at the same time.  Therefore an update to the metadata requires locking of the LUN using a SCSI reservation.

Multi-access and locking
The following was taking from Advanced VMFS Configuration and Troubleshooting.

  Distributed Lock handling by VMFS3

  • Done in-band
  • Hosts mount a VMFS3 volume
  • Hosts’ ids posted to heartbeat region
  • Heartbeat records are updated at regular intervals by hosts
  • Host X locks a file, the lock is associated with its ID
  • If host X dies or loses access to volume the file lock is stale
  • Host Z attempts to lock the same file which is locked
  • Host Z check the heartbeat record of Host X (~5 times)
  • If host X heartbeat record is not updated, Host Z will age the lock
  • All other hosts yield to host Z and not attempt to lock the file
  • Lock is broken and Host Z acquires the lock
  • Journal is replayed by Host Z

Extents are logical extensions of a file system.  They are typically used to grow a volume beyond the VMFS size limitations.  Essentially, an extent is the “joining” of two or more volumes into a single, logical VMFS volume.

Tree structure and files
The vmfs partition is mounted to the directory with the corresponding UUID found in /vmfs/volumes.  The human readable name of the volume is merely a symbolic link to that directory.  By default, all VMs are given a directory at the root of the partition.  So, for example, a VM with the name of AaronSweemer would have the directory /vmfs/volume/UUID/AaronSweemer.  In this directory you will find all files specific and relevant to that VM.  This is the default behavior as some (not all) of these files can be configured to reside elsewhere. 

Here is a table of common files found on the VMFS file system. 

Extension Usage
.dsk VM disk file
.vmdk VM disk file
.hlog VMotion log file
.vswp Virtual swap file
.vmss VM suspend file
.vmtd VM template disk file
.vmtx VM Template configuration file
.REDO Files used when VM is in REDO mode
.vmx VM configuration file
.log VM log file
.nvram Nonvolatile RAM

From Wikipedia …

A journaling file system is a file system that logs changes to a journal (usually a circular log in a dedicated area) before committing them to the main file system. Such file systems are less likely to become corrupted in the event of power failure or system crash.

Explain the process used to align VMFS partitions 

The following procedure was found in VMware Enterprise Administration Exam study guide 3.5 (page 5) and Advanced VMFS Configuration and Troubleshooting (slide 36).

Aligned partitions start at 128. If the Start value is 63 (the default), the partition is
not aligned. If you choose not to use the VI Client and create partitions with
vmkfstools, or if you want to align the default installation partition before use, take
the following steps to use fdisk to align a partition manually from the ESX Server
service console:
1. Enter fdisk /dev/sd<x> where <x> is the device suffix.
2. Determine if any VMware VMFS partitions already exist. VMware VMFS
partitions are identified by a partition system ID of fb. Type d to delete to
delete these partitions.
Note: This destroys all data currently residing on the VMware VMFS partitions you
3. Ensure you back up this data first if you need it.
4. Type n to create a new partition.
5. Type p to create a primary partition.
6. Type 1 to create partition No. 1.
Select the defaults to use the complete disk.
7. Type t to set the partition’s system ID.
8. Type fb to set the partition system ID to fb (VMware VMFS volume).
9. Type x to go into expert mode.
10. Type b to adjust the starting block number.
11. Type 1 to choose partition 1.
12. Type 128 to set it to 128 (the array’s stripe element size).
13. Type w to write label and partition information to disk.

Explain the use cases for round-robin load balancing

Multipathing is typically used for failover.  Meaning, if one storage path becomes available the host can failover to an alternate path.  However, multipathing can also be used in a round-robin fashion to achieve load balancing to achieve better utilization of the HBAs.  There are a couple different configurable options that specify when an ESX server switches paths.  From the Round-Robin Load Balancing technical note …

When to switch – Specify that the ESX Server host should attempt a path switch after a specified number of I/O blocks have been issued on a path or after a specified number of read or write commands have been issued on a path. If another path exists that meets the specified path policy for the target, the active path to the target is switched to the new path. The –custom-max-commands and –custom-max-blocks options specify when to switch.

Which target to use – Specify that the next path should be on the preferred target, the most recently used target, or any target. The –custom-target-policy option specifies which target to use.

Which HBA to use – Specify that the next path should be on the preferred HBA, the most recently used HBA, the HBA with the minimum outstanding I/O requests, or any HBA. The –custom-HBA-policy option specifies which HBA to use.

Skills and Abilities

Perform advanced multi-pathing configuration

  • Configure multi-pathing policy
  • Configure round-robin behavior using command-line tools
  • Manage active and inactive paths

Setting the Path Switching Policy
You can set the path?switching policy for failover and for load balancing by using the esxcfg-mpath command.

You can set the path switching policy on a per?LUN basis by using the esxcfg-mpath command’s –policy custom option. If you specify –policy custom, you must also specify one of the custom policy options. Because the path switching policy is set on a per?LUN basis, you must always specify the LUN using the –lun option.


If you set the custom-max-blocks and custom-max-commands, options, the system attempts to switch paths as soon as one of the limits is reached.

If you set the target or the HBA policy to preferred, the system chooses the target or the HBA of the preferred path when possible. If a preferred policy is set on an active/passive SAN array, and the preferred target is not on the active SP (Storage Processor), the system does not select the preferred target but a target on the active SP.

Path switching is not performed if an outstanding SCSI reservation is on the target, or if a path failover is underway. Path switching is delayed until an I/O request is performed when no reservations or path failovers are pending.


 Configure and use NPIV HBAs

<<I don’t have NPIV in my lab.  Need to revisit this section>>

Manage VMFS file systems using command-line tools

The command line tool you’ll use for managing VMFS file systems in vmkfstools.  It’s a very powerful tool and there are many options available, so I suggest you read the man page.  The following examples (taken from the online documentation) are certainly not inclusive, just a quick sample of what the tool can do. 

Example for Creating a VMFS File System
vmkfstools -C vmfs3 -b 1m -S my_vmfs /vmfs/devices/disks/vmhba1:3:0:1

Example for Extending a VMFS-3 Volume
vmkfstools -Z /vmfs/devices/disks/vmhba0:1:2:1 /vmfs/devices/disks/vmhba1:3:0:1

Upgrading a VMFS-2 to VMFS-3
-T –tovmfs3 -x –upgradetype [zeroedthick|eagerzeroedthick|thin]

Example for Creating a Virtual Disk
vmkfstools -c 2048m /vmfs/volumes/myVMFS/rh6.2.vmdk

Example for Cloning a Virtual Disk
vmkfstools -i /vmfs/volumes/templates/gold-master.vmdk /vmfs/volumes/myVMFS/myOS.vmdk


 Configure NFS datastores using command-line tools

Assuming your NAS is configured properly, this is pretty easy.  The following command will mount an NFS datastore on an ESX host …

esxcfg-nas –a –o –s /nfs/share NAS

In this example, the –a adds a host with the IP address followed by the –o flag using the share configured after the –s flag.  Upon successfully adding the datastore, the NFS mount will be found at /vmfs/volumes/NAS

The following command will remove the datastore

esxcfg-nas –d –o NAS

Configure iSCSI hardware and software initiators using command-line tools

I don’t know if I’ve seen an official, formal example of how to do this (though I’m sure it exists somewhere).  So, here’s how I do it …

Step 1:  Add the portgroup to vSwitch0 
esxcfg-vswitch –add-pg=VMkernel vSwitch0

Step 2:  Add the IP to the VMkernel portgroup
esxcfg-vmknic -a -i -n VMkernel

Step 4:  Enable iSCSI
esxcfg-swiscsi –e

Step 5:  Add the target
vmkiscsi-tool -D -a vmhba34

Step 6:  Rescan the HBA
esxcfg-rescan vmhba34


That’s it for section 1.1 … time to go reformat my notes for section 1.2!

Troubleshooting ESX

I was at the Louisville VMUG on Friday talking about Troubleshooting ESX.  In my preparation for the event, I was looking for a good PowerPoint presentation I could reuse and I stumbled across a sweet little gem of a document.  Dudley Smith, a VMware Technical Account Manager (TAM) out of Virginia, created a cool one page Mind Map for Troubleshooting ESX.  Does it address every potential issue you’ll come across?  No, of course not.  But it’s a heck of a good place to start.  One look at his Mind Map and I thought to myself, “that would be a great thing to have printed out and hanging over every VMware admin’s desk.”

Well, long story short, I snagged it and threw it up on the big screen behind me as I was presenting.  During the presentation (and many times since the presentation) I had many requests to post the PDF for download. 

But since I couldn’t just start passing out someone else’s work as my own, I sent Dudley a quick email asking for permission to distribute.  He responded by saying, “Sure, publish away!  You might enjoy this too… ”  Attached was another one page document that visually shows the TCP/UDP ports leveraged in VI3.5.  Nice!  Again, another great document to have printed out and hanging over your desk, IMHO.

So, courtesy of the author, Dudley Smith, here are two documents that I would recommend you add to your tool belt.  (click the images to download the PDFs)







If you like them, leave a comment for Dudley.

Some of my Thoughts on Cloud Computing


Everywhere I turn, everybody’s talking about cloud computing.  And I agree with Mike DiPetrillo, very few people understand what the cloud is today and what it could and/or should be tomorrow.  I’ve kept silent on the topic of cloud computing on this blog until now, mainly because I prefer to know what I’m talking about before I put something out there for the world to see :)  But now that I believe I’ve got a good grasp on it, I’d like to share a few of my thoughts …


What’s in a name?

First of all, I don’t like the name "cloud."  I think it’s a stupid name.  Let me explain why.  I was with a customer the other day talking about the future of cloud computing and he said, "man, what a horrible name, it just sounds like the most insecure, undefined, unmanageable place … why would I ever want to put my apps in a cloud?"  I couldn’t agree more.  For years we’ve been preaching about putting applications in data centers, and the importance of securing data with things like firewalls and intrusion detection.  You don’t even have to know what a firewall is, what it does or how it works, but the name just sounds safe.  And in my opinion, referring to the next generation of computing as the "cloud" would be like if someone called the first firewall "come on in" and the industry adopted it. 

To me (and many of the customer I’ve spoken to), the word cloud conjures up images of dancing ferries, unicorns and other mythical creatures all prancing around in some fluffy place somewhere northeast of never-never-land.  But alas, cloud is the name the industry seems to have settled on.  And since, at this point, I don’t have a better name to offer, I too will refer to it as the cloud.  But for the record, I think the name should be something strong and manly, like Spike, or Butch, or Krull the Warrior King! 

By the way, I like the name vShield Zones, a new VMware offering that will logically partition a cloud.  I think a name like this conveys a much better image about where an organization’s apps and data “live.” 


What is the cloud?  An explanation for the business owner (IT people, you may want to stop reading now)

Most people still think about IT as servers and networks and storage, all powered by a bunch of computer geeks that hibernate in a data center or crawl around under desks when a computer breaks.  Actually, they probably still think about IT this way because this is by and large the reality for almost all organizations. 

But let me ask you this … why are thinking about IT at all?  Unless you’re in the IT / Hosting business, shouldn’t you be thinking about, um, your business?  You don’t think about electricity or the plumbing do you?  No, you don’t.  Unless of course, the electricity goes out or the toilets backup … then you can’t stop thinking about them! :)  But I’m pretty sure an advertising company, for example, doesn’t have an electrician or a plumber on staff. 

When cloud computing is fully realized, IT should be very much the same thing.  It is a tool that should serve you and your business, not the other way around.  Now, it’s not like servers and networks and storage and IT geeks like myself will cease to exist.  No, we’ll still be here, but you won’t think or care about us anymore.  *sniffle*

To understand the cloud, you need to STOP thinking about the plumbing behind the applications (i.e. servers / networks / storage), and you need to START thinking about what matters most, the applications and data you need to run your business. 

When the cloud is fully realized, your applications will be always on, extremely reliable, accessible anytime and from anywhere, and they will “live” in a cloud.  Now that cloud might be external to your organization, or it might be an internal cloud, built on your existing infrastructure.  Either way, you’ll be able to self-provision new applications with a few clicks of a mouse and pay only for what you use. 

Sounds pretty good, yes?  Don’t go beating up your IT department just yet.  While many pieces of cloud computing are in place, the cloud is still forming.  Standards are being hammered out, committees are being formed, and it seems like everyone has a SOAP box, apparently even me :)  (yes, I meant to capitalize SOAP, it’s a little joke for the developers)


I have more thoughts. More to come.

Are you studying for the VCDX Enterprise Admin Exam?

Me too.  Actually, I’ve been studying for a few weeks now.  A while back, a friend of mine and fellow VMware SE (and SRM super freak), Michael White, turned me on to Evernote, an awesome tool for capturing my web research and lab notes (among other other things).  So, as I’ve been studying for the exam, I’ve been saving everything in Evernote, which I have installed on my iPhone and my XP desktop.  (Now, if only they made a version for Linux, I’d be all set.  *sigh*  Yet another company with a great app that chooses to ignore the Linux community).

Anyway, now I just need to go back and do some simple formatting and VOILA!, I’ve got quite a few ready-made blog posts.  Which is handy, given my recent commitment to a post a day:)

So if you too are studying for the exam (or planning to) and looking for study material, feel free to check back here from time to time over the next few weeks.  In addition, you might want to check out the following list, which contains everything I’m using to prepare.

Also, send me an email if you’d like to be part of a weekly online VCDX study group I’m trying to put together.