Author Archive

Welcome back, I hope you found our first newsletter helpful in some way shape or form.  The newsletter seems to be getting larger and larger which is a great thing.  It might soon start to qualify as a magazine rather than a newsletter.

We got some good feedback so we are going to keep going with this for a while.  Please let us know via the comments section if you are enjoying it, would like to see different content,  or just want to say hello.

-Scott

When talking about VMware virtualization bottlenecks, 9 out of 10 customers answer their number one bottleneck is memory. Notice how I said bottleneck, not problem. This relates to capacity planning or trying to understand and right size the environment so you can gauge when you need to order more physical infrastructure. Their number one problem is storage, which is quite a different story altogether and I won’t be covering storage in this article (this time). Since memory is such a common point of discussion with my customers, I thought I would dig a little deeper on this topic and share some information around utilization and what it all means.

My customers typically track their utilization in the most common area of vSphere that one might expect to find this information, the DRS Resource Distribution graph at the cluster level.

From the image displayed above, one might think that I am close to memory capacity and I should look at ordering more hardware for my cluster. While in a general sense that might not be a bad idea to begin planning for growth, but let’s take a closer look at what we are seeing.  Notice the blue informational icon and how it’s telling us that the displayed information is based on memory consumption. Let’s do a mouse over on the chart that’s being displayed to get some more granular information and what this means.

You can see in the above image that my Virtual Center VM is “Consuming” ~4GB of memory, but in all reality the active memory being used is sitting at ~700MB. DRS entitlement is a measurement that calculates what the load or demand is on the vSphere host/cluster over time, and then projects an average entitlement number for planning purposes. You can use the DRS entitlement numbers as a general planning/forecasting number, but to be honest you still have some capacity within the cluster.

Now I wouldn’t be doing my job if I didn’t make you aware of an easier way to track this information by using software rather than brain power. For those of you that haven’t seen Capacity IQ yet, I would highly encourage you to evaluate the product. Capacity IQ was built for this specific reason, to help you understand when you will need to start thinking about more hardware. It can also help you run your environment more efficiently. There are some great reports that help you identify which virtual machines are not using the resources that were allocated to them.  Take them back!

Coming from a VMware system engineer end user position, I can tell you that as your environment begins to grow, capacity management and planning becomes critical. I evaluated Capacity IQ when I was still on the customer side, and did a write up if you are interested in my thoughts on the product.

At one of our last internal central VMware meetings, a few of us had a similar idea to pull together a newsletter for our customers.  Some of us were already doing this to a degree, but collectively we agreed that one source of information would be better than many.  Several VMware SE’s and Specialists have helped pull this together so I wanted to thank everyone for their hard work. 

There is so much great content that is published to the web and sometimes passed internally, we wanted to consolidate this information to one common distributable location.  99% of the content is not really specific to the U.S. Central region other than the local events, so I think that many people will be able to benefit from out efforts.  The goal will be to test the waters and see if it’s something that people like and want to see continued.

As always we are looking for feedback, if you think this is something that should continue, let us know!  If you feel it’s lacking or could be improved in some aspect, we are also looking for your opinion to help shape it.

Enjoy!

-Scott

I spend a majority of my time talking with VMware customers trying to help understand their needs and how we can help them with some of their internal IT business challenges. I would say a majority of the problems and issues discussed are typically based around internal politics and the IT landscape changing, but their second largest concern is around performance and growth (capacity). VMware and virtualization in general, has been such a powerful driver for many organizations over the past several years. It has allowed IT organizations to run more efficiently, save capital expenditure costs, and ease administrative overhead, all in the midst of an economic downturn.

Capital expenditure costs savings are great, and very visible to the organization from a high level, but VMware needs to help customers with the next step. Now that we are moving so much of our infrastructure to a more elastic and flexible solution, (vSphere) we need to provide tools to help you manage this infrastructure because the same methodologies no longer apply as they did in the physical world. The more we can help automate and manage your virtual infrastructure; we can now begin to help with step two which is save your IT organization operational costs. A recent Gartner study determined that the average cost for a Windows server is $10,200 per year. Of that expense ~ 70% is on OPEX. Gartner also estimates that with automation and management that up to 80% of the OPEX could be saved.

VMware has made several acquisitions around management and automation, and I wanted to focus on one which was recently announced. VMware Operations is a “new” product that was released this past week. It’s actually not all that new of a product but a re-branding of a key acquisition announced at VMworld 2010. Integrien was an analytics and statistical based software company with a focus on management software. Notice that their primary focus was not management but analytics, a completely different approach to several other software companies out there trying to get to the same end result.

Rather than simply creating metrics to monitor and then setting thresholds on those metrics, Integrien will actually analyze the information that it’s gathering and understand when there is an actual problem. One of the coolest features about the full blown enterprise version is you can feed multiple data sources into the analytics engine. The more data it gets, the more accurate it’s able to predict when a problem is likely to occur.

This isn’t just your standard run of the mill monitoring software.

Those of you that have experience with enterprise monitoring software will know that it take a lot of effort to get these systems up and “fine tuned”. It takes a tremendous effort to begin sifting through all of the white noise alerts that come in and then adjust the threshold alerts to something that is tangible so it becomes useable data. VMware Operations removes that manual effort by dropping in an intelligent analytical engine that can understand what’s really going on behind the scenes.

Here are the different versions of the product, and how each version differs. I would suggest pulling down the virtual appliance and check out how awesome this product is. If you don’t feel like going to the effort, check out this video, it gives a great walk through of vCenter Operations and explains a lot of the same concepts I just wrote about.

There are a lot of the customers I cover in my region that are really starting to see the value in VMware’s management tools.  As virtual machines now outnumber physical machines, customers need some tools to help report against their existing infrastructure as well as predict and prepare for future virtual machine workloads.  One of my favorite VMware tools that I liked when I was on the customer side was a product called Capacity IQ.  I wrote up a blog post that I think people found useful that was basically an overview of the benefits of the product.  You can check that post out here.  I tell most of my customers about it, because it’s simple to setup (virtual appliance) and it gives you loads of great information about your existing infrastructure.

One of my customers that is moving forward with a CapIQ implementation e-mailed me about what types of storage metrics are available from the product.  I was happy to inform him that Capacity IQ 1.5 was just released and provides some great storage statistics that can now be reported against.  Much to my dismay, he told me that he wasn’t seeing the storage report data, the metrics were all blank.

Here are the requirements to get the reports to produce storage related information:

You  need the vCenter management webservices running for CapIQ to collect some of the storage metrics. The storage IO metrics require you to have ESX 4.1.  When you use ESX 4.0 or earlier hosts, the following metrics appear with dashes (–) and affect the Dashboard, the Datastores – List view, the Virtual Machine Capacity – Summary view, and the Virtual Machine Capacity Usage – Trend view:

* Disk I/O read/write
* Disk I/O reads/writes per second
* Disk I/O read/write latency
* VM Disk I/O read/write latency

-Scott

Introduction

VMware recently announced the general availability of a Zimbra virtual appliance that VMware customers can simply import into their existing infrastructure and get “e-mail in box”.  This is a great concept for administrators because the operating system is pre-configured and purposefully built for the application that is packaged with it.  The virtual appliance will import into the virtual center management console and will have the standard "OVF” (Open Virtualization Format) file extension for those that are new to virtual appliances.

I am no e-mail administrator, so I wanted to see how easy setting up the Zimbra virtual appliance would be and provide some instructions for those out there that are looking to test out Zimbra.

Get the Bits!

First things first, go out and grab the download of the Zimbra virtual appliance by clicking on the icon below.  Yes you will need to register to download the bits…

Import the Virtual Appliance

There are two methods of importing a virtual appliance, you can enter the url, which is supplied by the Zimbra website once you register, or you can download the appliance locally and import it locally.  I grabbed the full download in case I hosed something up I would have a copy of the ovf locally so I could start over from scratch.  I guess a snapshot would work as well, so it’s up to you how you would like to proceed here.  Below is a screenshot of the import:

Configure the Zimbra Virtual Appliance

The Zimbra virtual appliance is pre-configured to ask you the basic configuration parameters you will need to get the appliance up and running.  You can see below are the questions that you will to answer, pretty common stuff if you a IT administrator.  Make sure you use the FQDN for the hostname.

Power it up

Now that you have configured your basic system information you can now power up your new virtual appliance.  You can see below that it will automatically configure the appliance based off your information you have previously populated.  Very nice for a hands off approach and a streamlined installation process.

Finish it off

Now that you are powered up and on the network, you can login to the administration console to finish your configuration.  Point your web browser to https://<hostname>:5480  The administration console will be the place where you can create user accounts, configure licensing information, pull diagnostic data for troubleshooting and update the virtual appliance itself.

DNS is a big component of e-mail.  If you are doing split DNS or Dynamic DNS, I suggest  to reference this link to assist your efforts.  I am using a dynamic DNS service at home along with split DNS, so I had to go and update my host entry with a MX record so the world new where to route my e-mail traffic.  Once that was done correctly I was up and running and able to send/receive e-mails with no problem.

Licensing

The last thing you will want to to is license your installation, the nice folks over at Zimbra will give you a 10 user license free of charge.  Click the link below to go license your configuration or view some sample pricing on what a fully licensed configuration would look like.  Enjoy!

-Scott

It’s hard to believe that another year has flown by and Eric Siebert’s voting for the top Virtualization blogging contest is upon us once again!  If you enjoy the content that you read from Virtual Insanity, I encourage you to give back to the community and vote for us!

http://vote.vsphere-land.com/

What other site discusses great technical VMware content ranging from core ESX, Spring Source, Linchpin’s and even open’s its doors to great guest bloggers??  Thanks for reading.

-Scott

Introduction

I was talking with a local customer the other day that was inquiring about the differences between Microsoft virtualization (Hyper-V) and VMware virtualization solutions.  This customer was hung up on putting the two vendors into a cage match and making them go at each other to see who won.  I used to work in the end user IT environment, and know people that think this is a smart approach.  Competition is great for the end user environment because it drives innovation and keeps costs in check.   But let’s take a look at the technology rather than the cage and understand what’s under the hood.  Just to set the record straight this is not a slam Microsoft Hyper-V blog post.  I don’t get wrapped up into the battle of the hypervisor conversations, if you want to go with a competitive solution have at it.  We will be talking down the road again eventually. 

This individual understood the basics of virtualization but wasn’t that clear on how the hypervisor worked in conjunction with the hardware.  I point my customers to quality blog sites to answer questions or if they need instructions on how to configure or stand something up.  I figured I would do a write-up to help shed some light on how virtual machines work, how they are handled by the hypervisor and how they are lean and mean.

Hypervisor types

First let’s set the record straight by saying that VMware, Microsoft or Citrix did not come up with the concept of virtualization.  The idea of abstraction has been around for over 50 years, and was first mastered by the smart people over at IBM on some old iron in the late 60’s.  VMware did launch the first x86 based virtualization software in May of 1999, which since has changed the open systems world greatly.

Currently there are two types of hypervisors:

Type 1 – A native or bare metal hypervisors run directly on the host’s hardware to manage and monitor the guest operating systems.  Because it has direct access to the hardware resources and doesn’t go through an operating system, it runs more efficiently than a hosted model or Type 2.

Type 2 – A hosted hypervisor that runs within a conventional operating system.  This hypervisor does not have direct access to the hardware thus traditionally has more overhead that a Type 1 hypervisor.

Examples of Type 1 hypervisors would include, VMware ESX, VMware ESXi, Microsoft Hyper-V, Citrix Xen Server.  Examples of a Type 2 hypervisor would be, VMware workstation, VMware server, Microsoft Virtual Server.  The Type 1 hypervisors run more efficiently as they are designed specifically to handle virtual workloads.  They also don’t have a host operating system to have to share, schedule and contend with resources for.  The

The VMware Architecture

Two of the most important components of the VMware Type 1 hypervisor are the VMkernel and the VMM.  The VMkernel is the actual VMware ESX hypervisor product that we all know and love.  It is responsible for interacting with the physical server hardware that you install vSphere onto.  Sounds pretty simple in concept right?  It’s not.  I disagree with people that think the hypervisor is a commodity technology because there are some very special things that VMware does differently.  VMware takes on an approach unlike other virtualization vendors in the marketplace implements a hardware compatibility list (HCL)  to ensure you will be running a supported configuration.  That means when you install the product, VMware has already QA’d the configuration and is a eliminating the guess work of a supported stable environment. 

The VMkernel doesn’t actually run the virtual machines, it invokes yet another layer of protection called the virtual machine monitor.  This “Thin candy shell” (Tommy Boy reference) is the special sauce that takes various communications from the VMkernel and translates them to the actual virtual machine, and visa versa.  I put together a diagram here to help illustrate where the thin candy shell resides in the virtualization space:

The VMM implements the virtualized CPU, memory, network and storage into the actual guest operating system that is hosted on the hypervisor or VMkernel.  It also provides each virtual machine with its own personalized custom build BIOS!  The VMM detects and understands the hardware type that the hypervisor is running on.  It examines the advanced CPU functionality and then adapts (monitor mode) to pass along those benefits to the guest. 

The VMM handles three different types of virtualization, software, hardware and paravirtualization.  Software virtualization we already discussed above.  Hardware virtualization is leveraging technologies from our x86 based server vendors such as Intel and AMD.  Intel offers advanced processor virtualization features such as Intel VT-x, while AMD offers their own solution called AMD-V.  The hardware virtualization helps offset the overhead of virtualization by offloading the binary translation (BT) to the hardware.  Paravirtualization is the concept of reducing virtualization overhead by having both the host and guest work in conjunction with each other.  A classic example of this approach is pvSCSI, if you want more detailed information check out my write up over here.

As with all great idea’s come trade off’s, the use of the VMM adds a layer of “overhead” to the virtual machine.  There is a translation that has to take place to create this isolated secured environment.  The goal of VMware is to lower this overhead to help drive efficiencies (CPU and memory) and help you consolidate more with less.  Here is a diagram that helps illustrate this concept (BTW I have heard 4.1 has taken this down to 1-3% overhead):

Conclusion

Hopefully this helped shed some light on how VMware’s type 1 hypervisor works, and how it interacts with the virtual machines that is designed to support.  Remember that the VMkernel is responsible for working in conjunction with the hardware layer, and the VMM is responsible for translating that information to the virtual machine.  Overhead is a byproduct of this translation, but leveraging hardware and using VMware will help drive this overhead into complete transparency.

-Scott

Introduction

A fellow VMware Engineer recommended a book to me recently titled “Linchpin” by Seth Godin.  The book has nothing to do with VMware or virtualization but it hits home for me because it highlights a lot of topics that I find applicable to our industry.  I could not ignore this as a relevant force that has somehow affected me, so i felt I had to write something up and share some thoughts.  This post is a little more off paced from what I normally write about so  bare with me.  I think a lot of what Godin covers is present in the VMware community today, and many of you are already “Linchpins”.  I reached out to Seth to get his permission to share some of his insights, if your interested in purchasing the book just click the link above.  Here is the book synopsis:

There used to be two teams in every workplace: management and labor. Now there’s a third team, the linchpins. These people invent, lead (regardless of title), connect others, make things happen, and create order out of chaos. They figure out what to do when there’s no rule book. They delight and challenge their customers and peers. They love their work, pour their best selves into it, and turn each day into a kind of art.

Linchpins are the essential building blocks of great organizations. Like the small piece of hardware that keeps a wheel from falling off its axle, they may not be famous but they’re indispensable. And in today’s world, they get the best jobs and the most freedom. Have you ever found a shortcut that others missed? Seen a new way to resolve a conflict? Made a connection with someone others couldn’t reach? Even once? Then you have what it takes to become indispensable, by overcoming the resistance that holds people back.

Godin believes that our society has changed and we (the U.S. in this example) are no longer living in the industrial era that our parents and grandparents grew up in.  We are no longer the factory-driven-widget-producing  society that we once were, in fact most of these types of positions have been outsourced to cheap labor across the globe.  Going through the schooling process and obtaining a piece of paper no longer guarantees you will be promised a job for the next 30 years of your life.  Competition and technology have extinguished the promise of a secure job that pays well, offers health insurance, and a great retirement package when you exit. 

Sound melodramatic and doom and gloom?  It’s not really.  Godin goes on to explain that because our society is changing, we need to also identify this and change with it.  We are not cogs in a giant industrial machine.  You have a mind of your own, and have more to contribute that you might think.  Working off the same rule book is no longer going to apply if you want to be considered indispensible by forward thinking companies.

The old school of thought:  "”Keep your head down, follow instructions, show up on time, work hard, suck it up”.

The new school of thought:  “Be remarkable, be generous, create art, make judgment calls, connect people and ideas”

Become a VMware Linchpin

Here is how Wikipedia defines Art:

Art is the process or product of deliberately arranging elements in a way to affect the senses or emotions. It encompasses a diverse range of human activities, creations, and modes of expression, including music, literature, film, photography, sculpture, and paintings. The meaning of art is explored in a branch of philosophy known as aesthetics.

The new school of thought talks about becoming an artist, but don’t think of art as the class you avoided in high school.  Art is creating something from nothing, it’s also about creating something that invokes an emotional area in the brain for yourself and others.  Many virtualization evangelists are creating something from nothing, the VMware blogosphere is one of the best examples of this today.  The VMware community is alive with passionate people that are writing and creating new content daily.  Have you ever stopped to examine the VMware Planet v12n blog aggregator?  It’s really quite amazing the amount of new content created around this topic of virtualization.  Customers stepping up, and brining their content to local VMUG’s to share their personal experiences is another great example of creating this type of art. 

Twitter is now inundated with VMware virtualization metadata.  Not only can you find where this virtualization data resides but you can now make  connections with people that would have been impossible to make before.  There are experts in every form and fashion that are now open to communicating about all things that touch virtualization.  Storage experts, systems experts, networking experts, powershell experts, and perl experts are just a few that jump out.  Are you looking for a specific need that might have a benefit to others around you?  Pose the question and 9 times out of 10 someone will write the code and share it with the community at large.

Challenge Yourself

Maybe this is something you are already doing today, maybe it’s something your not doing and will never do.  That’s fine too, I’m just some guy that read a book and sharing my two cents.  I will tell you that as you start to consider some of these topics and look out at the industry in general (not just VMware) you will see this change come into play more and more.

Challenge yourself to get out of your comfort zone.  Go out of your way to make new connections.  Help someone out that might not be as skilled as you.  Write a blog.  Sign up for a Twitter account.  Stand out, create art, be noticed.  It will give you a sense of accomplishment, help define yourself as an expert in your field, and even open more opportunities down the road.

-Scott

Introduction

I come from a family of artists and I thought some of that intrinsic genetic value kind of flowed in the blood, but by my white boarding skills (illustrated above) apparently that isn’t the case.  I promise I will work on my happy little trees as time permits (Bob Ross reference) and try to move away from my chicken scratch art work.  I only hope that I can make both my family and Bob proud.

Read the rest of this entry »

Introduction

There has been a lot of activity here at VMware with acquisitions and partnerships over the past few months.  A fellow engineer at VMware summarized a lot of these acquisitions and how they are meaningful to VMware as an organization.  I wanted to share this information because I think it provides people with a better understanding of where we are going as a company and the overall strategic vision of VMware (Thanks again Andy!).

Read the rest of this entry »

Introduction

Being only three weeks into my career at VMware, I haven’t had much time to do any technical blog posts due to the fact that I have been drinking from the fire hose and trying to ramp up as quickly as possible.  I am writing this post from VMware’s annual tech summit, and I joked with a few people here that I am so new that I haven’t even gotten a paycheck yet.  My wife called to reassure me that I actually did get paid, so no more jokes about a “virtual paycheck” I guess.

Everything has been great so far.  The people I am working with are awesome, the job is going to be fun but challenging, and to be honest I think I am working for the coolest software company in the world.  VMware’s breadth of products is really quite amazing, they are literally covering the stack with many different applications, and driving a change in the industry that many people are excited about, revolutionizing IT.

Read the rest of this entry »

Introduction

“Going Green” has been a buzz word in the IT community for years but the more I deal with this topic the more I consider it a black and white issue.  I never thought I would be covering an energy blog topic, but there are some real world examples I wanted to write about.  Datacenters are enormous consumers of energy from the IT infrastructure itself, all the way down to the HVAC that is needed to cool these power thirsty systems.  While I think green initiatives are much needed in our industry, typically large corporations don’t consider these initiatives unless there is some intrinsic value associated with them i.e. money.  Business drivers outweigh the political pressures of saving the environment, and in all fairness isn’t that what a company should be about, their own salvation?

Maybe that sounds harsh to all of the eco-friendly readers out there, but don’t get me wrong I am all about saving natural resources and respecting our environment.  Understanding the underlying issue of the current state of our industry is critical if one is going to offer solutions to a problem.  If corporations can save operational costs on power and cooling and say they are a “green company” then we have just killed those two birds with one stone.

Read the rest of this entry »

Introduction

Networking is the fourth I/O component that I will be covering in this series of performance write ups.  Networking is another important component in the stack, if not well thought out, can lead to performance problems later down the road.  Security is an important design consideration when planning your network configuration.  One might argue that with a virtual environment your are more prone to risks since at times there is no longer a physical cabling restriction in place.  If someone has the appropriate rights in virtual center, they could bridge two logical networks together, or place a virtual machine into a DMZ.  VMware introduced vShields to mitigate your virtualized environment from some of these risks.  By creating zones you can enforce policies that can bridge, firewall, or isolate virtual machines between network segments.  When designing or upgrading your VMware environment, work closely with your network team to understand their design considerations.  If possible, leverage VLAN tagging (802.1q) to eliminate excessive physical cabling to different segments.

Read the rest of this entry »

Introduction

Personally one of the most interesting components of the VMware architecture I/O stack is storage.  There are a plethora of diverse storage solutions in the industry today that offer unique different ways of addressing storage performance, as well as the increase in capacity demands.  Storage problems are the most common mis-configuration effecting performance that exists in VMware today.  An oversaturated LUN will effect all virtual machines that share that same data store.  Take this concept up a level, a group of disks (RAID group) that are saturated with I/O will negatively impact all LUNS that share those same physical spindles.  Storage traditionally has been the “red headed step child” in VMware and hasn’t gotten a lot of visibility.  Storage I/O bottlenecks can create serious virtual machine problems and yet it wasn’t until ESX 3.x that graphic visibility was even displayed to VMware administrators, see 2.x MUI reflects CPU and memory (Management User Interface for those newer to VMware).

Read the rest of this entry »