Author Archive

 

2012 ushers in some great new changes from the field technical team at VMware.  I  am merging the Ohio Valley Newsletter with the Wisconsin based field newsletter (aka vNews) in an effort to make it more all encompassing.  This content is designed to inform our customers of important updates from VMware from a technical perspective.  It also highlights some great public blog posts that might have snuck by you while you weren’t looking.  We will be moving away from the older legacy.pdf based version of the newsletter to a modernized delivery method, “SlideRocket”.  Here is the link to the first addition! 

 

Please make sure you subscribe  to the newsletter if you wish to receive these monthly newsletters in your inbox.  As always feedback is welcome and will help shape the content for future issues of the vNews!  Special thanks to Ben Sier, Vitaly Tsipris and Jeff Whitman for their contributions and driving to pull this off.  Let us know your thoughts!

-Scott

 

Introduction

Virtualizing and running Java workloads on vSphere is absolutely a reality, but when I talk to customers I emphasize the same best practices as virtualizing Tier 1 workloads.  The rules are not the same as basic consolidation and containment and you need to understand, plan, and architect your virtualization platform if you want to be successful. 

I spend much of my time working with customer infrastructure engineers and architects, and when topics of Java come up, the conversation takes a turn.  The infrastructure teams typically don’t want to get into the application stack and I can’t say that I blame them.  Java and programming are a completely different skillset and the infrastructure engineers already have enough full time jobs keeping the datacenter running.  The purpose of the blog post is to help shed some light on a new technology in vSphere 5 called “Elastic Memory for Java” or EM4J and hopefully some other simple Java best practices and information as well.  The end state of this blog is to help you bring up an EM4J configuration of your own so you can begin to see the value and test your own JVM configurations.  I am also writing this to help educate some of the infrastructure engineers and help explain why this feature matters (Disclaimer = I am not a Java programming guy).

 

What is EM4J?

Hopefully you are somewhat familiar with the intelligent memory management features that come with the vSphere platform such as memory ballooning.  Ballooning is a great technique that allows you to reclaim memory from virtual machines if it’s not in use by the VM.  When dealing with Java workloads a VMware best practice has always been to set reservations for the virtual machine.  This means we are always guaranteeing (or backing) that the memory will be available to the VM when it needs it.  When a memory reservation is set for a VM the hypervisor won’t reclaim memory from this VM (which means VM’s memory won’t be ballooned, compressed or swapped to persistent storage) if memory is tight on the host.

If you consider the definition of JVM (Java Virtual Machine) the last two words are important to consider when talking VMware virtualization.  Running a VM on a VM creates somewhat of a problem for the hypervisor.  The JVM is essentially a black box to the hypervisor and it has no visibility into what’s going on inside it’s environment.  EM4J on the other hand allows one to reclaim memory through a much cheaper mechanism, and induces GCs at the moments when VM is handling relatively low load. It does not eliminate long pauses as VMs without full reservations can end up swapping, but it significantly reduces pause time and provides a more graceful performance degradation when running overcommitted, making workload’s performance more predictable.  Now that I have described some of the characteristics, here is the actual definition according to the VMware documentation:

 

“Elastic Memory for Java (EM4J) manages a memory balloon that sits directly in the Java heap and works with new memory reclamation capabilities introduced in ESXi 5.0. EM4J works with the hypervisor to communicate system-wide memory pressure directly into the Java heap, forcing Java to clean up proactively and return memory at the most appropriate times—when it is least active. You no longer have to be so conservative with your heap sizing because unused heap memory is no longer wasted on uncollected garbage objects. And you no longer have to give Java 100% of the memory that it needs; EM4J ensures that memory is used more efficiently, without risking sudden and unpredictable performance problems.”

 

As you can see VMware is taking the same underlying technology that has been used for years across our customer base and applying it to Java workloads to gain more/better efficiencies at scale.  The same performance characteristics apply to EM4J as they do to the ballooning in the VMware ESX hypervisor.  Ballooning will only be invoked if the system is over committing memory, and has to begin utilizing its advanced memory management techniques.  The benefit of EM4J is when the host is under memory pressure, the end user experience will be the same as if the VM was hard backed with physical RAM as we discussed earlier.

 

Getting started

EM4J is a product that works in conjunction with vSphere 5 and vFabric tc Server that is bundled with vFabric Standard and Advanced.  EM4J can also work directly with Apache Tomcat.  You might be asking yourself what is vFabric tc Server at this point and why the hell do I care about that?    vFabric tc Server is a Java application server based on Apache Tomcat that VMware maintains and supports.  This is a competitive product to a IBM WebSphere or an Oracle WebLogic, but is a much lighter weight Java container that allows faster deployments in development as well as production environments.   As a systems infrastructure engineer it is imperative that you understand these types of Java workloads from a high level.  Your success in moving these workloads into a virtual infrastructure depends on it and is irrelevant to EM4J.  Before I jump in and show you how to set this up there are a few things we need to get out of the way first.  Here is what your going to need to begin utilizing EM4J for your own testing, grab it now:

• VMware vSphere 5
• VMware vFabric tc Server 2.6
• VMware vSphere 5 Web Console (for reporting visibility)
• VMware vFabric EM4J Documentation
• VMware vFabric EM4J web console UI plug-in
• Redhat RHEL 5 operating system (Officially supported OS today)
• JVM Hotspot 1.6

Making it work in vSphere

As noted in my disclaimer above, I am not a Java guy so this took me some time to get my lab environment up and running with the right components since I am new to vFabric.  RHEL is the officially supported operating system today, but Linux is Linux so I chose to grab the latest Ubuntu 11 distribution for my testing.  Work with your internal Java guru to get vFabric tc Server setup and running on your Linux VM for testing.  Once you get through setting up and installing your operating system and vFabric tc Server, there are some technical pre-requisites you need to accomplish in order to enable EM4J balloon driver and gain visibility into the JVM itself.

The first step you need to perform in your testing is to enable an advanced parameter within the Linux VM your are testing with.  The virtual machine will need to be powered down to perform this action.  Right click on the virtual machine, select edit settings, and the select the options tab.  Go down to the advanced section and select “General” and then select the “Configuration Parameters” button that is now visible:

 

Once you select the “Configuration Parameters” button you are going to select the “Add Row” button and add the following configuration parameter to the VM:

sched.mem.pshare.guestHintsSyncEnable and set the value to “true” as shown below:

 

Making it work in tc Server

Once you have enabled the virtual machine for EM4J, you also need to ensure your instance of tc server utilizes the EM4Jbaloon driver.  Execute the command listed below to create a new instance, in this example my instance name is “scott” and the “elastic memory” option is what enabled the EM4J balloon driver.  Once you have created the instance, go ahead and start it up!

 

Next we will configure a few parameters within out instance so we can  monitor them via the VMware vSphere web console interface which I will show you next.  Add the following parameters to the setenv.sh file of your new instance name as follows:

 

JVM_OPTS="-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.port=6969
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false"

 

Next step we need to setup what is called the Console Guest Collector (CGC).  The CGC is a process that allows the vSphere web console to pull data from the EM4J balloon driver and place it with each VM so the web client can then display performance data about the current workloads.  This needs to be setup via a cron so we can continuously pull real-time data into vSphere.  The cgc.sh script can be found in the /opt/vmware/vfabric-tc-server-standard-2.6.0.RELEASE/templates/elastic-memory/bin/ directory.  Here is a command to add an entry to the crontab for every 5 minutes:

*/5 * * * * /opt/vmware/vfabric-tc-server-standard-2.6.1.RELEASE/templates/elastic-memory/bin/cgc.sh >
/dev/null 2>&1

 

Making it work in the vSphere Web Client

You downloaded the EM4J UI plug-in earlier and now we need to extract it and set it up on your vSphere 5 Virtual Center server.  Extract the contents of the following directory then re-start the vSphere Web Client Service:

C:\Program Files\VMware\Infrastructure\vSphere Web Client\plugin-packages\em4j-client

 

 

The data!

Now that we are through the tedious stuff we can actually see some of the more interesting performance data, and frankly the reason you are probably reading this blog post!  Log-in to your Virtual Center’s web interface and navigate to your virtual machine you are using to test with.  Select the fourth tab at the top of the options section which is titles “Workloads”.  You should now see something similar to this and the EM4J Agent Enabled should be selected if you setup everything correctly:

 

 

Selecting the “Alerts” tab will give you any relevant data and tell you if any issues are occurring.  This will also display some Java Best Practices and instruct you on how to fine tune your JVM.  Selecting the “Resource Management” tab will display much more performance centric detailed information which gives you full visibility into the JVM itself.  Excellent performance visibility into that problematic Java workload:

 

Conclusion

From the documentation, “EM4J helps the system behave gracefully and predictably when memory becomes scarce. It helps you to more easily determine the over-commit ratio that provides acceptable performance at peak loads.”  Hopefully you learned a little bit about what Elastic Memory for Java is and how it works within vFabric and VMware vSphere 5.  As with most technology features and functionality I suggest understanding the best use cases for EM4J and how it fits into your own environment.  The documentation that I linked to, gives plenty of examples of when EM4J should be utilized effectively.  Look for more performance benchmarks around optimal overcommit ratios as our vFabric team completes some great performance testing on this exciting new technology.  The EM4J architecture will not only allow you to run your JVM’s more efficiently, but will also provide you some great performance visibility and give insight into your Java workloads.

 

Introduction

VMware is in a perpetual state of change if you haven’t noticed.  Virtualization and the hypervisor will never not be a core competency of ours but we are continually expanding into other areas of IT software solutions for our customers.  I think Paul Maritz states it best with his quote “When we see a management problem, we will be replacing it with an automation solution”.  Take a look at what VMware vCloud Director is accomplishing by delivering the automation of IT services at the Infrastructure layer for our customers and service providers.  Take a look at what Horizon App Manager is doing to create and deliver a self-service enterprise application store to consume SaaS based applications.  One of the best parts about moving towards a self-service model, the system engineers can now have part of their lives back to focus on more important projects for the business since the end users can now consume services on demand. 

When I talk with my customers, half of my challenge is educating them on what we are doing to enable them to operate more efficiently from a solution perspective.  We are no longer just a hypervisor company.  Don’t get me wrong, I love talking speeds and feeds and can geek out and get distracted with the best of them on the “tech cool factor”.  Alas, at the end of the day isn’t it about finding a solution that is going to make you (systems engineer) be able to do more with less while also delivering value back to the business?  This is why our customers love us so much into the “infrastructure corner”. We have helped them to maintain happy lines of business by providing IT faster to their customers with virtualization.  As our former friend Patrick Swayze (RIP) once said,  “Nobody puts baby in the corner!”

 

 

Enter vFabric Data Director

Let’s think out of the traditional infrastructure box, there are challenges with virtualizing databases from a management perspective.  No it can be done, and many customers are out there deploying Oracle and Microsoft SQL databases on VMware vSphere.   It’s not the traditional I/O workload conversation that one must consider when going after these tier 1 workloads.  It’s more about the long term management of these resources that are constantly being requested, deployed, copied, backed-up,  and the backend management that goes into this entire process.  Database sprawl is a real world problem that many organizations struggle with.

Why not create a portal where a non-DBA type IT user can login and tear off a database by answering a few simple questions?  Why not pre-configure the DBA repetitive tasks from a list of options or a “catalog” and allow them to choose the correct combo meal they would like to consume?  Why not give the DBA’s back time in their day to do more productive forward thinking activities and take the easy operational stuff off their plates?  Enter vFabric Data Director (The artist formerly known as Project Aurora).  Notice the following features and functionality as your watching the clip.

• Self-Service Provisioning
• Linked Database Clones
• SQL Statement Execution from the Web Portal
• Backup and Recovery simplification

 

 

vFabric PostgreSQL

The vFabric Data director portal probably makes sense to you now that you have seen it in action.  The first database we have enabled on top of this platform is a vSphere-optimized PostgreSQL database, the most enterprise-ready open source database.  We have specifically custom tuned this fork of Postgres to make it virtualization aware and to run more efficiently.

 

 

The vFabric Postgres database  is delivered to the IT environment in the form of a virtual appliance that is intelligent and can self-tune itself as workloads change.  Database buffer sizes can scale up and down as I/O characteristics change, a special ballooning database driver can be invoked for more memory efficiencies within the virtual appliance.  The database us a standard SQL database that supports ODBC connections and supports JDBC tools to query the database just like the open source version.

What’s next?

Expect much more!  I can’t say a ton here in this forum, but know this is just the beginning for this product/solution.  EMC’s Chad Sakac put together a great video for VMworld 2011 (and apparently is allowed to say much more than I am) on his blog site.  Watch towards the end of the video for futures and where you can expect to start to see the flood gates open up as we take it to the next level! 

- Scott

 

 

 

Introduction

With vSphere 5 comes a plethora of new features and functionality across the entire VMware virtualization platform.  One of the core components that got a nice upgrade was the vSphere Distributed Switch (vDS).  For those of you that have not had the chance to use the vDS, it is a centralized administrative interface that allows access to manage and update a network configuration in one location as opposed to each separate ESX host.  This saves vSphere administrators or network engineers a lot of operational configuration time and/or scripting activities.   The vDS is a feature that is packaged with Enterprise Plus licensing.  Here are some of the new features that are included with the vDS 5.0:

• New stateless firewall that is built into the ESXi kernel (iptables is no longer used)
• Network I/O Control improvements (network resource pools and 802.1q support)
• LLDP standard is now supported for network discovery (no longer just CDP support)
• The ability to mirror ports for advanced network troubleshooting or analysis
• The ability to configure NetFlow for visibility of inner-VM communication (NetFlow version 5)

 

NetFlow Basics

I could do a write-up on each one of these components as they are all worth discussing in more detail, but I wanted to focus on the NetFlow feature for this post as I think it’s an awesome addition.  NetFlow has had experimental support in vSphere for some time, but now VMware has integrated the functionality right into the vDS and is officially supported.

NetFlow gives the administrator the ability to monitor virtual machine network communications to assist with intrusion detection, network profiling, compliance monitoring, and in general, network forensics.  Enabling this functionality can give you some real insight into what is going on within your environment from a network perspective.  Having “cool features” is a nice to have, but having features that you can utilize and show value back to the business is a completely different value add.

Let’s look at how to setup NetFlow on the new vDS, then take a look at the data you can extract from NetFlow with a third party NetFlow viewer.  Once you see the value of the data, you can then make some important IT business decisions on how you need to mitigate risk and protect your investment by getting ahead of the curve (aka VMware vShield or some other third party software).

 

Setup your vDS 5 Switch

Ensure you are running VMware vSphere 5.0 and have activated Enterprise Plus licensing to setup the vDS switch in your environment.  You can see below the new option to deploy a vDS 5.0 switch, and of course we offer backwards compatibility for those that need to deploy to their 4.x environments.  Select the 5.0 version and hit next.

 

In the “General” section give the vDS a name, in this example I am giving him “dvSwitch5”.  Select next the number of network interface cards you want to participate in the switch and then select next.

 

For each host in your cluster that you wish to participate in the vDS, you will need to configure the network interfaces that will support this vDS implementation.  In this example I have selected vmnic 4 and vmnic 5 to be members of the vDS 5 switch.  Select next.

 

That’s it, review the summary and select finish for your vDS configuration to come online and begin configuring NetFlow.

 

 

Setup Netflow on the vDS 5

Now you have a fully functioning vDS 5.0 switch, you can actually start to use it!  First let’s go ahead and configure NetFlow on the dvPortGroup, then we will move some virtual machines over to the new vDS so we can get some real data flowing.  Right click on your newly created dvSwitch and select “edit settings”.  Go to the “NetFlow” tab across the top of the page.  You will need to give your vDS an IP address so your NetFlow tool will know where to collect the data from.  Populate an IP address for the vDS, then you will need to enter the IP address of the collector you plan on using to pull the data from.  Make sure you enter the correct port number (default is 1) for how you setup your NetFlow application to communicate.

 

Right click on the dvPortGroup within the vDS and select the “monitoring” option and enable NetFlow so you can begin to collect data.

 

Move a few VM’s over to the new vDS so you can begin to capture some real data within your newly established NetFlow configuration.  I have highlighted below how you can change the network connection on a VM to now utilize the dvSwitch5 we created earlier.

 

Pull Some Data

You will need to utilize a third party NetFlow analysis tool to parse the data we have started to generate.  In the example below I am using a pretty nice application called Manage Engine Netflow Analyzer.  I won’t be covering how to install or setup this application here, as your organization might already have some network tool that they have standardized on.  Once you have moved some virtual machines over to the new vDS, ensure you start to create some traffic so there is some relevant data to examine.  Below I ran a few speedtest.net downloads, and hit some websites to make traffic appear below.

 

Below you can see the different virtual interfaces on my vDS that are being monitored.  You can see our application is showing us what type of traffic we are examining, and the consumption of the different tcp/udp ports that are communicating both inbound and outbound on the switch.

 

The “under the covers” reporting is great stuff, but let’s start to look at how this can help the business.  Consider a VMware View environment where you are supporting hundreds if not thousands of desktop images.  You can use the NetFlow data to start to examine if certain VM’s are communicating to production systems that they shouldn’t be communicating to at all.  How about reducing the overall workload on your VMware View ESX server?  Many of the NetFlow products like the one I am showing here will produce reports on where users are going externally on the internet.  See the report below.  YouTube is probably a website you want to keep an eye on, as streaming video can greatly impact a virtual desktop environment.

 

From an intrusion detection and compliance perspective, you can now gain visibility into the vSphere environment to begin to understand some of the network communications that are taking place.  See below:

 

 

From a risk mitigation perspective, VMware can help you eliminate these security vulnerabilities that you are beginning to gather data on.  VMware vShield has three different solutions that can help protect your environment from the edge to the core.  I would suggest to examine segmenting and protecting your internal workloads to eliminate these security risks.  From a virtual desktop perspective, the desktop workloads are better served being contained in their own protected segment (VLAN’s are broadcast domains not protected segments).  Below is an example of how a logical vShield configuration can begin to help you segment your virtual infrastructure.

 

Conclusion

VMware vSphere 5 offers some great new features that are integrated into the new vSphere 5 Distributed Switch.  Start to leverage your existing investment by examining your network infrastructure with the NetFlow data you can now begin to extract.  Once you have gathered this data, begin considering how you can mitigate some of the security and compliance risks within your organization.  VMware vShield is a product that can help you in this regard and will integrate into your current environment.

 

-Scott

 

 

Introduction

 

This is a follow-up blog post to a write up I did last year on upgrading your virtual hardware.  The post I did was really trying to show people how easy the virtual hardware to version 7 was, and that despite it being a manual effort, it wasn’t all that painful.  There have been several other write-ups in the community that cover how to automate this task to save you time and effort.  In the end, there was no easy automated way to accomplish this task that was officially supported.

There are so many great new features that are being released with vSphere 5 that some of the small stuff might get missed.  As a former VI admin, this is one of the small ones that can’t be overlooked for those of you in the trenches.  There is another new feature that is introduced with vSphere 5 called “VMware Auto Deploy” that somewhat competes with VUM from a ESX deployment methodology.  If you would like to learn more about Auto Deploy, check out Gabe’s write-up here.

 

In a Nutshell

• VUM can be used to upgrade your ESX 3.x hosts and vSphere 4 hosts to 5.0 (3.x makes a pit stop at 4)
• VUM can be used to upgrade your vSphere “Classic” hosts to ESXi
• VUM can now remediate multiple ESX hosts at the same time rather than queuing up (think multi-threaded)
• VUM can automatically upgrade VMtools at a scheduled maintenance window
• VUM can automatically upgrade Virtual Hardware at a scheduled maintenance window
• VUM can no longer be used to patch guest operating systems
• VUM requires a Windows Operating system and can not be installed on the VMware vCenter Server Appliance
• VUM can automatically upgrade your Virtual Hardware from version 4 or 7 to version 8 (vSphere 5)

 

Update Manager to the rescue

You can now use vSphere Update Manager to perform orchestrated upgrades to upgrade the virtual hardware and VMware Tools of virtual machines in the inventory at the same time.  Not only can you use VMware update Manager (VUM) to upgrade your ESX hosts to version 5 you can also leverage it assist with the hundreds of VM’s you need to address as part of the upgrade process!  This is a huge time saver and will help eliminate configuration drift across your environment, as I am sure your virtual infrastructure has only grown bigger since the last time we went through this. 

Let’s walk through what this process looks like, and how you can now configure update manager to accomplish this.  I am going to assume you have already setup or upgraded your Virtual Center to version 5, and you have also updated or installed VUM 5.

Automate the VMware Tools upgrade

The first step in upgrading your virtual infrastructure is to crate a plan of attack.  Most of my customers group their virtual machines by applications or by lines of business.  This typical grouping won’t lend itself well to our virtual machine updating that we need to do.  I suggest creating a few folders in the “VM’s and Templates” view that you can use to help facilitate this upgrade.  As you can see below I created three different folders that you can use to temporarily move the vm’s into for their scheduled maintenance.   I suggest creating different upgrade windows that you will attach to these three folders (after getting change management approval of course!).  Yes there is downtime required for this process!

 

For each of these folders you are going to want to configure it to apply the VMware Tools upgrade first.  You can see below that this option is selected for my first patch management window.

 

After I have selected my VMware Tools upgrade, I can now scan the VM’s that I have moved into this folder to discover which ones need be upgraded.

 

Now you want to select “Remediate” on the new baseline that we have configured.  You will be prompted to create a schedule for the VMware Tools installation as shown in the capture below.  I have configured my first VMtools patching to occur at 2:20 a.m.

 

VMware Update Manager gives you the option of taking a snapshot prior to the tools upgrade in case something goes sideways during the upgrade procedure.  Here you can also select if you want to retain your snapshots or have VUM remove the snapshots after a configured period of time (hours):

 

Automate the Virtual Hardware upgrade

Now let’s run through the same process again, this time we are going to select the “VM Hardware Upgrade” which will then bring your VMware virtual machine hardware version up to version 8.  As I mentioned above, you can be running at either version 7 or even version 4 for VUM to update your virtual hardware.

Same as before, but this time make sure you stagger your virtual hardware upgrade for 30-40 minutes later:

 

Same options as before, feel free to take snapshots of the vm’s in case you need to revert for some reason.  Be aware, if you are doing snapshots across hundreds of virtual machines, you should consider the disk space that they will be consuming in both the short and long term.

Below you can see in the recent tasks that our upgrades are taking place automatically which should give you some of your personal time back to do other more important things in your environment.

 

 

Conclusion

Leverage VMware Update Manager as part of your upgrade path to vSphere 5.  Automation is critical as your virtual environment continues to grow exponentially.  I haven’t spoken with one customer that is hiring more VMware engineers to their team, so we need to leverage tools/technology to automate whenever possible.

Hope this helps!

-Scott

** I have modified this post to include the updated licensing changes announced 8-3-2011, as well as the link to Alan’s updated PowerShell script.  The video clip has not been updated, the concept is the same only the number have changed.  Enjoy!

 

***Disclaimer – I am a VMware employee and receive paychecks that have the word VMware stamped on them.  My thoughts are my own but if you are afraid of an employee’s opinion, run away now!

 

Introduction

A lot of big announcements have taken place over the past week from VMware.  We had our largest launch event in history and announced to the world that our flagship product “vSphere 5“ is being released.  We also announced many other product updates and releases like SRM 5, vShield 5,  vCD 1.5 and an updated release of heartbeat for Virtual Center.  Loads and loads of exciting new features and functionality to allow you to run your mission critical workloads on the worlds #1 hypervisor.

I wanted to do a factual write-up on what has changed, hopefully explain it better since there seems to be a lot of misconceptions, and also give you my take on what I am seeing from my customer base.  For those of you that don’t know my background, I came from a customer environment where I designed and implemented VMware in a large scale deployment.  I then decided to convert to the dark side and go work for a vendor (VMware) so I actually have some perspective to offer from both sides of the fence.  Now that we are one week out from the new licensing change, I thought I would share some thoughts.

 

What is vRAM?

One of the changes that came along with a lot of exciting new features was the new vRAM licensing model.  VMware has decided to move away form the core based model to a “consumed virtual ram” model across the entire environment.  For those of you that are unsure of how the current vSphere 4.x licensing model works today, I have embedded a chart here to help you understand the core limitations as well as the features and functionality between the versions.

 

 

Below is the new vSphere 5 pricing comparison which also includes a feature/functionality breakout.  As you can see it lists the new vRAM entitlements and how much pooled vRAM you are allotted  per socket.  Notice we have removed the core limitations around the physical processors, and lifted the memory limitation from a physical perspective.

 

Your current licenses will be converted to the correct vRAM allocation model depending on the version of vSphere that you are paying maintenance on.   VMware has provided a power shell script to assist you with determining the amount of vRAM you have available to you when you decide to upgrade your environment.  My suggestion is to download the power shell script from VMware’s Alan Renouf that will automatically calculate this information for you.

I decided to put together a quick video link that discusses vRAM in a little more detail, and gives you an example of what this pooled model looks like across a 3 node cluster.  Watch the video below:

 

 

My thoughts

There has been a lot of emotional responses around this topic over the past week, which is understandable.  VMware has the best user community of any software company I have ever seen.  There are several reasons for this in my opinion.  VMware has made great impacts on our customer environments from a datacenter consolidation perspective.  We have allowed our customers to run more efficiently, do more with less, and enabled them to become hero’s for their internal customers by offering them agility.  We have also given them a portion of their lives back from an administrative perspective. 

I was one of these end users that came in to fix systems at all hours of the night when the hardware went south.  I was responsible for deploying hundreds and hundreds of physical servers that eventually consumed all of my time from a maintenance perspective.  VMware technology gave me my personal life back, and allowed me to start doing mundane maintenance tasks during the day!  Storage vMotion allowed me to retire 3 older storage array’s (30+ TB) that were at the end of life, and move them to newer technology with no downtime,  during the middle of the day!  I believe there is a personal component to all of this, and part of this is what makes the community such a strong force.

The goal of this article is to educate people, and give you my perspective.  It is not intended to defend VMware’s position or try and convince you why vRAM should make you sleep easier at night.  VMware has spent the past two years working internally and externally with customers to try and determine a fair licensing model that works for everyone.  The current model will not scale with the quickly changing hardware landscape driving core counts exponentially.  A handful of my customers are already having to double up on licenses as they are at vSphere Enterprise and are being restricted on the number of core’s.  I think the model is a fair one and as you walk through it hopefully you will see the logic behind why VMware had to make this change, to better support our users and the community.

 

Hello, it looks like from the downloads we are tracking, the newsletter is around to stay for a while.  If you have any suggestion or would like to see the format change, let me know!

Welcome back, I hope you found our first newsletter helpful in some way shape or form.  The newsletter seems to be getting larger and larger which is a great thing.  It might soon start to qualify as a magazine rather than a newsletter.

We got some good feedback so we are going to keep going with this for a while.  Please let us know via the comments section if you are enjoying it, would like to see different content,  or just want to say hello.

-Scott

When talking about VMware virtualization bottlenecks, 9 out of 10 customers answer their number one bottleneck is memory. Notice how I said bottleneck, not problem. This relates to capacity planning or trying to understand and right size the environment so you can gauge when you need to order more physical infrastructure. Their number one problem is storage, which is quite a different story altogether and I won’t be covering storage in this article (this time). Since memory is such a common point of discussion with my customers, I thought I would dig a little deeper on this topic and share some information around utilization and what it all means.

 

My customers typically track their utilization in the most common area of vSphere that one might expect to find this information, the DRS Resource Distribution graph at the cluster level.

 

From the image displayed above, one might think that I am close to memory capacity and I should look at ordering more hardware for my cluster. While in a general sense that might not be a bad idea to begin planning for growth, but let’s take a closer look at what we are seeing.  Notice the blue informational icon and how it’s telling us that the displayed information is based on memory consumption. Let’s do a mouse over on the chart that’s being displayed to get some more granular information and what this means.

 

 

You can see in the above image that my Virtual Center VM is “Consuming” ~4GB of memory, but in all reality the active memory being used is sitting at ~700MB. DRS entitlement is a measurement that calculates what the load or demand is on the vSphere host/cluster over time, and then projects an average entitlement number for planning purposes. You can use the DRS entitlement numbers as a general planning/forecasting number, but to be honest you still have some capacity within the cluster.

Now I wouldn’t be doing my job if I didn’t make you aware of an easier way to track this information by using software rather than brain power. For those of you that haven’t seen Capacity IQ yet, I would highly encourage you to evaluate the product. Capacity IQ was built for this specific reason, to help you understand when you will need to start thinking about more hardware. It can also help you run your environment more efficiently. There are some great reports that help you identify which virtual machines are not using the resources that were allocated to them.  Take them back!

Coming from a VMware system engineer end user position, I can tell you that as your environment begins to grow, capacity management and planning becomes critical. I evaluated Capacity IQ when I was still on the customer side, and did a write up if you are interested in my thoughts on the product.

 

At one of our last internal central VMware meetings, a few of us had a similar idea to pull together a newsletter for our customers.  Some of us were already doing this to a degree, but collectively we agreed that one source of information would be better than many.  Several VMware SE’s and Specialists have helped pull this together so I wanted to thank everyone for their hard work. 

There is so much great content that is published to the web and sometimes passed internally, we wanted to consolidate this information to one common distributable location.  99% of the content is not really specific to the U.S. Central region other than the local events, so I think that many people will be able to benefit from out efforts.  The goal will be to test the waters and see if it’s something that people like and want to see continued.

As always we are looking for feedback, if you think this is something that should continue, let us know!  If you feel it’s lacking or could be improved in some aspect, we are also looking for your opinion to help shape it.

Enjoy!

-Scott

I spend a majority of my time talking with VMware customers trying to help understand their needs and how we can help them with some of their internal IT business challenges. I would say a majority of the problems and issues discussed are typically based around internal politics and the IT landscape changing, but their second largest concern is around performance and growth (capacity). VMware and virtualization in general, has been such a powerful driver for many organizations over the past several years. It has allowed IT organizations to run more efficiently, save capital expenditure costs, and ease administrative overhead, all in the midst of an economic downturn.

 

Capital expenditure costs savings are great, and very visible to the organization from a high level, but VMware needs to help customers with the next step. Now that we are moving so much of our infrastructure to a more elastic and flexible solution, (vSphere) we need to provide tools to help you manage this infrastructure because the same methodologies no longer apply as they did in the physical world. The more we can help automate and manage your virtual infrastructure; we can now begin to help with step two which is save your IT organization operational costs. A recent Gartner study determined that the average cost for a Windows server is $10,200 per year. Of that expense ~ 70% is on OPEX. Gartner also estimates that with automation and management that up to 80% of the OPEX could be saved.

VMware has made several acquisitions around management and automation, and I wanted to focus on one which was recently announced. VMware Operations is a “new” product that was released this past week. It’s actually not all that new of a product but a re-branding of a key acquisition announced at VMworld 2010. Integrien was an analytics and statistical based software company with a focus on management software. Notice that their primary focus was not management but analytics, a completely different approach to several other software companies out there trying to get to the same end result.

 

Rather than simply creating metrics to monitor and then setting thresholds on those metrics, Integrien will actually analyze the information that it’s gathering and understand when there is an actual problem. One of the coolest features about the full blown enterprise version is you can feed multiple data sources into the analytics engine. The more data it gets, the more accurate it’s able to predict when a problem is likely to occur.

This isn’t just your standard run of the mill monitoring software.

Those of you that have experience with enterprise monitoring software will know that it take a lot of effort to get these systems up and “fine tuned”. It takes a tremendous effort to begin sifting through all of the white noise alerts that come in and then adjust the threshold alerts to something that is tangible so it becomes useable data. VMware Operations removes that manual effort by dropping in an intelligent analytical engine that can understand what’s really going on behind the scenes.

 

 

Here are the different versions of the product, and how each version differs. I would suggest pulling down the virtual appliance and check out how awesome this product is. If you don’t feel like going to the effort, check out this video, it gives a great walk through of vCenter Operations and explains a lot of the same concepts I just wrote about.

There are a lot of the customers I cover in my region that are really starting to see the value in VMware’s management tools.  As virtual machines now outnumber physical machines, customers need some tools to help report against their existing infrastructure as well as predict and prepare for future virtual machine workloads.  One of my favorite VMware tools that I liked when I was on the customer side was a product called Capacity IQ.  I wrote up a blog post that I think people found useful that was basically an overview of the benefits of the product.  You can check that post out here.  I tell most of my customers about it, because it’s simple to setup (virtual appliance) and it gives you loads of great information about your existing infrastructure.

One of my customers that is moving forward with a CapIQ implementation e-mailed me about what types of storage metrics are available from the product.  I was happy to inform him that Capacity IQ 1.5 was just released and provides some great storage statistics that can now be reported against.  Much to my dismay, he told me that he wasn’t seeing the storage report data, the metrics were all blank.

Here are the requirements to get the reports to produce storage related information:

You  need the vCenter management webservices running for CapIQ to collect some of the storage metrics. The storage IO metrics require you to have ESX 4.1.  When you use ESX 4.0 or earlier hosts, the following metrics appear with dashes (–) and affect the Dashboard, the Datastores – List view, the Virtual Machine Capacity – Summary view, and the Virtual Machine Capacity Usage – Trend view:

* Disk I/O read/write
* Disk I/O reads/writes per second
* Disk I/O read/write latency
* VM Disk I/O read/write latency

-Scott

Introduction

VMware recently announced the general availability of a Zimbra virtual appliance that VMware customers can simply import into their existing infrastructure and get “e-mail in box”.  This is a great concept for administrators because the operating system is pre-configured and purposefully built for the application that is packaged with it.  The virtual appliance will import into the virtual center management console and will have the standard "OVF” (Open Virtualization Format) file extension for those that are new to virtual appliances.

I am no e-mail administrator, so I wanted to see how easy setting up the Zimbra virtual appliance would be and provide some instructions for those out there that are looking to test out Zimbra.

 

Get the Bits!

First things first, go out and grab the download of the Zimbra virtual appliance by clicking on the icon below.  Yes you will need to register to download the bits…

Import the Virtual Appliance

There are two methods of importing a virtual appliance, you can enter the url, which is supplied by the Zimbra website once you register, or you can download the appliance locally and import it locally.  I grabbed the full download in case I hosed something up I would have a copy of the ovf locally so I could start over from scratch.  I guess a snapshot would work as well, so it’s up to you how you would like to proceed here.  Below is a screenshot of the import:

Configure the Zimbra Virtual Appliance

The Zimbra virtual appliance is pre-configured to ask you the basic configuration parameters you will need to get the appliance up and running.  You can see below are the questions that you will to answer, pretty common stuff if you a IT administrator.  Make sure you use the FQDN for the hostname.

 

Power it up

Now that you have configured your basic system information you can now power up your new virtual appliance.  You can see below that it will automatically configure the appliance based off your information you have previously populated.  Very nice for a hands off approach and a streamlined installation process.

 

Finish it off

Now that you are powered up and on the network, you can login to the administration console to finish your configuration.  Point your web browser to https://<hostname>:5480  The administration console will be the place where you can create user accounts, configure licensing information, pull diagnostic data for troubleshooting and update the virtual appliance itself.

DNS is a big component of e-mail.  If you are doing split DNS or Dynamic DNS, I suggest  to reference this link to assist your efforts.  I am using a dynamic DNS service at home along with split DNS, so I had to go and update my host entry with a MX record so the world new where to route my e-mail traffic.  Once that was done correctly I was up and running and able to send/receive e-mails with no problem.

 

Licensing

The last thing you will want to to is license your installation, the nice folks over at Zimbra will give you a 10 user license free of charge.  Click the link below to go license your configuration or view some sample pricing on what a fully licensed configuration would look like.  Enjoy!

 

-Scott

It’s hard to believe that another year has flown by and Eric Siebert’s voting for the top Virtualization blogging contest is upon us once again!  If you enjoy the content that you read from Virtual Insanity, I encourage you to give back to the community and vote for us!

http://vote.vsphere-land.com/

What other site discusses great technical VMware content ranging from core ESX, Spring Source, Linchpin’s and even open’s its doors to great guest bloggers??  Thanks for reading.

-Scott

 

Introduction

I was talking with a local customer the other day that was inquiring about the differences between Microsoft virtualization (Hyper-V) and VMware virtualization solutions.  This customer was hung up on putting the two vendors into a cage match and making them go at each other to see who won.  I used to work in the end user IT environment, and know people that think this is a smart approach.  Competition is great for the end user environment because it drives innovation and keeps costs in check.   But let’s take a look at the technology rather than the cage and understand what’s under the hood.  Just to set the record straight this is not a slam Microsoft Hyper-V blog post.  I don’t get wrapped up into the battle of the hypervisor conversations, if you want to go with a competitive solution have at it.  We will be talking down the road again eventually. 

This individual understood the basics of virtualization but wasn’t that clear on how the hypervisor worked in conjunction with the hardware.  I point my customers to quality blog sites to answer questions or if they need instructions on how to configure or stand something up.  I figured I would do a write-up to help shed some light on how virtual machines work, how they are handled by the hypervisor and how they are lean and mean.

Hypervisor types

First let’s set the record straight by saying that VMware, Microsoft or Citrix did not come up with the concept of virtualization.  The idea of abstraction has been around for over 50 years, and was first mastered by the smart people over at IBM on some old iron in the late 60’s.  VMware did launch the first x86 based virtualization software in May of 1999, which since has changed the open systems world greatly.

Currently there are two types of hypervisors:

Type 1 – A native or bare metal hypervisors run directly on the host’s hardware to manage and monitor the guest operating systems.  Because it has direct access to the hardware resources and doesn’t go through an operating system, it runs more efficiently than a hosted model or Type 2.

Type 2 – A hosted hypervisor that runs within a conventional operating system.  This hypervisor does not have direct access to the hardware thus traditionally has more overhead that a Type 1 hypervisor.

Examples of Type 1 hypervisors would include, VMware ESX, VMware ESXi, Microsoft Hyper-V, Citrix Xen Server.  Examples of a Type 2 hypervisor would be, VMware workstation, VMware server, Microsoft Virtual Server.  The Type 1 hypervisors run more efficiently as they are designed specifically to handle virtual workloads.  They also don’t have a host operating system to have to share, schedule and contend with resources for.  The

The VMware Architecture

Two of the most important components of the VMware Type 1 hypervisor are the VMkernel and the VMM.  The VMkernel is the actual VMware ESX hypervisor product that we all know and love.  It is responsible for interacting with the physical server hardware that you install vSphere onto.  Sounds pretty simple in concept right?  It’s not.  I disagree with people that think the hypervisor is a commodity technology because there are some very special things that VMware does differently.  VMware takes on an approach unlike other virtualization vendors in the marketplace implements a hardware compatibility list (HCL)  to ensure you will be running a supported configuration.  That means when you install the product, VMware has already QA’d the configuration and is a eliminating the guess work of a supported stable environment. 

The VMkernel doesn’t actually run the virtual machines, it invokes yet another layer of protection called the virtual machine monitor.  This “Thin candy shell” (Tommy Boy reference) is the special sauce that takes various communications from the VMkernel and translates them to the actual virtual machine, and visa versa.  I put together a diagram here to help illustrate where the thin candy shell resides in the virtualization space:

 

The VMM implements the virtualized CPU, memory, network and storage into the actual guest operating system that is hosted on the hypervisor or VMkernel.  It also provides each virtual machine with its own personalized custom build BIOS!  The VMM detects and understands the hardware type that the hypervisor is running on.  It examines the advanced CPU functionality and then adapts (monitor mode) to pass along those benefits to the guest. 

The VMM handles three different types of virtualization, software, hardware and paravirtualization.  Software virtualization we already discussed above.  Hardware virtualization is leveraging technologies from our x86 based server vendors such as Intel and AMD.  Intel offers advanced processor virtualization features such as Intel VT-x, while AMD offers their own solution called AMD-V.  The hardware virtualization helps offset the overhead of virtualization by offloading the binary translation (BT) to the hardware.  Paravirtualization is the concept of reducing virtualization overhead by having both the host and guest work in conjunction with each other.  A classic example of this approach is pvSCSI, if you want more detailed information check out my write up over here.

As with all great idea’s come trade off’s, the use of the VMM adds a layer of “overhead” to the virtual machine.  There is a translation that has to take place to create this isolated secured environment.  The goal of VMware is to lower this overhead to help drive efficiencies (CPU and memory) and help you consolidate more with less.  Here is a diagram that helps illustrate this concept (BTW I have heard 4.1 has taken this down to 1-3% overhead):

 

Conclusion

Hopefully this helped shed some light on how VMware’s type 1 hypervisor works, and how it interacts with the virtual machines that is designed to support.  Remember that the VMkernel is responsible for working in conjunction with the hardware layer, and the VMM is responsible for translating that information to the virtual machine.  Overhead is a byproduct of this translation, but leveraging hardware and using VMware will help drive this overhead into complete transparency.

 

-Scott