Archive for August, 2009
The editor of Virtual Strategy Magazine recently asked me if I would be interested in committing to a regular monthly column for them. After thinking about it for a few days, I agreed and my first article in the series was published on Monday. The title of the column is going to be “Confessions of a Virtualization-aholic” where I’ll talk about real world experiences, with plenty of exaggeration and embellishment for effect.
You can follow my new column by visiting Virtual Strategy Magazine and this month’s article is here http://www.virtual-strategy.com/Features/Sweemer-20090825.html. I’ve also republished the article below.
The data center lights are brighter than the sun. The air is particularly stale and warm today. Beads of sweat are forming on my forehead. One bead grows too large and trickles down my face into my right eye. The salty liquid forces me to squint and temporarily blurs my vision. I hear Jamiroquai’s "Virtual Insanity" playing in the background. Sweet.
My vision returns and I notice a lone leaf of paper, like tumbleweed, dancing across the raised tile floor. It’s caught in the airstream of the temporary fans brought in to combat the blistering heat pouring off the mountains of servers. Off in the distance, I see an Oompa-Loompa doing a perfect pirouette. That’s weird. He’s kinda freakin’ me out. But nevermind. Back to the heat. It’s bad and it’s getting worse.
There are just too many servers. Some renegade, old-school wahoo added more hardware when the existing hardware is way underutilized. Brainwashed products of an era dominated by inefficient operating systems and incompatible applications, you can’t blame them anymore than you can blame a dog for being a dog. But placing blame aside, they have an incredible knack for making a horrible mess of things. I need to put an end to this.
I can feel the temperature rising. The data center technicians have all stripped down to their knickers. Long hair and pot bellies abound. It ain’t pretty. Except for Megan Fox. She’s hot. Evidently, in between filming scenes for her next Transformers movie, she moonlights as a server admin. Who knew? She turns to me and says "Aaron, you have to help us!" There’s an air of desperation in her voice.
"Don’t worry, miss…everything’s going to be alright."
In super slow motion, she flicks back her hair, gives me a sexy smile and a wink. I can’t disappoint Megan! Now I’m on a mission, the heat must come down. Servers must be eliminated. Everything must be virtualized!
It’s show time. Without hesitation I start to P2V everything in sight. The growing crowd starts to cheer and I feed off their excitement. Web, database, middleware, you name it … nothing is safe, nothing is sacred. I’m a P2V monkey, but instead of flinging poo, I’m flinging servers. One by one, each server meets its timely end. The heat is retreating. Trees are being saved! Energy execs are sobbing as their profits diminish with every dead server!
The pile of lifeless hardware is growing quite large. Each server bears an official death certificate which reads "Virtualized by VMware." Most of us cheer. But a few of the old timers hover over the steaming pile of scrap metal screaming "NOOOOOOOOOOOOO!" One of the poor pathetic souls shakes his fist at me and calls me a murderer. I can’t help but snicker.
Just then, Megan breaks through the crowd and comes running toward me. Oddly enough, her knickers are gone and she’s covered by nothing more than a recent issue of Virtual Strategy Magazine. Carryl Roy, Editor in Chief of VSM, yells "We’re only digital, not print!" To which I exclaim, "I really think you’re missing the point!" And at that very moment, Megan leaps toward me and lands with her lips in perfect alignment with mine.
"My hero!" says Megan.
"Sweemer’s the name, virtualization’s my game."
She moves in for a long, wet kiss. But before our lips touch, she pauses, gives me a funny look and says, "BEEP." Huh? That was weird. Well, it’s Megan Fox, she can say whatever she wants ‘cause she’s so hot! But let’s try that again.
Okay, Megan sweetheart, you really need to stop that. It’s distracting.
"BEEP BEEP BEEEEEEEEEEEEEEEEEEEEEEEEEEP"
My eyelids open and I’m staring at the rear end of my 10-year-old boxer, Lucy, who apparently managed to crawl into bed in the middle of the night. I reach over and grab my alarm clock and mid BEEP, I throw it against the wall. I’m pissed. I lost Megan and woke up to old stinky dog butt. But then I roll over and see my amazing, beautiful wife. Megan’s got nothing on her! And life is good again.
My wife opens her eyes and says "You were dreaming about Megan Fox again, weren’t you?"
"Why do you say that?"
"Because you woke me up 15 times last night screaming, ‘P2V me, Megan! P2V me, baby!’"
I try to conceal my embarrassment as my wife just giggles. But she does a great job of comforting me when she says "Don’t worry, baby. I dream about Megan Fox, too." Sweeeeeet. I love my wife.
Well, time to get up and get ready, I’ve got a long day, which begins with my Virtualization-aholic’s Anonymous meeting. It’s been rumored that Simon Crosby and Steve Ballmer will be there. Care to join us? We meet right here, once a month, at Virtual Strategy Magazine. You’ll laugh, you’ll cry, you may even hurl, as you hear weird, wacky and sometimes seriously disturbing Confessions of a Virtualization-aholic.
I wish I could take credit for the following work, but everything below is brought to you by Michael White. Michael is a co-worker of mine, an SE out of Canada who we often refer to as the “SRM King.” He continually impresses me with his ability to crank out a weekly news letter loaded full of great content. Well last night, he happened to mention I could republish his work on my blog. Shoot, you don’t have ask me twice!
Keep in mind as you’re reading, everything is a direct cut and paste. So anything written in the first person (e.g. “I have found …” or “I have decided”) would be referring to him, not me. I certainly don’t want to take credit for all his hard work!
If you have any questions or comments for Michael, feel free to leave a message for him.
Notes from VMware:
Cluster BP, FT and Issue, HA Issue, vDS Cheat Sheet, vDR Issue, YAPOTAV, vSphere Reference Card, View Design BP, SRM FAQ, and really a LOT more!
vSphere Cluster – ESX or ESXi or Mixed – suggestion / recommended best practice
We say that one day that ESX will not exist, and that ESX and ESXi are the same. Or almost the same. However, I have found in Host Profiles and FT there is very good reason to not mix ESX and ESXi in the same cluster. As soon as VMworld is over, I am redoing my mixed cluster to all ESXi (instead of mixed). First, we all know of the problem I reported some time ago that the 8/6/09 patches for vSphere would break FT in a mixed ESX / ESXi cluster. There is no short term solution for that. The workaround is to have a cluster that is all ESX or all ESXi. Second, host profiles have a problem dealing with service console / management network ports. In theory you can manage that by using a reference server that is ESX and it will translate as necessary for ESXi. It doesn’t do so well at that. So using Host Profiles to do a push of a distributed virtual switch (only) ends up causing issues in ESXi consoles. I ended up doing the ESXi hosts manually. The real solution to the FT and HP type issues is to have a cluster all ESX or ESXi. And I am voting for ESXi in my lab. Make no mistake, if you don’t listen to this you will have some issues that are not pleasant.
Using ESXi and ESX and FT in same cluster? And FT broke with the 8/6/09 patches?
The only solution to this at this time is to separate your ESXi and ESX servers into their own cluster, or upgrade one or the other to be the same as the other – meaning all ESXi or ESX and your problem should go away. If you have not installed the 8/6/09 patches yet, and you are using FT, and you have ESXi and ESX in your cluster than either change your cluster to be all ESXi or ESX and than install the patches. Not installing the patches until we fix this is NOT an option. I have decided, and as mentioned somewhere else in here, to redo my cluster as all ESXi. It won’t take much time. Some background on this issue can be found at http://communities.vmware.com/message/1335428#1335428.
Update on odd issue with HA not working if the vSphere ESX console was using certain IP addresses
I hope everyone has already heard that the vSphere bug talked about in http://kb.vmware.com/kb/1013013 and something I mentioned, I think in my last newsletter now has a patch. This is the bug that when a very specific IP address scheme is in use on management ports / service console with no other IP schemes in use and a host crashed, the VM’s that should have been started by HA would in fact not be started at all. I have not tested the fix, as I am wrestling with SRM and trying to get ready for VMworld. To avoid this bug, only one of the addresses on your service console or management ports need to be using something outside of the ‘special’ scheme.
vDS Implementation Cheat Sheet
I worked with the distributed switches in the past in a lab sense, but recently. For my future SRM testing, I got it going for real in my lab. And it was hard, confusing, and not intuitive at all. So I wrote a cheat sheet so you would not have to suffer. It is attached. I have used it a few times and am happy with it so hopefully it will make things quicker and easier. Let me know if you need improvements or changes in it. http://www.virtualinsanity.com/wp-content/uploads/vDS-Implementation-Cheat-Sheet-b.pdf
Data Recovery Issue – which stops backups from happening
If you ever have an issue with writing to your destination when doing backups, you may see the restore point in red with a (Damaged) beside it. This can cause your backup to not work again. The events part of the Reports will show file access errors – 3902. The solution to this is not in the documentation for vDR but it is here. Expand the display of restore points to be bigger than the default 5. I used 25 when I had this issue. Now click all of the restore points that show as damaged. Then select the Mark for Deletion button in the top right of the screen. Now change to the Configuration \ Destinations screen and select the destination that is associated with your backup, and use the Integrity Check option near the top right of the screen. It will take a while. Once it is complete with no errors – check the Events view of Reports – you need to restart the appliance. Now your backups should work!
YAPOTAV – Yet another post on why to attend VMworld
New vSphere document reference card
Forbes Guthrie has done a wonderful job on a reference card for vSphere documentation stuff. It pulls stuff out of the documentation and highlights it as a result. Very handy and well done. Find it at http://www.vreference.com/public/vsphere4-notes1.0.pdf
View Design Best Practices training
Would you like to learn more about designing a View infrastructure? The more people you have that depend on it the more important training and experience becomes. Get some ideas on design at http://mylearn.vmware.com/descriptions/EDU_DATASHEET_ViewDesignBestPractices_V3.pdf
SRM FAQ online now thanks to Duncan at Yellow-Bricks
This is from information I have shared with Duncan but it is great information and I appreciate him sharing with everyone. Find it at http://www.yellow-bricks.com/srm-faq/. Duncan’s web site is one of the few you should read frequently. He is a PSO guy in Europe and is very smart, and knows what to communicate – does it real well and I appreciate it.
vSphere and VM snapshots and block size
This is something else that Duncan has done. There is a behavior difference between 3.5x and 4.0 that could catch someone. Find out more from Duncan at http://www.yellow-bricks.com/2009/08/24/vsphere-vm-snapshots-and-block-size/.
VMware View Cheat Sheet
I have had some help to update my VMware View Cheat Sheet and it has gone very well. Our next update of this will have a lot more but this is a good document to get you going with View. www.virtualinsanity.com/wp-content/uploads/VMware-View-Cheat-Sheet-a.pdf
Important patch for Celerra when using NFS with VMware
You can find more information about this at Virtual Geek, but it is important to understand that you need to upgrade your Celerra DART OS before you enable NFS datastores with VMware. Find out more at http://virtualgeek.typepad.com/virtual_geek/2009/08/important-patch-for-celerranfsvmware.html
Lab Manager 4 Upgrade issue
The installer during an upgrade of LM4 assumes all the default roles are present and unmodified. If the customer removes or changes any the upgrade installer will fail.
FT – Architecture and Performance
Do you know how to determine how many FT enabled VM’s your vSphere server can support? Do you know how to design your FT environment for the best performance? In fact, do you know what the performance overhead for FT is? All of this and more is answered in http://www.vmware.com/resources/techresources/10058.
How can I determine the exact build number for my ESX 4.0.x hosts?
You can find out the way to determine the build numbers for components of ESX 4.0 hosts at http://kb.vmware.com/kb/1012514
VMware Data Recovery Evaluator’s guide
This is a very nice document for someone who needs some guidance for testing VDR. It is a quite way to get started. http://www.vmware.com/resources/techresources/10055. My preso on VDR at VMworld is a combination of install / config / best practices and it will be very useful. Look for the session, or the preso after VMworld. It will fit with this eval guide nicely and is known as BC2142.
AppSpeed and Maintenance Mode
Currently AppSpeed has no when to listen to the ESX host it is working on, so when the host tries to enter Maintenance mode it will not be able to since the AppSpeed sensor VM will not listen to it and it will not VMotion off the host. This is a very high priority for us to fix. You will need to manually turn off this sensor before trying to do maintenance mode.
Need some help searching the VMware KB? Find it at http://xtravirt.com/xd10112 – some interesting info.
NFS Storage Configuration Help
Do you need some help configuring NFS support for your ESX servers. There is some help at
http://communities.vmware.com/docs/DOC-7900. This link has only a little info but it does include some troubleshooting info.
VUM and Cisco – conflict message
I got a conflict message from VUM when I tried to patch recently. It was a conflict with the Cisco Nexus stuff which I do not have installed. It turns out that I could just ignore it but it was a little bothersome. We are going to change that message in the near future to be more informative. That way if you know you don’t have Cisco (or whatever) installed you can just install with no issues. The issue is we download all the meta data or patches for ESX without any granularity. So the Cisco patches come done too. More info can be found at http://kb.vmware.com/kb/1013068.
Suggested VMware Employee Sessions at VMworld
This is a list that one of my co-workers put together. It might give you some ideas of what to look for.
- Michael White – BC2142 – Data Recovery intro and best practices
- Tiffany To – DV1790 – View TCO-ROI expert
- Mahesh Ramachandran – VM1724 – Capacity IQ Tech Preview
- Chris Rimer – EA2342 – Oracle sessions (especially around questions of Support and Licensing)
- Richard McDougall – TA3438 – vSphere Performance Guru
- Jacob Jensen – TA2103 – Virtual Networking guru (especially around the Cisco v1000)
- Andy Banta – TA3264 – iSCSI Best Practices (THE iSCSI Engineer/Expert at VMware!)
- Kaushik Banerjee – TA2942 – Performance Best Practices (This guys is a genius in performance and on the Perf. core team!)
- Paul Manning – VM3566 – Storage Best Practices (Many of you have been on calls with Paul for storage related topics!)
- Brian CS, Charu Charubal, and Rob Randell – VM2847, TA2544, DV2626, – Security Team extraordinaire
- Mostafa Khalil – TA2509 – Storage Best Practices (Mostafa is one of the first VCDX members!)
- Amir Sharif – TA3195, V13226 – ESXi PM – ESXi sessions
- Monica Sharma – VM2408 – ConfigControl Tech Preview
- Bill Call – VM2657 – LifeCycle Manager Uber-Guru!
- Dean Flaming and Travis Sales – DV2478 – ThinApp (These are some of the best sessions I have ever seen historically from these guys!)
- Gaetan Castelein – EA3605, EA 3606 – Virtualizing Tier 1 applications –
- Srinivas Krishnamurti – VM2280 – Managing VI from your mobile phone!
- Duncan Epping – TA2259 – Expert VI Design (Duncan runs the #1 Virtualization blog “Yellow-Bricks”)
- Dean Yao – BC3369 – FT Real World design
- Howie Xu – TA3521 – vNetwork Troubleshooting (Howie invented the vSwitch! – and wrote one of our TCP/IP stacks)
- Banjot Chanana – BC3425 – High Availability Futures
- Nicholas Jacques – PA4694 – AppSpeed PM
- Eric Horschmann – TA3880 – vSphere vs Hyper-V/XenServer
- Warren Ponder – DV2697 – View /VDI PM
- Mike DiPetrillo – TA3326 – Cloud (Mike is another uber-rock star and talks all things Cloud!)
- Rahul Ravulur- -VM4380 – vCenter PM covering future of vCenter
- Naeem Malik – VM3609 – Capacity Planner expert
- Aaron Sweemer – DV3567 – How to convert old PCs to Thin Clients using a thin Linux OS and VMware View Open Client.
**** Reminders ******
- vSphere Upgrade Center – http://www.vmware.com/products/vsphere/upgrade-center/
- vSphere Support Center – http://www.vmware.com/support/vsphere.html
- vSphere-land documentation links – http://vsphere-land.com/vsphere-links/documentation-links-2.html
- Connections and Ports in ESX / ESXi – http://www.virtualinsanity.com/esx-connections-and-ports/
- vSphere-land download links – http://vsphere-land.com/vsphere-links/download-links-2.html
- vSphere and other VMware products compatibility matrix – http://www.vmware.com/resources/compatibility/docs/vSphere_Comp_Matrix.pdf
- VMware TCO / ROI Calculator (for VI 3 consolidation, virtual lab automation, and VDI savings) – http://www.vmware.com/products/vi/calculator.html
- VMware Cost per application calculator – http://www.vmware.com/technology/calculator/costperapp.html
- Why choose VMware? – http://www.vmware.com/technology/whyvmware/
- Server Virtualization Validation Program – you can check the status of servers or virtualization technology approved by Windows in the SVVP program – http://windowsservercatalog.com/svvp.aspx?svvppage=svvp.htm
- Support policy – http://support.microsoft.com/kb/957006
- VMware Visio shapes at http://www.veeam.com/vmware-esx-stencils.html and http://engineering.xtravirt.com/products/vi3-visio-action-pack.html
- VMware graphics for PowerPoint – http://viops.vmware.com/home/docs/DOC-1338
- License issues – call 877-486-9273 and select option 1 – or firstname.lastname@example.org
- Interesting SRM links – http://tendam.wordpress.com/srm-links/
- Practical DR (before SRM) – http://www.vmware.com/files/pdf/practical_guide_bcdr_vmb.pdf
- VMware Security Center – blogs, RSS feeds, and more – http://www.vmware.com/security
- Enhanced VMotion Compatibility (EVC) processor support – http://kb.vmware.com/kb/1003212
- Patch Notice (via email) is available at http://www.vmware.com/support
- Reporting bugs to VMware – http://www.vmware.com/support/policies/defect.html
- Requesting a specific feature – http://www.vmware.com/support/policies/feature.html
- License File checker – http://www.vmware.com/checklicense/ – doesn’t work with SRM license codes yet!
- Dell and VMware alliance – includes white papers and VMotion compatibility info – http://www.dell.com/content/topics/global.aspx/alliances/en/vmware_resources?c=us&cs=555&l=en&s=biz&~section=004
- HP Certification matrix for VMware on Proliant – including VMotion compatibility info – http://h18004.www1.hp.com/products/servers/software/vmware/hpvmotion-compatibility-matrix.html
- Emma Explains Microsoft Licensing in Depth – A well done UK site that actually seems to make sense when it comes to MS license info – http://ladylicensing.spaces.live.com/default.aspx
I got an email from Dudley Smith (a VMware TAM and the author of Troubleshooting ESX and Connections & Ports in VI3.5) informing me that he had recently updated one of his documents. Wow, he sure did. Check this puppy out (click the graphic to download) …
Pretty slick, eh? Well it gets even better. He also created a version using The Brain in HTML … http://www.virtualinsanity.com/esx-connections-and-ports/. Nice! This is definitely a bookmark I’ll be keeping handy and I’d recommend you do the same.
Good work Dudley! Thanks for making it available for everyone. If you agree, be sure to leave a “Thank You” comment for Dudley Smith.
I’m not sure if anyone noticed (or cared), but the blog was down over the weekend. The reason is that I run my blog server out of my house and on Saturday we moved to a new neighborhood in Cincinnati. And while virtualization can do wonders for server availability, it can’t do much when your servers are sitting in the back of a moving truck!
Man, what an ordeal. Moving is such a pain in the butt!! Now, I’ve moved plenty of times in my life, but it’s been as a bachelor and I didn’t have a lot of stuff. Now that I have a wife, son and two dogs … holy cow!! I think I said the phrase “how in the world did we get so much crap?!” about 400 times over the past three for four days. But now we’re all settled in to our new home in Hyde Park, I got my Time Warner Internet set up and I got my servers unpacked and powered on in the basement.
My wife and I are excited about this new area. Hyde Park is a very cool place to be. The picture above is of Hyde Park square and was taken in 1901. Here’s a quick blurb from a Hyde Park website …
Hyde Park was named after Hyde Park in New York and is one of the 52 neighborhoods that make up Cincinnati located on the east side of the city accessible via I-71. It was once a suburb before being annexed by Cincinnati and is the wealthiest neighborhood in the city. It is home to stately, well-maintained homes with manicured lawns and tree-lined streets. The business district is called Hyde Park Square and offers an array of upscale restaurants, boutiques and galleries.
This morning I walked down to the Hyde Park square, which is about three blocks away from our new house, and got some coffee for my wife and I. Last night, we walked to Beluga, my wife’s favorite sushi restaurant. This was NOT possible in our old house because we were way out in the country. Of course, this kind of convenience could be dangerous for the waistline, and the wallet!
Anyway, the blog is back online. BUT, if you’ve been trying to email me at my sweemer.com or virtualinsanity.com email address, then I have not been getting your messages. See, I *thought* I had ordered business grade Internet service from Time Warner which would allow me to run a mail server. But evidently I got the consumer grade service, which blocks port 25. I corrected the problem yesterday, but it will take a few days for the service to get upgraded. So for the next few days, email still won’t work. Until then, I can be reached at asweemer [at] gmail [dot] com.