Author Archive

Several months ago, a small firm I consult for ordered a Drobo Elite (recently replaced by the B800i).  These guys had run ESXi for a while in one of their environments, and were wanting to explore some of the features requiring shared storage.  Like most small businesses, they wanted to get there without breaking the bank.  There aren’t a ton of options in the $6-7k range for iSCSI arrays on the VCG, so it was an easy choice.

Their CIO called up Drobo and placed the order.  He explained what they were going to use it for, and the guy configured it right over the phone and shipped it out.  A few days later, the Drobo Elite arrived configured with 8 x 2TB Western Digital (WD20EARS) disks at a cost of just under $6k.

Setup in ESXi was straight forward.  I followed the documentation from Drobo and set the PSP to VMW_PSP_MRU and SATP to VMW_SATP_DEFAULT_AA and started throwing VM’s on for testing.

The initial tests were okay.  I wasn’t really bouncing around the room yet, but I am used to larger FC array speeds.  Once I saw that IOmeter was pushing the expected number of IOPS, we were ready to throw on a few VM’s.  For some context, we’re talking about a 100 person company with about 20 servers in total.  They’re running 50% of those on ESXi right now on two hosts.  Once normal daily production started with 3-4 VM’s hitting the Drobo, everything screeched to a grinding halt.

Latency, as reported in ESXTOP, was showing 4-5000ms, and there wasn’t any single workload that was giving it a tough time.  I went back in and double checked the iSCSI config.  All the bindings were correct, as were the PSP and SATP.  Nothing had changed except adding a couple more VM’s to the Drobo.

I began to suspect the switch was misconfigured, so I pulled it out, and went direct to the Drobo.  That didn’t really yield a noticeable improvement.  After troubleshooting this forever, and deliberating on the phone with Drobo, they announced their verdict.  Apparently the WD “Green” drives are not supported with VMware.  They said we’d need to buy the Black drives.

Their site quickly confirmed.  But again, since Drobo configured the unit, knowing it was for a 2 host VMware environment, we both assumed the Green drives were sufficient.  The extra cost of the Black wasn’t warranted for this environment.  I could understand if the customer had gone out and bought some random drives, but they came with the unit directly from Drobo.

They had us run some of their own IOmeter tests directly connected from a Windows box using the MS iSCSI initiator.  We then went ahead and swapped the disks for the recommended WD Black disks, and below are a few charts showing the results.

 

The Black is faster in every way, but the most noticeable aspect is write latency.  I suspect this is due to the increased processing power and faster cache.  Nevertheless, the results speak for themselves.

Bottom line is, if you’re going to run ESXi on a Drobo, don’t go green!

 

BREAKING – PALO ALTO (VP)

The VMware licensing debate was killed this afternoon while trying to rescue the #vTax hashtag from the inside lane of the Ridiculous Interstate.  Witnesses say a bearded, balding “smart-looking” man was driving north at a very high rate of speed in a truck with the license plate VMW when the debate was struck.  The truck backed up and struck the debate again and again before authorities arrived and pronounced the debate dead at 5 PM PDT today.

I am writing this as a blogger at Virtual Insanity, and a customer of VMware. I don’t sell VMware, and I’ve never worked for VMware. I don’t even work for a partner. I barely get to chat with my fellow bloggers who work for VMware, and am certainly not privy to inside information, despite my company’s NDA.

With that out of the way, VMware has done the right thing here. The fact that they can take customer feedback and mold it into a dramatic licensing change, just a few weeks before a product GA’s, is astounding. That speaks not only to the agility of the company, but their willingness to please their customers.

They even went out of their way to please NON-PAYING customers with this change. The change to the free version was causing more drama than the change to customers who spend millions with VMware.

Should VMware have focus-grouped the licensing change more than they did? Yes. It would have preempted the customer perception wildfire they have had to fight for the past couple of weeks. I am sure they ran the numbers and knew that only a small percentage would be impacted. But the fact is an even smaller percentage actually ran the scripts to see how it would affect them. Once the fervor got started by a few, it wasn’t going to stop.

A price increase was inevitable. VMware has given us HUNDREDS of new features in the past several years for free. I think not increasing it with 4.0 was the right move, but they couldn’t hold out forever. The new vRAM allotments and policies are spot on, and are going to put a lot of customers’ fears to bed.

Now we can get on with discussing the amazing new features of vSphere 5.0 without that licensing cloud hanging over our heads.

Kudos VMware!

 

Recently I have been researching HP C7000 chassis connectivity options extensively.  Prior to diving deep into it, Virtual Connect FlexFabric seemed like a no brainer.  On the surface, it has many advantages.

The cabling / port reduction is an obvious win, as is the ability to have some control over WWID and MAC assignment to blades. Moving East / West traffic between chassis without having to go Northbound to a ToR or EoR switch is attractive as well.  Of course these are all things that are just standard with UCS, but I digress.

After many meetings with HP, I still had some questions that were unanswered.  I turned to the many thousands of pages of HP documentation on the subject.  Sifting through all the “cookbooks” and the in-depth guides to Virtual Connect, and talking with some current users of FlexFabric, I came to the conclusion that it is missing some key features that are needed in a VMware environment.  In fact, I would say that for Cisco shops running VMware, HP FlexFabric makes little sense.

The biggest problem I have with Virtual Connect FlexFabric is the lack of any real QoS.  Once traffic enters the Virtual Connect module, it’s anarchy.  There are no controls in there for prioritization or control of bandwidth.  In a VMware environment, where there will be multiple types of traffic, each capable of generating significant load, the only control you have on VC is egress rate limiting.

It’s akin to limiting the number of people one can put in a single car, right before driving through the middle of Rome.

For those who haven’t had that experience, trust me, it’s the same type of anarchy that occurs inside VC.  The only rule is try not to die.

 

 

Here’s a nice diagram showing Virtual Connect and VMware traffic flow design from M. Sean McGee’s blog:

When you have a Cisco 1000V on the ESXi host and a Nexus 5K on the other end, it makes little sense, in my opinion, to completely break awesome features like Priority Flow Control and Bandwidth Management.  HP states that they do support FCoE and DCB (CEE), which should include the above features, but their own guys cannot really say how one would configure, or troubleshoot it.  That’s part of the problem.  VC is a black box that abstracts your ability to see what is going on inside.

One of my other negatives for VC FlexFabric is that I have no choice but to split my 10GbE pipe into smaller pipes if I want to run an HBA off the adapter.  If I use the exact same onboard CNA without FlexFabric, I don’t have to do that.  This can be solved with separate HBA’s, or 10Gb NIC’s, but that negates the alleged cost savings.  So now I’m forced to try and guess how much bandwidth I need for each traffic class, when I already own switching infrastructure that is smart enough to do that for me.

In my opinion, this is akin to disabling DRS.  DRS is smarter than you, and faster.  Why would anyone disable it?  Cisco QoS is certainly smarter than me, as is VMware NetIOC.  So why would I want to throw some arbitrary limits on my huge pipe?  VMware admins understand that shares are better than reservations or limits.  The reasoning is the same on the networking side.

There are other problems I see with this solution, but I don’t want to bore you.  One complaint I have heard from close associates is the HP recommended method of “stacking” VC modules is problematic.  Not only do you have to give up 3 of the 8 ports per module for stacking, but it can create bandwidth issues as well.  Recently, a friend of mine had to completely revamp his setup to uplink everything, as opposed to stacking, which was allegedly causing bandwidth problems in his environment.  Ohh, and in addition to all this, the FlexFabric module will take FCoE and pass it North as standard Ethernet.  So you lose any of the FCoE features provided by your Nexus switch.

Companies that are not virtualizing certain applications, but will run them on blades, may find that the advantages of moving around MAC and WWID’s outweigh the potential disadvantages of FlexFabric.  Everything on my blades will be ESXi, so I don’t really have a need for quick physical ID recovery.

As of right now, I plan to use passthrough modules on the C7000’s.  At least until a better alternative comes out.  Passthrough is slightly more expensive on the uplink port side, but it doesn’t prevent my networking team from having end to end visibility and management.  And that takes some of the guesswork, and the administration off of my team, which is a good thing!  I would be interested to hear your experiences in the comments below.

 

Cisco decided to shut down Flip last month. Why? Because it’s a low margin business that Cisco has no business owning. There is talk about killing Linksys, or spinning it out. Why? Low margins and it doesn’t jive with Cisco’s core competency. UCS (datacenter unified computing system) is another product that has very low margins, and really should be sold if Cisco is to remain as strong as it has been over the past two decades.

 

I find it interesting that only a year ago, all the industry pundits were talking up Cisco and their stock was riding high. How quickly the sands have shifted under their feet. Shareholders and industry experts are calling for Chambers to resign, and some have even suggested they get rid of UCS. Last week’s Infosmack featured some interesting commentary on Cisco selling UCS. GigaOm thinks Cisco has lost that lovin’ feeling for VCE. They seem to be investing as heavily as EMC, but they get a much, much smaller piece of the pie on all those sales. And let’s face it, VCE sales are expensive. Maybe Cisco should have bought EMC when they had the chance?

 

I thought Robin Harris’ comment over on Storagemojo was profound:

UCS lowers Cisco’s margins; enrages large resellers; and has no sustainable competitive advantage. Cisco can’t wish those facts away, and the stock market won’t forget them either.

 

The sustainable competitive advantage thing is a big one.

 

Even with the latest IDC report showing that UCS has overtaken Dell to become the #3 blade player, there is still plenty of uncertainty in the market. I can say from my own experience that executives, who admittedly know very little about UCS and what it brings to the table, are shying away from it out of fear that Cisco could exit the server business.

 

From the very beginning, there was talk of Cisco not being “serious” about becoming a server vendor. Add the recent stock troubles, and decision makers are less willing to stick their necks out on millions worth of UCS. After all, nobody has ever been fired for buying IBM.

 

Companies often take a bath when they get into areas that go against their long standing value propositions. BMW lost $4 Billion when it sold Rover to Ford for $1. Cisco spent $600 Million on Flip only 2 years ago. The fact that Cisco first approached IBM and HP with the UCS idea, and was rejected only proves that Cisco knew it didn’t want to be in the server business before it . . .got into the server business. Perhaps now that they have made their point, one of the server vendors will be interested in a UCS purchase.

 

With HP getting amazingly aggressive on pricing of their network offerings, and Juniper introducing QFabric, Cisco’s attention needs to be focused on their core competency if they wish to maintain those luxuriously high margins into the future.

 

I am sure it comes as no surprise to any of our readers that virtualization is not the exclusive full-time focus for most of us.  Most of us have a breadth of responsibility spanning gobs of infrastructure layers in our respective organizations.  One common pain point that most of us have is backups.

For many companies, backup is an afterthought.  It doesn’t contribute to the profitability of the company.  It doesn’t help you make more widgets in the same amount of time.  The result often times is a neglected backup system when it comes to budgets and spending.  Most of the time, even though we know the importance of backups, we’re okay with it taking a back seat.  After all, who wants to goof around with tape drives when there are cool new blades and SSD storage to play with?

It was this frame of mind that I found myself in on Tuesday of this week.  I had signed up for W. Curtis Preston’s Backup Central Live a while back on Stephen Foskett’s recommendation.  I knew it would be decent, as I had used Backupcentral.com for a long time as a valuable resource to help deal with those dreaded backup problems.  But when Monday came, I found myself wondering why the heck I signed up for this seminar.  I had so much work to do this week, and most of it was fun SAN and VMware planning and design stuff.  I didn’t have time for baaaaackups. . . Grrrr.

In the end, my boss was pumped about the seminar.  I knew I couldn’t back out without getting grief, so reluctantly, I made the 1.5 hour drive to Cary, NC for a full day of backups.  I knew Curtis would be a great speaker, and have good insight.  I have heard him many times on Infosmack, and I know from his blog posts that he knows his stuff.  I just wasn’t looking forward to a full day of vendor pitches between the valuable information.

Ultimately, I was impressed with the event, and it was far from a waste of time.  Even the vendor presentations were decent, and they kept to a reasonable time limit, so the pace was perfect.  I’ll give you a quick rundown of what I learned at this event.

Often times we feel alone in our backup struggles.  At the seminar, there was wireless polling during the presentation, so we had real time answers to our questions.  That alone was a fantastic change; and I prefer this to raising my hand 48 times during a session.  From this polling data, I learned that I am not alone.  Many share in my misery.

• 49% of attendees still do backups DIRECT TO TAPE.

So while us 49% think that no one hears our screams, at least now we know that we’re not the only ones screaming.  I think we all know that tape is not a suitable target for server backups.  The problem only gets worse as tape drives get faster.  Disk, at least as a staging area, is a necessity now for reliable backup to tape.

That said, Preston points out that tape is a long way from being displaced from the datacenter.  Tape is still 50x cheaper than disk, and more reliable for long-term data storage.  One fact I found enlightening was that hard disks are not designed or tested to store data long term while powered off.  This is something I had never thought about, and only a couple of companies, like ProStor, are trying to solve this problem.  Even if we solve for the reliability difference, it will likely be decades before we see a significant degree of cost parity (if ever).

A speaker from Cambridge Computer Services talked about new cool ways people are using tape as part of a tiered strategy for primary data.  Some are even using tape as a mirror for their primary storage.  Of course this requires a gateway appliance with plenty of cache, and good software, but the savings are real.

Another crucial area we touched on was that of archival, especially as it relates to electronic discovery (ED).  Almost NO ONE is doing this.  The vast majority is using their primary backup software and methodology for archival.  This is an expensive mistake if you ever are called upon to do discovery.  In addition to my own experience with ED, Preston tells a story of a client who spent millions to satisfy a single discovery request.

Apparently a single user’s e-mail for the past three years was requested.  As they were only doing normal Exchange backups, that meant restoring 156 different monthly Exchange backups, and then fishing for this guy’s mails.  It took an army of consultants working three shifts MONTHS to do this.  Since we live in a litigious world these days, it might be a good idea to get your ED and archival in order.  One product that was recommended at the seminar was Index Engines.  I haven’t had time to look at it yet, but it sounds brilliant!

One interesting statistic we saw in the polling data was that the majority of attendees had an overblown opinion of themselves when it comes to their own backup environments.  The majority said their backups ran well.  Preston’s experience tells quite a different story.  The scary part of this is that people don’t know that their backups suck.  They find out when it’s too late.

The most valuable part of this seminar was the discussion time at the end.  There was many interesting discussion around cloud backups, AWS outage, and snapshots.  This brought everything together that we had learned during the day.

There isn’t space in a single blog post to cover all the material from a full day seminar, but I hope I’ve given you enough to help make a decision to check this event out when it comes your way.  I have to give it to the Backup Central Live crew for taking a topic that most people hate, and turning it into a valuable day of learning.

A couple months ago after the introduction of the new EMC VNX arrays, I posted my thoughts on it here.  One of the engineering choices I questioned was the use of SSD’s for extending cache versus a PCI card.  It was always obvious why it would be better when cache was being added or replaced, but I questioned the throughput potential of a SAS interface versus a PCI one.

 

I got some interesting feedback on that from several people, and I appreciate it.  It wasn’t until the other day that I realized that the argument really did not have that much merit.  In a moment of blinding brilliance, I realized that the only time this might make a difference is when warming the cache.

 

How did I come to this realization?  I was in a VNX deep dive session presented by Chad Sakac, and I had every intention of asking him the question of PCI versus SAS when it comes to cache.  Lucky for me, he brings it up during the session, before I could ask.  Before he was done with the rest of the presentation, I realized an error in my prior way of thinking.

 

Chad pointed out that the time it takes for an IO to go through the controller, loops, and hit the flash is measured in nanoseconds (10^-9).  Once it’s there, the flash has latencies in the microseconds (10^-6).  So there is not likely to be a significant difference in latency between SSD, and PCI when it comes to cache.

 

PCI obviously has greater throughput potential, which is why I previously asked the question.  But a realization jumped up and bit me while I was sitting through this presentation.  Cache IO’s are usually small chunks of data that benefit from the reduced latency of flash / DRAM.  They aren’t giant read / write operations that generally require extremely wide bandwidth.  Will the increased bandwidth of PCI make a difference?  I have my doubts that it will be noticeable on the vast majority of workloads.  But this is just my opinion, as an outsider without the benefit of a storage engineering background.

 

I am looking forward to seeing the SPC1 benchmarks from the VNX.  I believe it will objectively tell the whole truth.  A slight difference on an anomalous workload is not significant enough to outweigh the benefits of SSD versus a PCI cache.  It’s easily swapped, and it’s non-volatile.  It only needs to be warmed once.  If a controller fails, the cache doesn’t die with it.  Replace a controller, and no need to rewarm cache.

 

Like I was alluding to in my last post on this. . .every design decision, whether it is in storage engineering, vSphere design, automobile design, is one of compromise.  The SPC1 will tell the whole story, but I think what we’ll see here is that this particular compromise was overall a good one.  What do you think?  Let me know in the comments.

I’ve been running some tests in the lab lately, and trying to solve a problem that I don’t think is solvable right now. I’m hoping some of our readers will point out a potential solution that I have missed. Kendrick Coleman posted a write-up of how VM performance can be impacted by VM placement within the cluster.  This is almost exactly what I have been testing in my lab, with a few twists.

As Kendrick points out, VM’s that need to communicate with one another regularly are better off on the same ESXi host.  With VMXNET 3 NIC’s, one can achieve massive throughput between VM’s on the same host.  However, that is not always the case.

The issue I am running into as I design my production environment is a requirement to have everything segmented off into hundreds of VLAN’s.  This means that there will be servers on the same host that are on different VLAN’s that will need to communicate, sometimes frequently.  This completely negates the benefit of having the VM’s on the same host, as the traffic will have to leave the box to be routed.

Here are some tests I did using iperf from the VM Advanced ISO v0.2 just to further expand on the idea:

 

2 VM’s on same host / same VLAN

 

2 VM’s on same host / different VLAN

 

2VM’s on different hosts / same VLAN

 

2 VM’s on different hosts / different VLAN

 

As you can see, it makes almost no difference that VM’s are on the same host, when the VLAN’s / subnets are different.  Just for fun, I bumped the TCP window size, and was able to achieve 3.5Gbps from VM to VM on the same host, and the same VLAN.  When the VLAN is changed, the ratio of slowdown is the same regardless of host affinity.  This is because traffic is leaving the host, going all the way to my Cisco 6509, and coming back into the same host.

Just for reference, all the hosts in these examples are connected to the same Cisco 1GB switch.

I brought this up with Cisco when they were in talking about UCS.  They were mentioning their roadmap, and virtual appliances, so I thought it was a good time to ask whether there would be a virtual layer 3 appliance in Cisco’s roadmap.  The response was about what I expected.

Even if I go UCS, which brilliantly handles east / west traffic across multiple chassis, the top of rack 61xx Cisco devices don’t route, so I’ll still have to go all the way out to a 5000, or soon a 7000, to get routed back into the same host on the same wire.

Talking with a few friends who know more Cisco than I do, we discussed the idea of a virtual router.  The inherent problem with a virtual router in this environment is the VM is still bound by its default gateway.  When DRS runs, and moves VM’s around, now a VM is living on a host that does not have the particular virtual router with his gateway interface.  That defeats the purpose.

We talked about how it might be possible to work around this using Cisco’s Gateway Load Balancing Protocol (GLBP), but even then, you’d have to set preferred active paths, and it wouldn’t always work the way we need it to work.

The only solution to this issue I can think of is a Distributed Virtual Router, which doesn’t exist.  If someone could make a virtual router that operates like the Distributed Virtual Switch, it would help all us people out here in the financial world who are ever more constrained by tons of VLAN’s, and (virtual) firewalls in between those VLAN’s.

Is there a need for this in the marketplace?  Or am I making a bigger issue out of this than I should?

As always, your comments are appreciated.

 

I have run the entire gamut of virtual SAN appliances so far in my VMware lab environment, and I always come back to the Celerra UBER VSA.  The best thing about this is that it’s free, and there’s no expiration.  The LeftHand is easier to use, and has some neat clustering features, but it’s a 60 day license.

I don’t know about you, but there are times when I get busy, and don’t get a chance to touch the lab for weeks.  60 days is just not enough.  I got the NetApp ONTAP 8.0 appliance as well, but it’s a pain to use unless you’re running VMware Server (who still runs that?) or Workstation.

Anyway, I’ve been struggling with the performance of the UBER VSA, and trying hard to find a way to make it faster.  Thanks to Clinton Kitson, I was able to dramatically improve throughput and latency on my Celerra VSA.  Time to deploy a VM from a template went from 30, down to 9 minutes.

Clinton says they may try to include this in the next UBER version, as long as there is consensus that it is beneficial and does no harm.

Here’s the tweak:

Login to the VSA using the root account.  Password is nasadmin.

Type in this command: /sbin/swapoff -a

Then, go in and edit the sysctl.conf file using this command:

nano /etc/sysctl.conf

Add the following lines to the end of the file:

vm.dirty_background_ratio = 50

vm.dirty_ratio = 80

Ctrl+X to exit out.  Save the file over the existing sysctl.conf.

Restart your VSA.

The screenshot below shows my VSA.  I had already changed the caching settings, but here’s what happens by simply turning off the swap.  You can clearly see the huge improvement in both latency and throughput.

Basically what we’re doing with these commands is telling the VSA not to swap.  We’re also changing the way the underlying RedHat OS caches data before it writes to disk.  Be aware that this does increase the risk of data loss, as we’re caching much more data in RAM before it’s written to disk.  If data loss is a concern in your lab, you may want to stick with the standard settings.  Also, my VSA has 6GB of RAM allocated.  Still only a single data mover.  Obviously more RAM = more performance when we’re turning up the caching.

Thanks to Clinton for pointing out these settings.  It’s hard to find performance information on the EMC VSA.  I hope this post helps you get more work done in your lab environment.

Yesterday I had a marathon five-hour executive briefing with EMC, and I learned a lot about the VNX, and EMC’s strategy going forward.  This was good information, and I want to give my take on what I learned, as well as maybe open up some discussion in the comments.

First off, I have to say one thing about EMC.  Regardless what you think about their storage technology, and product line-up, their sales is absolutely fantastic!  I have yet to experience pre-sales as good as EMC.  They are polished, professional, and knowledgeable about their product.  They brought in a wide array of talent, including a vSpecialist, and there was not a single question they could not answer.  This is impressive, considering some of our questions were a bit. . . creative.

Aside from all this, I am most impressed with their ability to put in an awful lot of work to analyze my current environment, and figure out exactly what my needs are.  No one else has even come close in my recent experience.  This alone goes a long way toward overcoming what could be perceived as product weaknesses versus their competition.

In my opinion, the VNX launch is an attempt at addressing the lessons learned by EMC over the past several years.  I believe we can all agree that EMC was caught off guard by the ability of other vendors to integrate more tightly with VMware than VMware’s majority owner.  Their recent moves have shown their willingness to correct that, and a strong desire to leapfrog the competition.

Considering the combination of VNX, recent tighter integration with VMware, and the intense growth of EMC’s battalion of vSpecialists , one can only presume that they are willing to use brute force to become the VMware platform of choice.  I find it remarkable that a company with as many divisions and products can move with the degree of agility they have shown recently.

VNX introduces some interesting changes besides the concept of a truly unified platform.  Fiber Channel is out, and SAS is in.  This makes sense to me, considering the current pricing trends of SSD.  It no longer makes as much sense to go with FC as a tier of storage.  When SSD was 30x more expensive, not many considered it a viable alternative to FC, despite the huge performance advantage.  These days, it’s maybe 3-5x more expensive, and that’s easy to justify with the performance.  The price should continue to plummet as more manufacturers come online with more fab plants, so it won’t be long before there is price parity between FC and SSD.  This was the right call, in my opinion.

A new version of FAST VP is another significant change introduced with VNX.  When you lose an entire category of disks (FC), and only have SAS and SSD, you can eliminate the issues with FAST where you still could have hot spots on slower disks, and cool spots on your faster disks.  Now there are only two tiers, so there should not be an issue with data finding itself on the wrong tier.

There is no way I can go through all the features and software changes introduced with VNX, and that is not really my intent.  I do want to flesh out one area where I see a potential design flaw.  Of course, this is all based on my opinion, and I am not in the same league as EMC, NetApp, HPar, or XioTech storage engiNerds, so take it for what it is worth.

The whole idea of FAST Cache bothers me.  EMC is using SSD’s for caching.  While there are advantages to FAST Cache over a standard pool of SSD’s, they do not seem to justify this design decision.  FAST Cache uses 64k chunks, which is more flexible than normal FAST operation on an SSD pool, which is using 1GB chunks of data.  I can see the advantage of tiering things in smaller increments.  I cannot see any reason why EMC did not go with a PCI based cache option.

In my opinion, flash is so fast, wrapping it up in a traditional disk interface makes little sense for best performance.  I guess at this point, I should disclose that we use a few Texas Memory Systems RamSAN devices here, so i know how fast PCI based flash can be.  Maybe this skews my opinion a bit, but EMC engineers who were here yesterday were touting the VNX’s use of the PCI 2.0 bus that was so much faster.  I agree.  So why not use it for cache?

Maybe I am missing something, and maybe in the real world, it won’t matter.  But if EMC ever bothers to perform an SPC benchmark, I suspect we would see a bottleneck caused by this method of crippling SSD’s with SAS interfaces.  That said, no one is going to argue that replacing a hot swappable SSD is not 100x easier than shutting an array down to change out a bad PCI flash card.  I am just not sure the performance penalty is worth the extra convenience, for me.

If someone who knows a lot more than me can help me understand why this decision was made, feel free to comment below.  For now, I will hold out hope that maybe there will be a VNX-p at some point with some faster cache.

In the coming weeks, I will have similar briefings with a few other vendors, as we try and narrow down our choices for a new storage platform to replace our aged HP EVA’s.  If I come across anything else interesting, I will certainly pass it along to our readers here.

OK good. . . the catchy title worked, and you’ve committed to reading this post. So let’s get to work on making your millions.

Lucky for us technology guys and gals, there are still plenty of C-level employees at companies everywhere who have yet to witness all the benefits of VMware first-hand. Steve Duplessie with ESG just published the following statistics:

• 58% of organizations have virtualized less than 1/3 of their servers.
• Thus far IT owned applications dominate what’s being virtualized. File/Print, etc. 59% haven’t virtualized ANY “mission-critical” applications.

So why are there still this many laggards? These execs have all read countless articles showing how much money they’ll save with virtualization, but often times they haven’t seen the benefits of virtualization beyond simple consolidation. I’m going to show you how to open their eyes in just 15 minutes.

I have done this demo multiple times for audiences from CEO, CIO level, all the way down to customer facing business executives. Each time it has literally been shock and awe, and although I can’t give all the detailed numbers, I will say that the purse strings were blown open and FINALLY our project has a green light. . . millions of them in fact.

This post assumes you have at least rudimentary presentation skills, and understand how to communicate well with people at the executive level.

Fitting vSphere’s “Greatest Hits” into only 15 minutes is not an easy task. It took weeks of careful planning, and test runs before I was able to get everything timed just right. Each time I have presented this, it has gone beyond 15 minutes, but only because there are lots of great questions coming from the audience. I expect you will see the same results if your audience is somewhat intelligent.

After going through a list of features I thought might be interesting to executives, I pared it down to just three in order to make my time limit of 15 minutes. If you have more time, that’s even better, but we all know these guys are very busy.

And that brings me to my first tip: Try and schedule your demo during the winter months so that tee times won’t conflict with your meeting.

Here’s how my 15 minute demo goes down.

Most of your audience will probably already have some idea of what virtualization is at this point, but it’s still a good idea to make sure you cover a few quick points on the basics. I do this while showing them the vCenter console. I show them the virtual machine view, and tell them about typical x86 workload resource usage, and how virtualization allows us to maximize our hardware investments, etc.

Important: Make sure you clarify for them what a “host” is, and whatever term you use for a guest, whether it’s “server”, “VM, or “guest”. You need to emphasize this point up front, and multiple times during your demo so people stay on track and understand what exactly they are seeing.

The first feature I show is a simple cloning operation. By now in your career, this may be old hat, but believe me your executive audience will be impressed by this.

Always make sure you give the benefits when showing any feature. I start my cloning operation while telling them what I’m doing, and then while it runs you’ll have time to explain the benefits. Try and tailor the benefits around how they will impact the customer. Whether the customer is an internal one, or an external one, your audience will appreciate this point of view.

Tip: Use something like BgInfo on your VM’s to show the server name so that your audience can follow more easily. If you don’t use BgInfo, at least change the wallpaper to show the name.

With server cloning, I touch on how valuable it is to be able to clone a server that is having issues so that Development or QC will be able to reliably duplicate a bug or issue. Explain how tough this is in a traditional environment where you have to try and duplicate an issue on a server that is not 100% identical. Also explain how a server can be cloned to test patching or an application update without impacting the production environment, or the customer.

This section should run 3-4 minutes, depending on your SAN speed. If your SAN is slower, do it on local storage to get it done faster. When it’s done, make sure you change the VLAN so you don’t get a conflict, and boot it up. You can quickly just login to show them the server is identical to the one you cloned. You don’t want to spend more than 5 minutes on this one if your allotted time is only 15 minutes.

This is a perfect time to share the next tip: TEST ALL THESE STEPS several times. Time yourself each time, and even do a few dry runs with your team if possible. This will ensure your demo comes off flawless, and that’s important. Believe me when I say some of these people are looking for a reason NOT to virtualize. Most of the time, it has less to do with virtualization, and more to do with fear of change.

My next demonstration is vMotion / Maintenance Mode. This will be mind-blowing for your audience, especially if any of them have a technical background. I start off by telling a story about how some fans have stopped working on one of our hosts, and we need to get the new fans installed before it overheats. (Make sure you know which host has your intended vMotion candidate ahead of time) We can’t wait for the maintenance window.

Normally in this situation, a second cluster node, or hot spare would have to be brought on line, which would mean a short outage for the customer. In this demo, we don’t have time to enter Maintenance Mode, so we’re going to vMotion a single server. I explain that this is how Maintenance Mode works, and how this will be transparent to our customers, and then I prove it.

Bring up a console session on the server to be vMotioned. I use an IIS server as an example, as it’s customer facing, and they understand that well. In my console session, I start a ping -t to another server. In this case, it’s an application server, which the IIS server needs to maintain contact with, or customers will be impacted. Then I execute my vMotion. You might need to explain what “ping” is, so that everyone is on board.

After the server vMotions, I show them that we didn’t drop any packets, and that the customer has not been impacted, and then show them that the VM is on another host. I always reduce my DRS automation level before the demo. I don’t want them to see other migrations happening while we’re demoing. That would spawn a discussion we don’t want to have right now. This takes us to the 10 minute point, barring any questions.

Inevitably at this point, someone usually asks what would happen if the server just failed with no warning. This plays right into your hands, as it’s the perfect segue into your next feature. Fault Tolerance. If they don’t ask, then you ask.

For FT, I setup an FTP server using FileZilla. You can setup whatever works best for your business, but make sure it is something that can clearly demonstrate that customers will not be impacted by an outage. I have preselected a “customer data file”, and setup a simple “FTP Client” VM with FileZilla Client.

I did have to adjust my incoming FTP speed for the server so that the file wouldn’t complete too quickly. You’ll want to make sure you have enough time to test a failover operation, and show the file transfer still going from both the client, and server perspective. So either select a huge file, or bump down your bit rate for the client in the FTP software.

Open up a console session on the FTP server, and then point out the “secondary” instance. Open a console session to the secondary. With both sessions side by side, poke around in the primary and open some windows, a browser, or whatever. You’ll want to demonstrate that the servers are in lockstep with one another. Open a console session to the FTP Client.

At this point, I explain how the customer is sending us this file, and start the transfer. I then explain that the particular host that the FTP server lives on is going to go down without warning. Show them the host name. You can simulate this with the Test Failover option.

It will take less than a minute, during which you will want to point out the bytes transferred on the server, and the client. Point out that the client has no errors, and then you’ll see the secondary come back online.

Again, show the host name so they can see that it has indeed changed servers. I found it helpful to time the file transfer so that it would complete right around this time. Then you can show them the server, the completed file, and the client, once again explaining how the customer has no idea that a server went down in our datacenter.

If you don’t get gasps or applause at this point, you did it wrong.  Once again, PRACTICE this over, and over before taking it to the executives! You don’t want to be up there looking like Bill Gates demoing Win98.  You want to look like John Chambers demoing the Cius.

Wrap up with an explanation that these are just three of hundreds of VMware features, and then answer the questions that follow. At the end, these people will be frantically searching for their checkbooks. Your millions should start flowing after the next budget committee meeting.

The characters, companies, and products mentioned in this video are fictional.  Any resemblance to actual ones is purely coincidental.

Happy Thanksgiving!

While cruising the show floor at VMworld San Francisco, the Xiotech booth seemed to be abuzz every single time I walked by.  Finally, I stopped in to see what all the commotion was about.  It was not about Xiotech at all.  It was about iPads.  As soon as I stepped close to the booth, there was a rush to scan my badge so I could win an iPad, but not a real clear picture of what the heck Xiotech actually did.  I moved on to the next booth, and I suspect most of the 17,000 attendees did as well.

The first time I heard about Xiotech was a year or so ago, and at the time, the only thing I understood was that they offered storage in a sealed box that was supposed to be more reliable than your average monolithic array.  At the time, it sounded far-fetched to me.  I didn’t see how they could make those claims while using the same disks we all use in some special locked box.

It wasn’t until I heard Xiotech’s CEO Alan Atkinson on an Infosmack podcast that I decided to investigate further.  Alan didn’t really go into details, but after hearing him talk about some of the heavy hitters on his team, I was intrigued.  Their own website doesn’t really spell out in detail what they are doing that makes them stand out.  I had to make a call and talk to their engineers.  What I found was so amazing, I have to share it here.

Since Seagate is one of Xiotech’s major shareholders, these guys have unfettered access to the inner workings of the disk drive, from the firmware on down.  Inside a Xiotech “magic box”, the disks are the exact same model as one would find spinning inside an array from another manufacturer.  The firmware is where the magic actually happens.

Apparently disk drives have all kinds of cool things they can do besides reporting your typical “OMG I’m going to fail soon!” messages.  Seagate reports that nearly 80% of the “failed” disks they receive are actually fine.  They cannot find any issues whatsoever.  What does Seagate do with those disks?  They remanufacture them and toss them back into the “refurbished” bin.

Although the disk has the capability to “remanufacture” itself in the firmware, it cannot be done inside your traditional array.  The vibration is simply too high to do this reliably.  A typical shelf full of rotating disks vibrates at over 40 rads (units of rotational vibration).  When a dozen or more disks are all rotating at the same speed, in the same direction, it’s not hard to imagine the vibration in a tray of disks.  Xiotech actually mounts its disks so that they are counter-rotating.  One disk rotates clockwise, and the disk beside it is mounted so that the rotation is in the other direction.  This reduces vibration inside the Xiotech box to 2 rads.  This means they can reliably remanufacture a disk inside the box while the array is in operation.

Cool huh?

Here’s a breakdown of what a Xiotech array does when it detects a potential disk error:

1. Data is migrated from the suspect disk to another disk elsewhere in the array
2. Disk is power cycled
3. Complete factory remanufacturing process
4. Recalibrate heads
5. Rewrite servo tracks
6. Perform a low-level format

It gets better.  The unit of storage inside the Xiotech is not actually the disk.  It’s the head.  This means that each individual platter is a unit of storage, and the data is striped as such.  Why is that a big deal?  For one, as disk drives get larger and larger, they take longer and longer to rebuild.  This increases your exposure to another failure.  Second, if the most common catastrophic disk failure is a head crash, wouldn’t it be nice just to be able to disable that particular head and move on?  When all the above steps are complete, if a head is still not responding, it will be disabled, and that platter will not be usable.  The rest of the disk is good to go.

The net result of all this firmware magic is that the Xiotech array comes with a 5 year warranty at no cost.  Why not include it when you’re seeing 99.9983% uptime in the field?  To prove the point, Xiotech took 200 disks that were marked “failed” and returned to Seagate, and stuffed them into a rack of Xiotech arrays.  They ran for TWO YEARS without a service event.

So we’ve established reliability.  What about performance?  Disk drives aren’t really getting any faster.  How can we squeeze more performance out of spinning disks?  Xiotech has answered that with yet another firmware tweak.  Often times, the disk subsystem has to wait for data to be read because the head is busy reading another part of the platter.  Even under optimal conditions, disks have a few ms of latency built in for the heads to move.

One of the storage geniuses over there actually came up with a plan to have the heads constantly move back and forth across the platter.  When I heard this, I wanted to fly out to Minnesota and buy this guy a beer.  If the heads never actually park, then statistically speaking, the head will always be closer to the data it needs.  What does this mean for you and me?  It means that a single 3U Xiotech ISE performs at 12,600 IOPS on the SPC-1 benchmark.

Since the cache, and controllers are on board each ISE, these IOPS scale linearly, as opposed to a monolithic array which could run out of gas if its controllers get saturated.  So with 5 ISE’s at 15U, one can expect to hit 63,000 IOPS.  This is clearly not your father’s storage array.

Xiotech has their own storage virtualization appliance with the ISE 9000 for larger enterprises, which works quite well with VMware, and is ICON capable for ease of integration with just about anything.  This would also be absolutely amazing behind some virtualization from FalconStor, NetApp, Nexenta, or really any of the other storage virtualization products out there.

With all these patents, and truly ground breaking technology, one is left wondering why Xiotech has yet to secure a huge OEM deal.  Chris Mellor did a nice write-up on this topic, and his argument is that manufacturers are worried about sourcing disks from only one manufacturer.  If Seagate were to have a bad batch of disks go out, it could be crippling.

I can understand his point of view, but I was born a skeptic.  Considering that storage array manufacturers generate huge chunks of highly profitable revenue from servicing arrays, I doubt we’ll be seeing a commercial with an EMC / NetApp, or HP guy out fishing with the Maytag repair man anytime soon.  Regardless whether they land a huge OEM or not, Xiotech has proven that spinning disks are far from the end of their useful life.

I haven’t seen many reviews of this class, so I thought I would share my experience for those thinking of registering for the VMware vSphere Troubleshooting course.

I was originally registered for Fast Track, but after I took a long, hard look at the course outline, I realized I already knew most of the material they would cover. I’m glad I changed my registration to Troubleshooting, especially since the requirements changed to include it as a prerequisite for VCP4 certification.

When I sign up for a technical class, I always worry that I’ll get a professional instructor with not much real world experience. My trainer for this course was John Davis from New Age Technologies. He was definitely well versed in the material, and since he had also spent considerable time in the field, he was able to share a lot of his real troubleshooting experiences. This was the most beneficial part of the class, so if you register, make sure you’re taking it with someone who has been in the trenches.

My intent is not to disparage the excellent VMware official curriculum, as it was very well put together, and quite thorough. I just learn much better when I can discuss with folks who have been where I am, and seen things I haven’t seen. I can read the book and do the labs anytime.

Day One

The first day, we jumped right into some CLI troubleshooting. Since I had been using ESXi, and the vMA, I was on somewhat familiar turf. In the first few labs, we configured the vMA, vi-fastpass, session files, and started right in with some vicfg commands. These are mostly network related commands in the first few labs.

We played around with tech support mode a bit, although this was a couple days before the 4.1 release where it became officially supported. In class we got the official “don’t try this at home” line from the instructor. We enabled SSH on ESXi and played around with Putty a bit. One of the coolest things I saw on day one was the vsish command on ESXi. For those who haven’t played with it, I recommend checking it out. It’s a bit like the Windows registry for ESXi. One can view tons of info about the system setup using this command.

We had an in-depth look at ESX, ESXi, and vCenter log files in the console, and in vSphere Client. We covered viewing, exporting, bundling, and even setting up log collection with the vMA. There were 5 labs on logging, and setting up the vMA to host the logs.

Day Two

On day two, we jumped into network troubleshooting. We covered a lot of new info on the inner workings of the dvSwitch, including synchronization, and what happens when it breaks, which I found helpful. There were a few labs where the instructor broke our network setup, and we had to fix the problem. Some of these breaks surrounded dvSwitch timeouts, obsolete dvSwitch info, and uplink issues.

We played with CDP, port binding methods, and VLAN / PVLAN troubleshooting. I think the PVLAN stuff was most beneficial. Even understanding how those work, it’s great to get some hands-on troubleshooting with them, not having used them in my own environments.

Day Three

Today we continued with networking, which is probably the most information packed module of the course, and with good reason. There were labs on Wireshark, tcpdump, net-dvs, and viewing / changing network configs from the vMA and CLI. As a CCNA, I had a pretty solid networking background going in, but I did notice some in the class struggled a bit with this. If you don’t have much networking experience, or at least a solid understanding of VLAN’s, I recommend brushing up before this course.

We went into management troubleshooting after wrapping up the network break / fix labs. Firewall configuration errors were something I really didn’t care about, but not everyone has the luxury of using ESXi exclusively. There was some database connectivity and lots of vCenter troubleshooting.

Storage was up next, and there was a ton of information on PSA, NMP, MPP, SATP, CHAP, and PSP. After we got all the acronyms out of the way, we did some great iSCSI break / fix labs. These were fun, and our sneaky instructor gave us some pretty hairy issues that may never happen in the real world, but provided fantastic troubleshooting opportunities.

Day Four

VMotion, DRS, HA, and cluster troubleshooting was the topic of the morning on day four. Reservations and swap file space issues cropped up, as did VMotion failures and admission control policies.

My favorite part of the class was absolutely the break / fix labs. On day four, we got the chance to have as many of these as time would allow. Previously all the break / fix scenarios were related to the chapter we were on, so you’d have a good idea where to start troubleshooting. But on the last day, it could have been anything. I had a blast troubleshooting the breaks that our instructor did for us, and I probably learned the most during this exercise.

In summary, I found the troubleshooting course much more valuable than the ICM or Fast Track would have been for me. For someone with zero to only a few months of vSphere experience, ICM or Fast Track may be a better choice. If you’ve been playing with it at least in a lab for 6 months or more, read the books, and VMware Documentation Roadmap, you’d probably find Troubleshooting a more beneficial course.

Will it prepare you for the VCP4 exam? No. But, in my opinion, neither will the others. The only thing that can adequately prepare you for the exam is the Blueprint.  I know that’s the standard response everyone gives, but it really is true. If you go through the Blueprint step by step, you’ll be good to go for the exam.

If you’re at VMworld 2010, and you haven’t been to session ALT2004 - Building the VMworld Lab Cloud Infrastructure, I recommend you put that one on your schedule.  Dan Anderson (VMware Principal Architect) is a great presenter, and hearing him describe in detail how he architected and built this year’s lab environment is mind-boggling.

This guy is basically doing something that might take us a year or two, in only a few months.  There’s no load testing, and no way to know if it’s all going to work well until we all get in the labs and hammer on the environment.  Talk about flying by the seat of one’s pants!

Here’s a quick summary of what’s in the lab environment:

• 329 TB of storage from EMC and NetApp
• 352 Servers (HP Blades and Cisco UCS)
• 736 CPU sockets with 3,072 cores
• 7.5 THz of total CPU
• 14.6 TB RAM
• 480 thin clients
• 4,000+ VM’s deployed every hour
• 12 Lab Manager Instances
• 4 vCenters per site

This is all spread across 3 sites:

• Verizon – Ashburn, VA
• Terremark – Miami, FL
• Moscone – San Francisco

Each site can run the entire lab environment if necessary, so Hurricane Earl won’t be spoiling our fun!

Taking note of the lessons learned during Dan’s experience will help all of us, even if we’re not building at that speed or scale.  I encourage you to go to his session.  Here are the remaining opportunities to see this one:

Tuesday 11:00 AM		Moscone West Room 2014
Wednesday 03:00 PM		Moscone West Room 2014
Thursday 12:00 PM		Moscone West Room 2014

Now back to your regularly scheduled long line.

I think VMware is currently well positioned to become the Novell of virtualization.  Consider the similarities:  technical superiority, “career-safe” choice, large vested corporate user base, complex pricing models, arbitrary price increases in a captive market, Microsoft buying market share with free and nearly-free product.  http://www.networkworld.com/community/tech-debate-microsoft-vmware

Recently, the above comment was made by Guy Chapman on an interesting Network World debate over who has the better hypervisor, Microsoft or VMware.  With respect to Guy, I’d like to touch on the reasons I think this is not (currently) a fair comparison.

First off, a bit of background on Novell for the younger readers.  Novell developed the first network operating system back in the 80’s.  By 1990, any business that had a need for networking was using Novell NetWare.  In a decade, Microsoft’s massive marketing machine had relegated Novell to the background in the corporate world.  MS continued gaining market share with NT, and the Back Office suite of server products.  Y2K gave larger customers a reason to upgrade their old stuff, and most of Novell’s larger customers began moving toward Microsoft, which was cheaper than Novell, and GUI based, which attracted enterprise customers.

You have to hand it to Microsoft.  NT was a solid product, priced right, and with the introduction of Active Directory in 2000, Novell faded fast.

That was then.  Let’s take a look at Microsoft’s track record of innovation over the past decade. What has Microsoft brought to market in the past several years?  Zune?  SCOM?  Windows Mobile?  Bing?  Each one of these is nothing more than a re-hash of a product that someone else brought to market first.  Were any of them better than their predecessors?  The market says no.

In contrast, VMware over the past decade has brought incredible innovations to the market faster even than master innovators like Apple.  Who would have dreamed 10 years ago that one could vMotion a server onto different hardware, and even different storage?  The feature differences between 4.0 and 4.1 releases of vSphere alone are more innovative than anything I have seen out of Redmond for many years.

Microsoft is quite good at commoditizing technologies that have been developed and incubated by others.  I suspect this is what Guy was referring to when he made the Novell comment.  I think the difference here though is the sheer speed at which VMware is innovating.  Microsoft is at least a couple years behind vSphere 4.0, and with new features like VAAI, SIOC, and NIOC, they might have to add another year just to catch 4.1.  Microsoft is capable of throwing amazing amounts of cash at the problem, as evidenced by the current pricing structure of Hyper-V.  But is it enough?

I submit that if VMware continues innovating at their current pace, Microsoft will never catch them, unless there is a massive shake-up.  With the current MS management team and company culture, it is impossible.  Even if there were a tectonic shift this week in Microsoft’s management, and the culture change started next week, you’re still looking at a decade before they can innovate like VMware can today.  Microsoft is a behemoth of a company, and culture shifts are exponentially harder to pull off as a company grows larger.

Rest assured, if VMware is indeed the Novell of virtualization, they’re going to enjoy a couple more decades at the top before disappearing into irrelevancy.