If you have HP BL 460 G7′s with the on-board 10GB CNA, you’re going to want to read this post regarding a problem with the latest firmware.
This was first noticed this issue when updating firmware to troubleshoot an issue where the storage doesn’t come back up after rebooting an upstream Nexus switch.
The symptoms are: the NIC comes back up, and the vfc is up, but all storage paths on that side of the fabric are still dead in ESXi 5.0. To fix this issue, the vfc or port channel must be shut /no shut.
I also saw an issue where the storage paths were dead, and the NIC never came back up. A reset of the Ethernet port will not fix this. A reboot of the ESXi host is required. Pay attention to the NIC state if you lose storage paths in this configuration with FCoE.
As part of my troubleshooting, I went to update the firmware on the CNA. The latest version of the firmware from HP is 4.0.360.15a. When updating using the Emulex utility, on about 20% of my blades, I got a CRC error during the upgrade process. Below is a screenshot of this error.
After retrying the firmware update, as stated in the utility, the same error occurred.
This is where you need to pay attention!!
Since the firmware version is correct, one might assume the update was indeed successful. That’s a bad assumption. Upon further testing, we found the blades that failed the firmware update were the ones failing during the switch reloads.
There were only 2 blades that did NOT fail the firmware update, but still failed the switch reload process. They were replaced, and now I have no blades failing to reacquire storage paths after an upstream switch failure.
I must point out that HP has been unusually proactive with this issue, which is a nice change! I still have several blades in another datacenter that are not taking the firmware update. When I scheduled to have those all replaced, HP got some of their top people on it and scheduled a call. I tested their proposed fix this morning, which didn’t work.
They are actively working on a fix, so you won’t have to replace your blades. I will update this post as soon as I get word back from them on that fix. Meanwhile, if you’ve seen this, you might want to schedule some switch reloads during a maintenance window to make sure you are good to go.
As of today, there is no fix that I’m aware of. . . HP replaced the remaining blades after we tried a couple more proposed fixes. If I get word of a fix, I will post it here.