NetApp E-series: Disk firmware upgrade causes volume reconstruction

This post describes an scenario that was presented to me while upgrading disk drive firmware for NetApp E-series array. 

Recently, as part of NetApp E-series firmware upgrade, the disk drive firmware upgrades were initiated. The disk firmware upgrades resulted in volume reconstruction (which completed successfully) and the disk firmware upgrade process stopped without any errors. While some drive firmware's were upgraded, there were some where the upgrade process was not initiated. 

Upon investigation, it was identified that a UI timeout was causing the user to logout and gracefully stop the disk firmware upgrades. The fix was to disable the UI timeout for the duration of the upgrade and then trigger the disk firmware upgrade. 

NetApp E-series : The session cannot be logged out since a device on that session is currently being used.

 While removing iSCSI connection from a windows server to NetApp E-series storage LUN , the following error was reported. 

The session cannot be logged out since a device on that session is currently being used.

The session cannot be logged out since a device on that session is currently being used. The device list showed that LUN0 was still in use. Upon further triage, it was identified that Santricity Storage Manager Agent services was accessing this LUN0 to manage the storage array. 

Resolution: 

Stopped the Santricity Storage Manager Agent services and the ISCSI connection was terminated successfully. If this doesnt help, then look for similar services that might be holding up the disk. 

Hope this helps someone. Please leave a note if it does. 

Cisco UCS Partition bootflash on fabric interconnect A is clean but with errors.

 The UCS infrastructure components were upgraded to 4.0.4m and both FI reported the following error:

Partition bootflash on fabric interconnect A is clean but with errors.

The plan of action to resolve this issue is to run  the command 'reboot e2fsck' one at a time on both FIs. The commands can be run from the local management of the FI (login to UCS CLI then run the command  'connect local-mgmt a|b') . 

Hopefully the issue resolves after the fsck process completes. This process requires a reboot of the FIs and that can be disruptive if UCS is not configured right.

If the fsck process doeesnt fix the issue then the plan is to raise a case with CIsco TAC . 

Commvault backup copy failure: No disks were found on virtual machine

 If the Commvault backup copy jobs failed to backup a VM and report the following error then you try out the two possible solutions: 

No disks were found on virtual machine [VM-name].  Please ensure that the virtual machine is configured with one or more virtual disks, and validate that the proxy is able to communicate with the Host.

Possible solution: 

1) Verify the VMDK are not INDEPENDENT or RDM disks. If yes, then you will need to review and possibly configure the subclient to backup the VMDK. 

2) Verify if there are any filters configured that might stop the VMDK from being backed up. 

Commvault: UpdateIndex index processing failed

 Issue Description: 

The backup copy of VMware VM backup in Commvault reported failure. The backup jobs failed with the following error: 

     Error Code: [14:21]

    Description: UpdateIndex index processing failed for     

Resolution: 

Login to the proxy VM and review the updateindex.log and search for the string 'FATAL ERROR' 

UPDATEINDEX FATAL ERROR: extent size change detected for vm. 

When the VM extent size changes, Commvault incremental backup fails to run. The way to fix the issue is to run a FULL backup and then attempt an incremental backup. 

VMware ESX host lost a path to the storage device

 Recently, there was a customer issue that was brought to my notice. What was happening was that the storage device had lost one of the storage path on one ESX host (while it was working ok on other ESX hosts in the cluster). 

The ESX host had two standard vswitches running vmkernal ports on dedicated physical NICs. 

Here is what was done to troubleshoot and fix the issue

1) Identified the two host vmkernel IPs assigned to the vmk1 (vswtich1) and vmk2 (vswitch2). Identified the storage IP address. Finally, identified the two vmkernel IPs that was assigned to a working ESX host. Here is a pictorial representation of what was found. 

2) Ran a ping from the working host to vmkernel IPs 192.168.10.19 and 192.168.10.20. Both the IPs were reachable on the networks. The pings to the storage IPs responded fine. It looked the physical connectivity was working ok. 

3) Ran a ping from the faulty ESX host using the two vmkernel IPs. 

vmkping -I vmk2 192.168.10.40 -- > Ping failed 

vmkping -I vmk1 192.168.10.40 --> ping successful. 

This led us to the conclusion that vmk2 was not connected to the storage IP. 

4) Ran the following command on the faulty ESX host to identify the state of the IP connections. The command output showed that vmk2 connection was in SYN_SENT connection state.  

esxcli network ip connection list 

tcp 0 0 192.168.10.20:45796 192.168.10.40:3260 SYN_SENT 66477 newreno vmkiscsid
tcp 0 0 192.168.10.20:45795 192.168.10.40:3260 SYN_SENT 66477 newreno vmkiscsid
tcp 0 0 192.168.10.20:45794 192.168.10.40.10:3260 SYN_SENT 66477 newreno vmkiscsid

 5) To fix the issue, changed the vmk2 IP address from 192.168.10.20 to 192.168.10.21. This fixed the issue for me. Later, I was able to change the vmk2 back to 192.168.10.20 without any further issue. 

It looks like the network IP connection was in a weired state and might have been caused at the time of connection negotiation. 

Hyperflex native snapshot error: Snapshot not supported on VM with files on different datastores

Recently, I came across the following Hyperflex native snapshot error. The fix was to ensure the VM resides on a single datastore. 

Here are the steps that I followed to fix the issue: 

1) Perform storage vmotion so that all VMDKs reside on the same datastore. 

2) Verified that there is no CD/DVD ISO connected to a different datastore.  If so, I would change the setting to “Client Device”. 

What I was trying to achieve was that when I select the guest VM in vCenter and then click on the “Datastores” tab, it should only list 1 datastore.