Commvault : DR backup to cloud fails to run

 The Commvault DR backup to cloud (an option within Control Panel of Commvault console) was reporting failures. 

The CVCloudService.log reported a ''CURL perform failed" error and suggested SSL certificate problem. 

As it turned out, the Commserve was not able to reach https://cvdrservices.metallic.io and was being blacklisted by the firewall. One the web portal was added to the whitelist, the Cloud DR backup to cloud completed ok. 

Commvault webconsole is in accessible - It looks like something went wrong

 

It was a lovely sunny morning and the weather was fine , UNTIL.....the Commvault webconsole reported the above error. 

The usual troubleshooting step was :

1) Restart the tomcat services. That didn't work.

2) The next pro :-) step is to restart stop Commvault (CV) services, restart IIS (iisreset /noforce) and then start CV services. That didn't work. 

The webconsole.log was reporting invalid reponse returned :503 error. What was interesting was that the webserver.log was not touched for a while. The event log errors showed that W3WP.exe process was crashing. 

SOLUTION: 

As documented in the following link, the new Cylance agents that were deployed on the Commserve was responsible for the W3WP.exe process crashes. The following exclusions were added to Commserve and the issue was resolved. 

\Windows\SysWOW64\inetsrv\w3wp.exe

\Windows\System32\inetsrv\w3wp.exe

https://docs.microsoft.com/en-us/answers/questions/590694/net-runtime-errors-iis.html

NetApp E-series: Disk firmware upgrade causes volume reconstruction

This post describes an scenario that was presented to me while upgrading disk drive firmware for NetApp E-series array. 

Recently, as part of NetApp E-series firmware upgrade, the disk drive firmware upgrades were initiated. The disk firmware upgrades resulted in volume reconstruction (which completed successfully) and the disk firmware upgrade process stopped without any errors. While some drive firmware's were upgraded, there were some where the upgrade process was not initiated. 

Upon investigation, it was identified that a UI timeout was causing the user to logout and gracefully stop the disk firmware upgrades. The fix was to disable the UI timeout for the duration of the upgrade and then trigger the disk firmware upgrade. 

NetApp E-series : The session cannot be logged out since a device on that session is currently being used.

 While removing iSCSI connection from a windows server to NetApp E-series storage LUN , the following error was reported. 

The session cannot be logged out since a device on that session is currently being used.

The session cannot be logged out since a device on that session is currently being used. The device list showed that LUN0 was still in use. Upon further triage, it was identified that Santricity Storage Manager Agent services was accessing this LUN0 to manage the storage array. 

Resolution: 

Stopped the Santricity Storage Manager Agent services and the ISCSI connection was terminated successfully. If this doesnt help, then look for similar services that might be holding up the disk. 

Hope this helps someone. Please leave a note if it does. 

Cisco UCS Partition bootflash on fabric interconnect A is clean but with errors.

 The UCS infrastructure components were upgraded to 4.0.4m and both FI reported the following error:

Partition bootflash on fabric interconnect A is clean but with errors.

The plan of action to resolve this issue is to run  the command 'reboot e2fsck' one at a time on both FIs. The commands can be run from the local management of the FI (login to UCS CLI then run the command  'connect local-mgmt a|b') . 

Hopefully the issue resolves after the fsck process completes. This process requires a reboot of the FIs and that can be disruptive if UCS is not configured right.

If the fsck process doeesnt fix the issue then the plan is to raise a case with CIsco TAC . 

Commvault backup copy failure: No disks were found on virtual machine

 If the Commvault backup copy jobs failed to backup a VM and report the following error then you try out the two possible solutions: 

No disks were found on virtual machine [VM-name].  Please ensure that the virtual machine is configured with one or more virtual disks, and validate that the proxy is able to communicate with the Host.

Possible solution: 

1) Verify the VMDK are not INDEPENDENT or RDM disks. If yes, then you will need to review and possibly configure the subclient to backup the VMDK. 

2) Verify if there are any filters configured that might stop the VMDK from being backed up. 

Commvault: UpdateIndex index processing failed

 Issue Description: 

The backup copy of VMware VM backup in Commvault reported failure. The backup jobs failed with the following error: 

     Error Code: [14:21]

    Description: UpdateIndex index processing failed for     

Resolution: 

Login to the proxy VM and review the updateindex.log and search for the string 'FATAL ERROR' 

UPDATEINDEX FATAL ERROR: extent size change detected for vm. 

When the VM extent size changes, Commvault incremental backup fails to run. The way to fix the issue is to run a FULL backup and then attempt an incremental backup.