Commvault SQL restore fails

Recently, while performing a restore of a SQL DB in Commvault, the restore job failed.  The attempt tab of the restore job showed no details and the events tab had the following entry with a timestamp [note the time stamp as it will be handy when reviewing logs].

Received failed message for job [job ID] , phase [Database Restore]. 

The next step was to login to the SQL client and review the SQLiDA logs. (if the access to the client is not available then right click the Commvault restore job and click on 'view logs' and look for the following section and search for the string 'VD_E_TIMEOUT'

File    : SQLiDA.log

Here is how the logs might look:

Solution: 

1. Restart the commvault services on the client. 
2. Ensure the client is running the latest service pack (in my case the client had to be updated from V11 SP 9 to V11 SP13) 
3. Most importantly, follow the Commvault link and apply both the VDI settings (VDI timeout and VDI retry).  http://documentation.commvault.com/commvault/v11/article?p=18144.htm

NetApp ONTAP Service Processor : Connection Refused

Recently, there was an issue with connectivity to the service processor of one of the cluster nodes. The SSH connection to the IP address of the service processor would throw an error 'connection refused'.


The issue was resolved by rebooting the service processor.
<Clustername>::> system service-processor reboot-sp -node <node name> 

OnCommand Performance Manager migration to OnCommand Unified Manager fails

Background: 

Recently, the OnCommand Unified Manager (OCUM) was upgraded to 7.2. After upgrade, OnCommand Performance Manager (OCPM) data from 7.1 instance was migrated to OCUM 7.2.
The following document was used to perform the migration.
https://www.netapp.com/us/media/tr-4589.pdf


Issue: 

The migration process was reported as FAILED . One easy way to find out what caused the failure is to re-run the migration. As you re-run the migration, the workflow will show what caused the previous job to fail. Please note the cause of the failure

Alternatively, you can generate a support bundle (KB is here) and review the migration.log under JBOSS directory of the support bundle.

Resolution: 

Based on the cause of the failure, you can use one of the following KBs:

Migrating performance data to OnCommand Unified Manager 7.2RC1 shows status 'FAILED':

https://kb.netapp.com/app/answers/answer_view/a_id/1070622/loc/en_US

Performance data migration fails between OnCommand Performance Manager and Unified Manager 7.2x due to error java.io.EOFException: unexpected end of stream:

https://kb.netapp.com/app/answers/answer_view/a_id/1005077/loc/en_US

Performance data migration to newly upgraded UM 7.2 fails due to missing table partitions

https://kb.netapp.com/app/answers/answer_view/a_id/1005199/loc/en_US

Performance data migration from Unified Manager 7.1 to 7.2 fails with error: "Error writing file '/tmp/MYn75pE5' (Errcode: 28 - No space left on device)"

https://kb.netapp.com/app/answers/answer_view/a_id/1072873/loc/en_US

CommVault V11 SP update : error Required package [56] are not present

Recently, a CommVault customer requested an update of their CommVault environment from v11 SP10 to SP12. While the upgrade of the CommServe was successful, the upgrade of the media agent failed with the following error: 
"Required package(s) [56] are not present in current DVD. Record a new custom package with all required packages." 

A lot of time was spent on verifying if the procedure used to download the software package was correct. Later CommVault support identified that the antivirus software was blocking an MSI file at the following location:
<Software Cache>\DownloadPackageLocation2\DownloadPackageLocation_WinX64\BinaryPayload\NetworkStorageServer.msi 

Fix:

The antivirus software on the server(media agent) was configured to exclude CommVault directories and that helped resolve the issue. Here is the document that I followed:  
(http://documentation.commvault.com/commvault/v11/article?p=8665.htm

CommVault VM restore : VM Blue Screens (BSOD)

Recently, a CommVault customer requested a VM restore. The routine browse and restore procedure was used to restore the VM successfully. However, the restored VM would blue screen when powered on. 

hmmm... was the VM not backed up properly ? Is the restore procedure right ? Are we missing/ ignoring a flag ?  These were some questions that needed answer.  

but ... CommVault support looked at the issue and identified it to be a bug in V11 SP10 and suggested an upgrade to SP12. (note: there is a workaround available in SP10). 

Netapp Storage : check if a network port is open via telnet

If you need to check whether your storage system can access an IP :<port> pair then you can use the following command
cluster*:> system node systemshell -node <node name>  -command telnet <IP>  <port>

Monitoring NetApp systems

If you are NetApp system admin and want to monitor the storage system using a tool outside of unified manager then you want to consider monitoring the following areas:
          1) Availability (For ex: Is the aggregate offline? )
          2) Capacity (For ex: Is the aggregate full ?)
          3) Configuration ( For ex : Has the aggregate been reconfigured? )
          4) Performance (For ex: Has the IO latency been high on the aggregate)

It is advisable that you consider the above mentioned 4 factors on the following components:
          a) Aggregates
          b) Cluster events
          c) Disks events
          d) Enclosure events
          e) Fan events
          f) Flash card events
          g) Inodes
          h) LIFs
          i) LUN
          j) MetroCluster bridge
          k) MetroCluster connectivity
          l) MetroCluster switch
          m) cluster nodes
          n) NVRAM
          o) Ports
          p) Power Suppy
          q) SnapMirror
          r) Qtree
          s) SnapShot
          t) SnapVault
          u) SFO
          v) SP
          w) Storage services
          x) Storage shelf
          y) SVM events
          z) user and groups
          a.a) Volume
          a.b) Volume move
          a.c) Unified Manager monitoring