How to Detect and Resolve “State in Doubt” Errors

One of our customers recently came across a problem in their environment that I think warrants some attention. The VMWare administrator had gone to the storage team and asked if they saw any issues on the Fabric or IBM SVC storage environment because the infamous “state in doubt” message was popping up in the /var/log/vmkernel log file messages were similar to what is shown below:

The error indicated that there was a time-out by the HBA because the command took longer than 5 seconds to complete.

Typically this implies a problem anywhere along the SAN path from your physical ESX host/HBA to the back-end storage including:

• Fabric F ports to which the host is connected
• Fabric ISLs (E ports) if in path
• Storage ports

VMWare has an excellent article in their knowledge base for more information on this topic: Information about the error: (1022026).

Being the savvy storage engineers that they are, our customer did a quick check of the IntelliMagic Fabric Error dashboards and SVC Port Dashboards.

Invalid Transmission Word Errors

While there were no issues observed on the SAN Fabric, the IntelliMagic Vision SVC Port Dashboard identified issues with Invalid Transmission Words as demonstrated in Figure 2:

Figure 2: Bad SFP – Invalid Transmission Word Errors

A transmission word consists of four 10-bit codes that must be sent in a precise format.  If they are not in the correct format, the switch detects that the Transmission Word is invalid. This can happen for a number of reasons such as a faulty cable, a bad Small Form-factor Pluggable transceiver (SFP) or as a result of a cable being temporarily unplugged. If this happens once in a while, it might be that a cable got unplugged. However, if this happens continually, there are definitely issues that should be addressed.

In this case, the only visible symptoms were “state in doubt” errors in the VMWare error log and the red flag in the IntelliMagic Vision dashboard. Because of a still relatively low error rate, the issue did not cause significant performance impacts. But if left unresolved, the situation could degrade further and could have resulted in significant performance and connectivity impacts for all hosts connected to the offending SVC port.

Drilling into the “State in Doubt” Issue

Drilling down from the IntelliMagic Vision dashboard the customer found that the issue was with port ‘node1-2’, as shown in Figure 3. It was decided to replace the SFP on this port. The graph shows that this was the right decision: the Invalid Transmission Word errors ended abruptly on 6/16/2015 at 10:00 AM when the SFP was replaced. After that, both the “state in doubt” errors, as well as the Invalid Transmission Words on the SVC port ended.

Figure 3: Invalid Transmission Words on SVC node1-2

SAN storage infrastructure is complicated because there are so many components. IntelliMagic Vision can help reduce the visibility gaps and improve the availability of your SAN fabric and connected storage by providing deep insights, practical drill downs and specialized domain knowledge for your SAN environment.

The short video below demonstrates how we allow you to visualize all of the components from the VMware host through the fabric to the storage volume.

To learn more about IntelliMagic’s support for VMware and our Topology Viewer, visit intellimagic.com/vmware.