Clogged Device Drain? Use Your Data Snake!

Have you ever run into high I/O response times that simply defy explanation? You can’t find anything wrong with your storage to explain why performance is degraded. It could be a classic “slow drain device” condition. Unfortunately, you can’t just call the data plumbers to clean it out! What is a storage handyman to do?

Slow Drain Device

A slow drain device may be caused by several factors, but a typical one is a bad port either on the storage or the switch. Let’s take a look at an example where an IBM SVC suddenly developed degraded write performance at 9 AM on June 20th. We dug into the performance data to see if we could find out what caused the problem.

Isolate the Problem

First, let’s see if the write response time is degraded everywhere or isolated to particular nodes. By using IntelliMagic Vision, we drilled down to see the write response time by node. It was easy to see that the problem was isolated to the two nodes in I/O Group 3.

Node Write Response Time (ms) and the obvious point where the problem begins

Next, we checked if the overall workload to this SVC had any dramatic changes that could have caused the problem. We looked at the node throughput and saw no significant change at 9 AM on June 20 when the response time problem began.

Simple Solution

We then tried to see if something changed on the ports on the two nodes in question. Indeed, we did see some degradation for port 3-3 on both nodes in I/O Group 3. This pointed to an issue somewhere in the path that includes these ports. From here a good tactical solution was to simply take the path offline to see what happens. That was done at 9 AM on June 21st and it turned out the path was indeed causing a slow drain device. Taking the path offline restored I/O response time back to normal.

Front End Write Response (ms)

Troubleshoot Performance Problems

This is just one example of how IntelliMagic Vision for SAN can help you troubleshoot thorny performance problems. When slow drain devices strike, having IntelliMagic on your side as a “plumber’s assistant” can help you avoid a serious mess!

If you would like to see how IntelliMagic Vision can help you proactively manage your SAN and Storage infrastructure, please visit https://www.intellimagic.com/san-products/intellimagic-vision. Feel free to contact us with any questions!