Have you ever run into high I/O response times that simply defy explanation? You can’t find anything wrong with your storage to explain why performance is degraded. It could be a classic “slow drain device” condition. Unfortunately, you can’t just call the data plumbers to clean it out! What is a storage handyman to do?
Slow Drain Device
A slow drain device may be caused by several factors, but a typical one is a bad port either on the storage or the switch. Let’s take a look at an example where an IBM SVC suddenly developed degraded write performance at 9 AM on June 20th. We dug into the performance data to see if we could find out what caused the problem.
Isolate the Problem
First, let’s see if the write response time is degraded everywhere or isolated to particular nodes. By using IntelliMagic Vision, we drilled down to see the write response time by node. It was easy to see that the problem was isolated to the two nodes in I/O Group 3.
We then tried to see if something changed on the ports on the two nodes in question. Indeed, we did see some degradation for port 3-3 on both nodes in I/O Group 3. This pointed to an issue somewhere in the path that includes these ports. From here a good tactical solution was to simply take the path offline to see what happens. That was done at 9 AM on June 21st and it turned out the path was indeed causing a slow drain device. Taking the path offline restored I/O response time back to normal.
Troubleshoot Performance Problems
This is just one example of how IntelliMagic Vision for SAN can help you troubleshoot thorny performance problems. When slow drain devices strike, having IntelliMagic on your side as a “plumber’s assistant” can help you avoid a serious mess!
If you would like to see how IntelliMagic Vision can help you proactively manage your SAN and Storage infrastructure, please visit https://www.intellimagic.com/san-products/intellimagic-vision. Feel free to contact us with any questions!
Storage Performance Analysis for IBM SAN Volume Controller (Spectrum Virtualize)
This white paper discusses the end-to-end I/O path, SVC architecture and key measurements.
Availability Intelligence for End-to-End SAN Performance & Capacity
Transform unpredictable outages into predictable ones by creating Availability Intelligence from the storage performance and configuration data.
Noisy Neighbors: Identifying Root Cause of Performance Issues for SVC
We demonstrate how some extremely busy volumes created problems for an entire SVC/Spectrum Virtualize cluster, and how IntelliMagic Vision helped identify them.