Why Am I Still Seeing zIIP Eligible Work?

Ghost zIIP Eligible Work  

zIIP-eligible CPU consumption that overflows onto general purpose CPs (GCPs) – known as “zIIP crossover” – is a key focal point for software cost savings. However, despite your meticulous efforts, it is possible that zIIP crossover work is still consuming GCP cycles unnecessarily.

Sometimes our customers encounter some unique situations. Jon Ulrich, at HCSC is the source of this shared experience. Perhaps, this post can equip you to identify and take actions where appropriate to get the ghost zIIP eligible cycles out of your house! After a recent deep dive into applications as defined by report classes, Jon made an observation that some zIIP crossover work didn’t really make sense for a specific workload.

In this blog, I will review this scenario, show how you might identify similar situations, and provide some examples of actions where appropriate; to get the ghost zIIP eligible cycles out of your house! I will also try to reinforce two key concepts:

• First, performance work is often a learning process and not an easy snap of fingers even when at the end of an investigation, the change seems simple.
• Secondly, and more specifically, what do you do when something doesn’t pass the sniff test in this business?

Basics of zIIP 

Before I dive into our unique situation, it’s helpful to establish a baseline of what makes sense in regards to zIIPs. This nice summary below has been provided by our own John Baker:

1. All work initially dispatches on a general purpose processor (GCP) and the “zIIP-eligible” bit is checked. If yes, it will move to the zIIP queue on the next dispatch cycle.
2. IBM exclusively decides what work can run on zIIP (zIIP eligible) – no one can control that.
3. The general purpose processors regularly poll the zIIP queue to see if they need help. We don’t know IBM’s algorithm but from observation we see that the generous tendency to ‘help’ (i.e. run zIIP work on GCP even when zIIP utilization appears low). The purpose is to minimize application delays.
4. Any zIIP-eligible work that crosses over to the running on GCP is work that could run on a zIIP processor (i.e. meets criteria #2)
5. The recommended method to limit #4 (force a workload to wait/stay on zIIP) is the Workload Manager HONORPRIORITY option by Service Class.
6. Given #1, and the fact that some instructions must run on GCP, even some ‘fully eligible’ zIIP workloads will show GCP time.

In the case of this blog we are inspecting something in category #4 above, even with #5 in play.

Identifying Excessive zIIP Eligible Work  

As noted earlier, while establishing some application definitions based on report classes, Jon observed that some WebSphere related workloads had significant zIIP crossover work that didn’t really make sense.

Figure 1 below is a report from IntelliMagic Vision showing zIIP crossover reporting by workload.

Figure 1: zIIP Eligible Reporting by workload

Figure 1 provides broad workload categories, but the two largest components W_CB (the large yellow area), and W_STC (purple) are of specific interest for this case because they are the lion share of zIIP crossover work and they happen to contain service classes that should be running mostly or largely on zIIP engines.

Is it possible that the zIIPs are too busy? A quick check there yields the result in Figure 2 below. Not at the CEC level.

Figure 2: zIIP Busy by Processor shows no zIIP constraints at the CEC level

Each system running on these two processors has ample capacity such that there should be very little zIIP crossover activity. Additionally, each system was evaluated further for sufficient online cores and weights for zIIP demand. There were no constraints. So Jon quickly had the data revealing that something was amiss and costing the organization real money in GCP MSU consumption.

Configuration Setting Active as Root Cause 

The workloads in question had zIIP crossover work, and WebSphere and Connection broker subsystems are enabled to exploit zIIP processors. Since these workloads were not executing on the zIIP, Jon chose to test the Honor Priority option in WLM (#5 in the Basics of zIIPs above) and created a new service class that should force the zIIP eligible work to the zIIP.

When that test didn’t result with the offload expected, an inquiry went out to IBM, as to why this was happening. After IBM investigated the console DUMPs provided, it was observed that the FSACCESS security-checking settings were active and indicated that the setting may have been causing zIIP crossover work.

The 3^rd party security vendor was also contacted to determine some of the impacts of the authorization setting and whether the system(s) could tolerate disabling the FSACESS checking. The setting is available to ensure filesystem access is authorized for a dataset, and this particular instance involved a USS file system. There may have been a specific condition where this particular setting was necessary, but the switch was put in to remove what is a redundant check for an authorized user. It turns out there was also some CML lock activity on the zFS identified that validates IBM observation that the setting was enabled.

Figure 3: Reduction of Suspended Tasks on CMS Latch Lock & Security Setting Access Change

A quick look at some aggregated history in Figure 3 shows the suspended CMS Latch Lock tasks dissipating after the FSACCESS change was made.

The Big Save 

This is a clear example of common sense and perseverance paying off. Evaluating an obscure, seemingly unrelated setting change takes time and energy to chase down and understand the details and then it takes more time to work with your software vendors.

An access security software setting is not something with obvious performance related impacts. Figure 1 shows the performance side of the value proposition at stake. One CP of work running on a GCP when it could run on a zIIP.

If peak demand exceeds 40 processors, saving one CP seems like a small percentage. However, due to the scale of most mainframe systems, these small percentages reap large ongoing software cost reductions, and hardware cost avoidance going forward.

The MSU consumption savings represented in Figure 4 helps your executives value the performance improvements you pursue since MSU consumption is the primary driver of mainframe software expense. Executives love this stuff and often give public acknowledgement for individuals who persevere with these types of results. We also believe one customer’s experiences often benefit others and we are thankful for the partnership we have with HCSC.

Figure 4: MSU Consumption Demand by Workload following Setting Change

A Bit More on zIIP Eligible 

IBM provides a list of zIIP exploiters. We’ve provided various aids (blogs), and others (W&W) have, too. So when there still remains some significant zIIP crossover work, it doesn’t pass the sniff test. Since Db2 is the biggest exploiter of zIIP, I would recommend you to one of the leading sources of knowledge in this arena, Robert Catterall. His blog also provides some good guidance.

Of course, IBM isn’t the only vendor that is authorized to exploit zIIPs, but it’s not always well documented, so you would be wise to ask about it in your next software renewal. As with many of you, we still may find ourselves unsatisfied that things are misbehaving. Asking IBM about zIIP exploitation and other savings options is also always true when you are renewing your IBM software. If you’re convinced there’s something going wrong, what’s next?

First, there must be some data that demonstrates there is zIIP eligible work and one needs to have done some poking around to confirm there’s not a valid reason for it. A quick export of a report like Figure 1 sent off to IBM may trigger a conversation that begins the journey to an answer.

zIIP crossover reporting might be a common report, but having some help on whether something is misbehaving is best. The visual queues (warning, exception) and drilldown capabilities available with IntelliMagic Vision make quick work of this type of investigative work.

This investigation required little time to get to reports, but the journey may take longer as you clarify details with software vendors. A few quick reports can help eliminate other zIIP crossover causes mentioned in the Basics of zIIP.

Share Your Latest 

After making the discovery, and implementing a successful change like this, it would be easy to tuck this gem away and quietly receive the acknowledgment. I get the privilege of having a front row seat to some of these experiences as we partner with folks like Jon at HCSC. I hope by sharing this gem, we can encourage you to listen to your instincts when something doesn’t seem right. If it doesn’t pass the ‘sniff test’, we would love to hear from you. Please let us know if this example is something you would like to explore further, or if there is another mystery we can help you solve.