Jack Opgenorth - 4 December 2018

Despite the growing number of reports that raise concerns with maintaining your mainframe performance while outsourcing z/OS infrastructure, the appeal of outsourcing’s promised benefits has yet to wear off.

While having a conversation with a former transportation industry IT executive, we observed over 40 people in the lobby for a new employee orientation that day. This interesting point was also shared: the company was in the midst of a 5-year plan to move the workload completely off the mainframe!

The original plan estimated this would occur in 5 years. When that plan was announced, the company lost close to 50% of their mainframe experienced employees. Now it’s looking more like 10 years, and my experience is that 10 years is still a low estimate. This result is not really surprising, but it speaks to the pains of change.

An IT Soldier’s Perspective on the Outsourcing Front Lines

With Veterans Day recently behind us, I want to honor our veterans and thank them for their service! While I have not served our country in this way, I have borrowed the soldier metaphor for this post.

A recent post on LinkedIn read something like this: “…after almost 20 years, the layoffs finally caught me in the net. Essentially, I was always working ‘for’ company A in one way or another. I went from company B to C to D, & E (all outsourcing for company A), and then finally back to A again once outsourcing didn’t magically solve all their problems as promised!”

It wasn’t surprising that the post was followed by several “me too”’s! As a veteran IT soldier, I have experienced similar uniform changes, and I have seen skilled, reliable people become expendable. I’ve soldiered through the outsourcing continuum:

  • the beginning, including the development of the RFP (Request for proposal)
  • supporting a number of accounts while working for an outsourcing company
  • untangling these agreements on the client side as new leadership drove for more direct control of the infrastructure delivery (specifically in the performance and capacity management arena)

This dialogue is also painted primarily with the colors of a z/OS performance practitioner.

Thoughts on Mainframe Performance Outsourcing

Many times, it seems that outsourcing a mainframe and migrating applications off provides justification to bypass all investment in the care and keeping of those systems before the critical applications are actually migrated. After all, those IT assets and the care of them will eventually go away.

In fact, business always has limits on what they choose to spend the IT budget on. Before you blame the outsourcer completely, you need to look in the mirror, too. As our partner Cheryl Watson has stated: “I have never met an outsourcer that doesn’t try to do the very best they can for their customer.”

Risks of z/OS Migration and Outsourcing

Migration and outsourcing can be a big mistake, and some folks are taking another look. Why?

  • Risks increase dramatically when large migration projects delay by a factor of two or three.
  • Skilled employees find other pastures quickly if they believe their jobs are at risk.
  • Mainframe migration and consultant costs involved with that process might be piling up far faster than expected.
  • Preventable outages occur due to lack of oversight of the infrastructure performance.
  • The database of record is on the mainframe and the core applications using that data may still be growing rapidly driven by mobile apps and general economic growth.

Another recent LinkedIn post for a Senior Performance Engineer Consultant position confirms the desire by some organizations to take out some insurance as they mitigate the risks above. With a bit of context, you would find that some of these postings originated from a company that has already outsourced IT infrastructure management.

In effect, this position is outsourcing insurance. It confirms that outsourcing agreements are both optimistic in assessing the particular requirements for technical skill retention, and technical skills are being lost when migrations and outsourcing occur.

Outsourcing Skills Gap

In my experience, when the outsourcing agreement divided groups, the performance and capacity team was split between the client and the outsourcer. Within a year, the client was trying to hire one of us back to help them meet the skill loss on their performance team.

That kind of recruitment isn’t cheap for the client. The provider side of the fence gets messy and is likely to vary, but my experience indicates that providers are optimistic that their scaling capabilities can meet the demands of their clients. This often results in an expanding drain on people required to meet their client’s needs.

First, staff reductions are common within year one of an agreement being signed where staff is often cut to meet the budgeted savings to meet financial commitments made to get the deal. This has a secondary effect on the remaining quality talent. Those folks see that the new goals will be met with more hours on the job for those that remain.

In today’s economy, it’s easier to become a well-paid freelancing mercenary with less stress and more engaging work.

Finally, the nature of outsourcing agreements drives some of the passionate performance people away because the outsourcing cost models are driven by IT consumption. Therefore, outsourcing providers often lose skilled talent that choose to pursue their passions elsewhere after cuts are made which compounds the skills gap. Personally, this was one of the primary reasons I pursued a change.

The nature of outsourcing impacts to organizations creates ‘scar tissue’. For some IT disciplines, this may drive a need for a net increase in folks with specific skill sets because the processes still work, but there are valid reasons for things slowing down.

Furthermore, outsourcing drives more complex collaboration across all aspects of IT delivery. The expansion in complexity drives increases in technical resource time to deliver.

If you’re looking for help, how can you influence the situation? There are actually a few options available to you.

How to influence the outsourcing situation

1. Establish a shared cost savings clause in your outsourcing agreement

Cheryl Watson, one of our partners, has suggested this again recently, and I totally agree!

Establishing a shared cost savings clause in the outsourcing agreement has several benefits:

  • Reduces your cost!
  • Motivates outsourcers to deliver cost savings ideas
  • Helps prevent some of the skills loss outsourcing may otherwise cause. Keep those passionate performance folks around and engaged.
  • For growing mainframe workloads these savings amount to significant ongoing cash flow that can be used for investing in your other IT initiatives.
  • It may help tilt your technology debt by providing new opportunities through investing in compression (zEDC) and how you operate your mainframe workloads. Are you still running z/OS at 100% CPU because you think that’s the best way to reduce your MLC?
  • Outsourcers could leverage wins on one account where shared savings are happening to win new business from others that continue to bill by demand only or apply it to at risk accounts and retain business.

2. Establish collaborative practices that include reporting that is not ‘useless’

The adjective ‘useless’ was quoted from a prospective client describing the current provider’s reporting. The provider is often not at fault, rather, the vague agreements and use of archaic existing reporting are.

The existing reporting may already be one step away from extinction and there’s a reason for that. Many folks have created their own systems with available tools. Meanwhile, many vendor solutions are just under-performing.

Unfortunately, this only enables Splunk to help customers ‘re-invent’ costly new views, without any of the expert knowledge that was included in the older reporting. I have seen ‘useless’ reporting and tried to help folks see the need for improvement with an outsourcing arrangement and it is NOT easy.

3. Establish collaborative reporting solutions that will promote cooperative management of IT.

Good solutions include:

  • Automated analysis supported by best practices, and embedded expertise
  • Helpful visualizations that can be easily shared
  • Actionable information that can be easily shared
  • Flexibility to meet the needs of dynamic situations and complex systems.

Check out this white paper for more thorough examples: 5 Key Attributes of an Effective Solution to the z/OS Performance Skills Gap

4. Develop a regular communication plan around key reporting areas

5. Engage Advisory expertise to come alongside existing staff to:

  1. Help fill staffing shortages
  2. Develop new staff
  3. Identify and prevent IT soft spots that may contribute to future availability loss.
  4. Identify opportunities for savings (i.e. MLC, reducing technical debt)

This summary of ideas is not comprehensive, but I hope it helps you think more clearly about some important issues you will likely encounter regardless of your role in IT.

We would love to help you improve your relationship with your outsourcer, your client, or to manage transitional mainframe strategies. We will work with you to reduce your costs with our solutions – which include software, software-as-a-service, and consulting. Contact us here.

This article's author

Jack Opgenorth
Senior Technical Consultant
Read Jack's bio

Share this blog

Health Check for Coupling Facility

Subscribe to our Newsletter

Subscribe to our newsletter and receive monthly updates about the latest industry news and high quality content, like webinars, blogs, white papers, and more.

Related Resources

Video

Integrating z/OS Performance Management with Splunk & Enterprise Dashboards

IntelliMagic Vision can export data into a Splunk-ready format, after which it can be imported into Splunk. This means not much data is ingested into Splunk, which can be a significant cost-saver.

Watch video
Video

Forecasting Usage and Capacity Growth

Processors are expensive, and that is why it is crucial to have an accurate understanding of how much capacity you are using and have left before you run out.

Watch video
Video

Understand z/OS Workload Peaks in Context

By being able to compare any chart or dashboard with any previous time interval, performance analysts can quickly determine whether a certain workload peak is normal or unusual.

Watch video

Go to Resources