How many mainframers does it take to change a lightbulb?

By Chris Harvey, Senior Technical Delivery Manager

The answer is nobody knows: Dave did it for 30 years but he retired last month.

Joking aside, the serious question is: how many people are holding your mainframe together? Long-term resilience is about more than duplicated hardware. It means looking after the people, knowledge and processes that keep the platform running.

My previous article described a situation when we had five days to take over a client’s mainframe service. That was the sharp end of managed services: get in, understand what’s going on, keep the lights on and, preferably, don’t make a drama out of it. But there’s another side to the coin: the work you do before something becomes a crisis.

Because sometimes a single point of failure isn’t a piece of hardware. It’s Dave. And Dave has just retired.

Four people isn’t much of a safety net

It’s a pattern we see quite often. Once, a mainframe team might have had 15 people. Over the years, through natural attrition, better software and changing requirements, that becomes ten, then six, then maybe four or five.

Which is fine. Until it isn’t. Good luck, Dave, and thanks for everything.

Those four or five people may have been looking after the same environment for 20 or even 30 years. If one retires and another leaves, that’s a third of the team and an even larger percentage of its accumulated knowledge. Unfortunately, you can’t quickly recruit 30 years of experience on a Tuesday afternoon.

In a trend that’s evident across the industry, Vertali is working hard to bring younger talent into mainframes – creating what I sometimes call a valuable blend of the “old & wise” and the “young & hungry” – but knowledge transfer takes time. That’s one of the advantages of a managed service model. The knowledge stays with Vertali, not a person. If someone retires, goes part-time or moves on, the service and the accumulated understanding of it remain.

This brings me back to the runbook, which I mentioned in my previous article.

Write it down. Now.

The runbook is the bible. When Vertali starts working with a new client and asks what documentation they have, the most common answer is somewhere in the region of “next to none”. Technical people are generally good at fixing things. But they are not so good at writing down exactly what they fixed, why, what else it affects, and what somebody should do next time.

Undocumented knowledge is potentially unlimited risk.

That’s why we build a runbook with a client. What software is running? Who supplies it? Are licences synchronized? What is current and what is heading out of support? Which systems matter most? Which processes exist for a very good reason? The point isn’t documentation for documentation’s sake. It’s making sure the next person doesn’t have to rediscover 30 years of accumulated “Dave knowledge” under pressure.

In other words, the runbook means 30 years of experience don’t simply walk out of the door.

Testing is better than hoping

Disaster recovery is another area where plans can look reassuringly complete, right up to the point when something goes terribly wrong. DR is a bit like an insurance policy. You hope you never need to use it, but you need to know it’s there and that it works.

Having a second site or another mainframe available is a start. But can you actually fail over to it? Do the licence keys work on the alternative hardware? Does everybody know the process? Who makes the decision? Who gets called at two in the morning?

We recommend testing DR at least annually.

There’s a slight irony here. The more risk-averse an organization is, the harder it can be to persuade it to conduct a full DR test, because the test itself introduces some risk. I understand the nervousness. But finding out during a real-life outage that your recovery plan doesn’t quite recover anything is an expensive lesson. I heard about one organization recently whose data center shut down because of heat. That is when the diagrams stop being theoretical.

Stuff occasionally breaks

Incident management and change management are closely connected for a simple reason: making changes introduces risk. That doesn’t mean a change that creates an incident was automatically badly managed. You’re working on live, complex systems. Things happen.

What matters is what happens next.

When you have an escalation, you want the technical experts concentrating on fixing the problem. They don’t need to stop every ten minutes to explain themselves to five different people. Someone else handles the communication: what’s happening, what’s the impact, is it under control, and when will we know more? And this isn’t the moment for an enthusiastic root-cause analysis. Restore the service first. Work out exactly why it happened afterwards.

If something goes wrong in that window, it’s going to be fixed. That’s what the managed services client is paying for.

Up to 25 people and back down again

Resilience also means being able to change shape. A company may have a perfectly adequate team for BAU then acquire another company or begin a major modernization programme. Suddenly a team of five or ten needs to become 25 for the next 18 months. And recruiting a permanent team for a temporary requirement doesn’t make sense.

A managed service lets you scale up as needed and scale back down afterwards. Crucially, the people responsible for BAU remain involved, so the knowledge created during the project doesn’t disappear when the project team does. Continuity matters.

Flexibility matters more than a fixed answer

If I had to pick one thing an organization should look for in a managed services partner, it would be flexibility. What you need today may not be what you need in 18 months, never mind five years. You may begin by needing one specialist to cover somebody approaching retirement. There goes Dave. Again. Later, you might need full managed support. Then a major project arrives and you need additional skills for a year.

The service has to move with you.

And, for all the technology involved, relationships matter. Get the teams talking directly. Meet face to face when possible. Break bread together. Build enough trust that people will pick up the phone before a small problem becomes a large one.

Resilience isn’t a single product or one process. It’s having enough people, preserving the knowledge, documenting what matters, testing the backup plan and knowing who does what when something goes wrong.

So when Dave retires, you can wish him well, safe in the knowledge that he isn’t taking the mainframe out the door with him.

Time for one last joke. How many mainframers does it take to change a lightbulb? The answer is one, plus a change request, a rollback plan, a maintenance window, and somebody on standby at 2am.

I’ll get my coat…

I’ve recorded a podcast on the subject titled “Beyond the Single Point of Failure: Building Long-Term Resilience in Mainframe Operations” which you can listen to on the Vertali Podcast page here.