Are you afraid of a dark corner of your codebase?

In multiple teams I have worked on or collaborated with, I have seen the same pattern. Over the years, an old system emerges. One that is in “maintenance”. No active development is needed. So there is no reason to keep upgrading it. Let’s call it the Linchpin system in the basement. New hires aren’t trained on how to work on it, cause it’s hard, and not needed. When eventually something has to be changed on it, everyone is dreadful. The dev working on it will be giving the “Trying to get X to work” update for an unknown number of standups. The reviewers will be giving the “The change I see looks logical to me, although honestly I have no idea” approval. And in the end it will work. And will stay like that. Until the next small change is needed. Cause we shouldn’t spend the time / effort on it. After all, it will be retired / rewritten soon. No really!

But as the greek proverb goes, and as all developers have come to learn at one point or another:

Nothing is more permanent than the temporary

You are not alone

Believe me, I am not judging you. It was a couple of years into my first job at CERN, when we gathered to celebrate the retirement of such a system. Back then, I knew it was important, but couldn’t fully grasp why the need to “celebrate”. We even brought champagne, and extremely happy colleagues from infrastructure and security. Why? Cause they would not longer have to run special machines running Java 5, a decade after its end of life, so that people could connect to them virtually, and then connect to our Linchpin system in the basement. Although I am not sure why, after all it was super user friendly. You didn’t even have to write your whole password correctly, the first 6 characters were enough #greatUX

In my last team, work on our Linchpin system in the basement’s retirement started… 12 years ago. But now it’s definitely being retired. Soon. As in, this decade. Although I have to admit, having to wait for the next multiyear shutdown of the largest machine ever built so that multiple teams are allowed to update their critical systems and use your new API with thousands of devices, is not what we usually call “fast feedback loop”.

Shall we blame the language?

Usually such a system, because of its age or other reasons, may be written on a different language / framework than the rest of what the team uses. Which is the first thing to blame. After all, no one knows X these days. We hire for people with experience in the stack that the rest of our systems are written in. And yes, we do have old team members who used to work more with this system, but even they have, over time, lost the ability to confidently work with it. They must have been forgetting the syntax… After all, it’s old and quirky. Look. A modern language’s console would never evaluate this to true. Right? [1]

    {} + [] === 0 // Looks like an undeniably true statement to me

However, didn’t everyone in your team at one point or another have to work with languages, tools and systems they didn’t know? Didn’t you? Haven’t all programmers? Why is it that they were able to do it for everything, multiple times at different points in their careers, just not with the Linchpin system in the basement? Sure, its documentation is outdated, and while there are some “ancient textbooks” acting as guides, you can’t fully trust them; they are probably stale. Very different from every other piece of your documentation, which has been meticulously kept up to date after 5 years and which you trust blindly, don’t you?

So the question arises. Who is the real culprit?

Wait, it was the tooling? - Always has been

Developers need their playground

Non developers always talk about the skill of “writing code”, how hard it is, etc. Developers, talk a lot about the skill of “reading code” and its importance.

I love scuba diving. So I’ll steal a metaphor someone I know used very successfully.
Before doing my first underwater dive, I had done the theory. Read about things, both the theory of scuba diving, and the “practicals” of it. I was shown the equipment, and could tell you what each part did, what it was for. And I could tell you what I had to do when underwater, in the different situations. Which means, when I found myself for the first time ever 10m underwater, I was completely calm, my system was not panicking thinking “I can’t breathe”, my buoyancy, my balance, was perfect and steady. Right?

Reading and understanding how things work, is a crucial skill, that all developers need to train. And today, AI agents can assist even more in understanding a legacy system. But it’s not enough. We, developers, need our playground. Need to be able to tinker. Experiment. Break. And fix it again. Change, build, deploy, test. That’s how deeper understanding builds over time. That’s why even the people who worked on the system before, become less confident when not actively working on it over time.

So, the first thing to look into, is the tooling, the infrastructure. Do you have at least a Dev environment, where a developer knows they can safely play around without destroying everything? Does it actually match production, or is it “similar” cause over the years fixes and changes have been done directly to prod? Do you have local environments where I can tinker with things and get fast feedback?

If you don’t, you need to set one up. Now. At least a dev environment. Document the process. Document what it takes to deploy. - But, this system is in maintenance, it’s not being actively developed. It’s a waste of time.

Is it? When the hard disk of your current production system breaks, how long will it take you to be back and running? Minutes? Hours? Days? If you think minutes cause at least you have backups, have you actually tested deploying from them? If yes, congratulations, setting up a dev environment should be easy. If not, then the conventional wisdom is clear: you don’t have backups.

With a dev (and hopefully local?) environment set, developers are now able to tinker with a system. They are confident they can deploy and see changes on Dev, understand what’s going on. The small fixes that people always wanted, but you skipped cause it’s not actively maintained, can become quick wins, which will allow the knowledge to rebuild in the team. Cause that knowledge is important, not only for the operation, but also for the retirement of the Linchpin system in the basement. To be able to have discussions and options about migration strategies. To be able to setup analytics for the migration, or before that, for the design of the new system. To not have to replicate a black box sitting in a scary corner, but to understand the good and bad parts of it, what you want to keep or not, and what is a mandatory piece.

So that, when the sudden requirement to update all company wide systems to the new SSO comes, even if it’s a system built on libraries which can no longer be found online, you will be able to do it. So that, when you need analytics, to figure out which methods of the old API are mostly called and how, which users would make for good pilot users, which users can migrate today, you will be able to set them up. So that, one day you can open a champagne with your colleagues, gather around and tell the new hires about the old horror stories of the Linchpin system in the basement. Or if you have left the team, get an invite in the next few years to visit and celebrate. Hopefully this decade.

References

[1] Obligatory callback for anyone who has at least once in their life exclaimed “Wat?”, even though it’s details are always more interesting