The devil is old

Age and experience are the sources of wisdom

(Like this article? Read more Wednesday Wisdom!)

One of my favorite sayings in Spanish is: “Mas sabe el  diablo por viejo, que por diablo”. In the accidental lingua franca of the world: The devil knows more because he is old than because he is the devil.

Imagine that for a moment: You are the literal devil. You are an angel who was there at the start of the universe and you probably had a few good looks at the celestial design documents that kicked it all off. Maybe even wrote bits of it. You were probably the tech lead of a whole gaggle of angels and other celestial beings who hammered the universe together and you were most likely deeply involved with the UAT and launch of the whole thing.

Since then you fell out with the big product manager in the sky and for your sins you were assigned to the risk & compliance department, but regardless you still sit on a mountain of knowledge about it all because you were there and designed key parts of it.

You were assigned these momentous tasks in the design and implementation of the universe because of your education at a celestial Ivy league school and your excellent grades in quantum mechanics. You are on top of the latest research in universe design and implementation. However, when others evaluate the source of your wisdom, it is not ascribed to your education, hard work, and innate talent, but "just" to you having been around for so long…

I resonate with the Spanish saying because if I can make any claim to some “wisdom” in the field, it is because I have been working professionally with computers for longer than computers had existed when I started. Being around for so long means that I have made all of the mistakes and, since experience is what you get when you were expecting something else, I gained knowledge about what to do, what not, and why. I apply and share this knowledge liberally; people sometimes mistake this for wisdom.

Because of my long track record in the field, I start many monologues with: "In 19xx, I was on a project that ...", after which I go on to explain what happened and why the proposal on the table is not a good idea. This probably annoys many of my younger colleagues, but they are nice enough not to complain too much. At least not to my face.

One other, probably equally annoying, thing I say is: "Well, I have seen this tried a dozen times before and I have never seen it work, but maybe this time we'll get it right".

"But Jos", I hear you think, "surely things change over time, especially in technology, so how come that ye olden experience is still valid?"

You are absolutely right that our field has experienced large, deep, and significant changes. We have more bits to play with, more places to store these bits, and the bits move faster from where we are to where we want to be. In the olden days, customers would come with their business requirements and I regularly had to say: "Nope, not possible, we just don't have the computing power to do what you want". That is never the case anymore. There are still some unsolved problems, but regular business requirements are pretty much all solvable these days with commodity hardware that I can get for not a lot of money at Amazon or Best Buy.

But please get it at Amazon, because every time you click "Buy", a tiny little bit of your spend goes to me 🙂

But here is something that hasn't changed: People.

Using the technology we have at our fingertips, a single developer can write a mobile phone app that does facial recognition and shares information with other instances of the app around the world with the same amount of effort a developer of yore needed to write a BASIC program to balance their checkbook.

Question from a European, WTF is actually "balancing a checkbook"? I've heard the phrase coined hundreds of time, but I got rid of checks in the early 1990s and hadn't seen one until I moved to the US, and even here I write at most one a quarter or so because, in essence, the US banking system is stuck in the eighties, which was a great decade to remember for its music and hairdos, but not for its consumer banking.

However, groups of people are about as good at writing software together as they were in the 1970s.

Which, by the way, explains why Fred Brook's book "The Mythical Man-Month" (written in 1975) is a timeless classic.

When writing software in groups, it is all about clarity of communication, effective leadership, good prioritization, and excellent planning. We are unfortunately not any better at that now than we were in the past, even though we now have spreadsheets and Microsoft Project. Working in groups really hasn't changed much, which means that everything I have learned over time on how to do that, still applies.

Two other things that haven't changed since the olden days are math and physics. State machines today work exactly like state machines in the 1980s (or the 1890s for that matter, or the 980s, 80s, or the year zero). And since people really haven't changed, by and large people's understanding of state machines haven't improved much. We still insist on not explicitly designing the state machine that underlies our software and keep hiding it all over the place so that there is every opportunity for invalid state transitions. Same for physics: The speed of light is still the speed of light and I keep having to explain to people what the consequences of its limitations are, like that even with the best networks money can buy, doing an intercontinental two-phase commit is going to take half a second or more.

So because math and physics haven’t changed and people haven’t gotten smarter, every generation of engineers has to learn that “once and only once” delivery of a message over a lossy channel is fundamentally not possible and that the best you can do is "at least once”, which means that you have to take measures to deal with receiving duplicate messages. Over the decades I have pointed hundreds of people to the "Two Generals Problem" page on Wikipedia to try and make them understand.

The last time I did this was literally yesterday (at time of writing).

And yes, literally. One of my major pet peeves is people saying “literally” when they mean figuratively. “I literally died”, I once heard someone say. No you didn't, but for the English language it might have been better if you had!

One thing that actually has gotten worse is people's grasp of basic math.

Every now and then I give workshops on how to do well in system design interviews and I always start with a basic math test that the attendants need to solve with only pen and paper. This involves questions like: “I am reading a 1 TB file from a disk at 40 MBps. How long does it take me to read the file?” Shockingly, even a group of CS PhD students I once taught, didn't manage to answer all these questions correctly.

This is important because people keep making the same mistakes because numbers are hard. I once sat through a design review where the team defended their storage solution choice with an argument based on the incredible amount of information they were processing and storing. I then led them through an exercise that showed that even under the most exaggerated conditions they wouldn't store more than a $25 SD card bought on Amazon could comfortably hold.

In the olden days people couldn't do the math and thought that our massive 1.8 GB hard disks could hold their data and today people can't do the math and think that tiny flash disks cannot hold their data. Same problem.

There are other areas where people as a whole really haven't improved. Here is one example: Concurrent programming. People cannot write correct multithreaded code if their life depended on it. It is just a limitation of our brain. The difference between me and an L4 just out of college is that I know that I cannot write multithreaded code. Literally every time I do that, I develop some huge deadlock or something that takes extensive testing to find and fix. And, for reference, I used to teach concurrent programming in college 🙂.

Even in programming practices we really haven't progressed as much as we like to think. Yes we have RPCs these days, but all complex systems I deal with use a lot of pre-historic techniques, including golden oldies like screen scraping and transferring information using hourly file dumps. The screens are browsers now (instead of 3270 terminals) and the files get transferred using SFTP to an S3 bucket instead of using a tape, but the architectural principles, and therefore the successful and not-quite-so-successful patterns stay the same. Everything I need to know to implement UIPath’s screen scraping reliably I learned in 1988 while scraping IBM mainframe terminals.

That is why the devil's age is so important. He might understand all the math and physics in the world, but what makes him really good at his job are his millenia of experience in what people do with all of that. He knows their mistakes, behavior patterns, modes of thought, inability to learn, tendencies, and above all, weaknesses.

The devil has seen it all.

And that's why the devil is good and annoying, and starts every explanation with: "Let me tell you about this thing I was involved with in the middle ages which didn’t work out so well either…"