The complexity of abundance

Why more is not easier...

(Like this article? Read more Wednesday Wisdom!)

My first computer was small. Very small indeed: 1 lovely CPU at 3.25 MHz and a smashing 1 KB of RAM. Pretty soon I upgraded to something beefier, but the size and scale of the computing power on my desk was still not much to write home about. 

Also, we had no email, so there was nothing to write home about with 🙂.

Because of the limited CPU power and RAM, every bit counted. Even my second (beefier) home computer (the 1980s BBC microcomputer) had only 32 KB of RAM that needed to contain the program, the system variables, and the screen buffer (the OS and the BASIC interpreter were in 32 KB of ROM). Everything was either small, slow, or just both. The lovely book “Acorn, a world in pixels” is full of stories about programmers pouring over their source code because they needed to save 12 bytes while trying to achieve things that were clearly impossible.

The game Elite was known to be impossible to write…

Fun fact: The A in ARM stands for Acorn, the company that made the aforementioned BBC Microcomputer. If you are reading this article on a mobile device or on an Apple MacBook with Apple’s own silicon, you are using their tech.

Growing up in that world made me frugal. I wrote software for PCs and I did so on my IBM PC/XT compatible computer with an 4.77 MHz 8088 CPU and 512 KB RAM. I continued to do that even when the market had already progressed to IBM PC/AT compatibles with an 80286 CPU at a whopping 12 MHz.

Sometimes these came with a “turbo” option that increased the clock frequency to an even whoppier 24 MHz! <mind blown>

Being frugal I never saw the need to upgrade my development system: I just continued tinkering with the software until it fit into my computer’s memory banks and was fast enough for my measly CPU and super slow hard disk (I used Stacker to increase available disk space using compression, making it even slower). The upshot of this was that when I went to the customer site (who typically had bigger and faster computers than I did) the software was figuratively flying.

Pet peeve: People who say “literally” when they mean “figuratively”. “I literally died!” Uhhh, no, you did not and the English language is mourning your continued existence.

In my first real job I worked on “big” iron. Mainframes! Multiple CPUs! 64 MB of RAM! Disks that could store 1.8 GB starting at a modest $97,650 (in 1980s dollars).

Now you’re playing with power! 

Big though the iron might be, it was still not big enough for the needs of the business. To the extent that we ever spoke with business people, it was usually to tell them that whatever they wanted was technically not possible; we just didn't have the computing power.

Software engineers were like a cast of high priests that knew the ins and outs of the magical machine. That knowledge was important because pretty much everything we did required us to squeeze the last drop of power from the CPU, RAM and I/O devices. That requires wizardry and we were the wizards. It was cool and I loved it.

Fast forward 35 years.: We went from scarcity to abundance. Hurrah! Abundance!

But uhhh, writing software is still slow and difficult. How did that happen?  

In the world today pretty much the only problems left are organizational ones: To do what we want to do we need to write lots of software and that new software needs to work well with the software we already have. That means problems, and to solve these problems we need lots of people, documents, Zoom meetings, and time. Lots of time. I am regularly amazed how long it takes to get something, anything, done and how much time and effort it takes to agree on something.

Regardless of the enormous amount of computers we have now, and regardless of the size of these computers, our world is still a world of complexity. No longer the complexity of scarcity, but instead the complexity of abundance.

In this world of abundance people are the scarce good. You can see this in the way we use the term “resources”; 35 years ago, if you said that you couldn't write a program because you didn't have the resources, that usually meant compiler time, RAM, or disk space.  In contrast if you say that you can't develop some piece of software because you don't have the resources today, you most likely mean that you don't have the people. RAM and CPU cycles are really not an issue anymore. People are.

In our industry everything becomes faster, smaller, and cheaper, except for the software engineers…

Another problem with abundance is that there is a lot of everything already and how should it all work together? And how should it evolve? And how do we take the mountain of data and functions that we have implemented and turn these into meaningful experiences? It's really not obvious. I spend heaps of time wondering about relatively simple questions like "Is this person in this database over here the same person as that one over there in that other database?" You think that'd be an easy question but often it is not.

A former colleague of mine was blessed with twin girls, both of whom he gave first names starting with M and no middle name. That's how you find out which government institutions have their administration in order and which ones don't. Eventually the girls had to invent a middle name for themselves that they used for dealing with organizations that couldn't correctly register two different people with the same first initial, last name, date of birth, and address :-)

Another complexity created by abundance is expectations. I am already getting annoyed if Amazon takes two days to ship something and I am not unique: Customer expectations are sky high because in most people’s eyes there is no obvious reason that something cannot be done. And indeed, very few things are beyond our technical means. Meeting the customer expectation became a competitive necessity, but even with an abundance of computers, lots of code still needs to be written. And I wrote before that that needs resources and these resources need time to figure out how it is all going to fit together.

On top of that, at some scale things still get really expensive really soon, even with cheap and powerful computers. Frugality still has a role to play, but it’s at a completely different level now…

The next problem with abundance is that it leads to more complex choices. For one project I am working on we are going to pre-explode a data structure that is optimized for a particular type of access into a bunch of denormalized views that afford other types of access. The number of choices we have for storing these views is incredible! A key-value store (and if so, which one)? OpenSearch? A relational database? Some columnar storage engine?

Abundance of choice comes with the existential fear of making the wrong one, so people fret about that endlessly. This kind of choice is also the ultimate bike-shedding topic, leading to significant delays.

Finally, abundance leads to clutter. You never have to throw anything away, never have to optimize, it's always easier to just add to the heap. Scarcity breeds clarity and abundance breeds confusion. Maybe that's why I keep on meeting lots of confused people. There is so much of everything that nobody seems to know anymore what to do with it or how to go about it. In the olden days life was easy: You stored things on a tape or you didn’t store them at all 🙂.

The upshot of all of this is that I have more computing power than I ever imagined but writing good software is still hard. Not just because the goal posts have moved, but also because abundance creates its own complexity that we now need to deal with. Maybe that's why in my personal life I keep toying with smaller and smaller computers. I started with Raspberry PIs running Risc OS, but pretty soon this was followed by even smaller computers based on the 65816 or the eZ80. In my work I have to deal with the complexity of abundance, in my hobby I want to deal with the complexity of scarcity…

And there it is: Everything that is old is new again, if not be necessity, then at least by choice :-)