Chaos is the price of freedom

(Like this article? Read more Wednesday Wisdom! No time to read? No worries! This article is also available as a podcast).

I started my career as a systems programmer on IBM mainframes. We ran IBM’s operating system, used IBM’s compilers, and even paid money for IBM’s sort program (DFSORT). All of this was software, so of course it contained plenty of bugs, but here is something that was (and is) amazing: The amount and quality of the documentation, training courses, and support. For instance, the current DFSORT manual runs to 958 pages and contains extensive explanations about how to use the tool to, well, sort data. Eat that, Linux sort manpage! Another example: The mainframe’s storage management system (DFSMShsm) has an introductory book that runs to a cool 498 pages and it is supported by a handful of courses from IBM’s training division. The software might be expensive, but the support is amazing.

For all of its amazingness, the platform is of course a completely closed ecosystem. IBM has started to support some open source software on their mainframes, but the core OS and all essential services are firmly controlled by IBM. This also means that the speed of development is slow as molasses. For instance, it took IBM years to start supporting TCP/IP because, obviously, it takes years to write decent software and to write the 1,024 page supporting manual detailing the ins and outs of the File Transfer Protocol (FTP). In the meantime, where were their customers going to go? To a compatible mainframe that did support TCP/IP? Imagine a satanic laughing sound here…

Compared to the walled garden approach of IBM, the open source software world is a chaotic jungle and always has been. For example: When Linux started making inroads in the 1990s, one of the more annoying aspects was the ongoing and very confusing battle between libc5 and libc6. These were two libraries that both provided core functionality and that contained compatible but competing implementations of basic library functions like standard I/O and mathematical operations. As I understand it, libc5 was an early fork of the GNU C library (glibc), created because the Linux people wanted to move at a higher velocity than the GNU project (whose main target was probably a GNU Hurd based OS) allowed. Also, the Linux people had specific practical problems they needed to solve and they were on a fast-paced quest for world domination. In the meantime, the GNU people continued to develop glibc, which was eventually released in 1997 and which, in the Linux world, became known as libc6.

By that time, there was lots of software around that depended on libc5. The two libraries were at the same time mostly compatible and annoyingly different, which created all sorts of chaos. Binaries built with libc5 could not run on systems that came with libc6 installed and vice versa. It was also not at all trivial to have both runtimes installed at the same time, leading to cryptic error messages that were beyond anybody but the most skilled hackers.

In a sort of “everything that is old is new again” kind of way, we are there again with MUSL versus glibc.

Coming from the IBM world, the libc5/libc6 chaos annoyed me greatly, accustomed as I was to IBM giving me (mostly) working core operating system libraries. On the mainframe, I never had to worry that a program I compiled would not run on another system for reasons such as not having the right modules in the system path (give or take the odd incorrect configuration of the LINKLIST).

I was exposed to similar chaos recently when I decided to create a crypto token for Mrs. Wednesday Wisdom on the occasion of her birthday.

It is called the ”Li$a Coin”; ping me if you want some 🙂…

While researching how to accomplish this feat, I was confronted with quite a lot of chaos, such as incompatible blockchain technologies, different chains using the same technology, a choice of programming languages, competing wallets, things called “faucets”, and various API providers, all vying for my attention. To add to the confusion, the documentation was mostly missing, incorrect, outdated, incomplete, or presumed a level of knowledge that I didn’t yet have. Of course I eventually managed to get the token out of the door, because I know how to debug, but it required some persistence on my part…

In chaotic environments, you tend to make a lot of mistakes. For instance when I tried to deploy my new ERC20 compatible contract on the Ethereum mainnet, I was very lucky that my test wallet contained only a very small amount of ETH because only after the deployment failed did I learn that deploying on the Ethereum mainnet is a very expensive operation.

I assume the Ethereum people learned this particular trick from AWS: Make deployment easy but expensive and arrange it so that people only find out after the fact 🙂

I also learned that my first choice of a wallet was not able to export its private key and that you need something called a “bridge” to send cryptocurrency between different but compatible blockchains.

Chaos seems to interfere with efficiency. Surely, if I want to hack out a smart contract quickly it would be optimal if there was clear documentation and other resources in place to support you in getting the required solution out of the door fast. If you want to sort some data on the mainframe, the 958 pages of the DFSORT manual, with all of its examples and explanation of every possible error message, is a great place to learn how to start sorting.

Chaos prevents easy entry into the market. Back in the 1990s, if you didn’t understand a lot about Unix and C already, the whole libc5/libc6 mess was very hard to deal with and might easily convince you to throw your hat in with Microsoft instead. In fact, I once facilitated a Windows NT training (yes, I did that too) and my students were blown away by the NT services control panel, which provides a handy list of background services and equally handy buttons to stop and start tasks. Compared to the whole inittab, init.d and rc.d setup of the Unix systems of that time, they thought it was much better. Until such time as they needed to add a new service of their own of course, but who wants to do that?

When dealing with chaos, you need to figure out why that chaos is there at all.

It can be that there is chaos because things are just extremely badly managed. US public transport comes to mind. There is really no reason that a public transport system needs to be chaotic; the Swiss have been showing us for decades how to run public transport well. The Dutch newspaper NRC recently published an article where they researched the reason for this. The answer: Good management, including enough investment. It is, quite literally, not rocket science. By the way, ample investment alone is not enough, because if an influx of oodles of money would be sufficient for high quality, US healthcare and US public education would be amazing, and as we all know, they are not.

Another cause of chaos is when there exists a fundamental lack of consensus on how to solve a set of extremely difficult problems for which there are no obvious solutions that we can all agree on. The chaos exists because there are different forces at work, all trying to move the system forward in different directions while balancing competing interests. Most social democracies fall in this category. Many people in the West are deploring the chaotic and inefficient nature of their countries and the way they are run. They fail to see that that is a feature, not a bug. Governments that represent the will of the people invariably suffer from multiple personality syndrome, because the people who voted them in are not unified in their desires and opinions. So they struggle forward while doing this and that and trying to harmonize the often conflicting or impossible policies they are asked to implement. Chaos is desirable here because the alternative is the singular mind of the dictatorship.

In tech, chaos is often the result of high velocity.

The libc5 people forked the GNU C Library because things were not progressing fast enough for them and they felt they couldn’t afford to wait. In the blockchain world entire new paradigms are being developed and new ideas follow each other up so quickly that there is literally no time to figure out which ones are good. So, things happen in parallel at different speeds. In the ensuing chaos, you get overlapping technologies, competing technologies, different solutions for the same problem, and imperfect implementations. There is literally no time to build a decent version of anything with enough documentation and support because if you do, you end up with a great implementation of yesterday’s solution that really nobody is interested in anymore.

The price of that velocity is chaos. When you are moving fast you are probably also breaking things; it takes time to get the details right and you don’t have that time. Neither do you have time to take care of secondary or tertiary matters like documentation and other types of support. This is especially true in the open source world, where even if there is time, nobody will want to write a 958 page manual about the sort utility unless they get paid for it by the hour.

Chaos favors people who don’t mind, uhh, chaos. These are people who don’t need simplicity because they know how to do things, even when the environment is suboptimal.

A few decades ago I was teaching a course on how to write Linux device drivers for a group of developers of a large electronics firm that decided that their new consumer device hardware platform was going to be based on Linux (moving away from VxWorks). On the first day of the course, I asked the students if they had toyed with Linux already.. “No,” they answered, “because our internal tools department hasn’t released the Linux development platform to us yet.” Apparently the fact that you could download Linux from the Internet had completely passed them by. In my book, that attitude makes you quite unsuitable for doing any Linux kernel development work. Working with Linux requires you to embrace the chaos: Things don’t work, lots of components are underdocumented, and obvious features are missing. And when that happens, there is nobody to call, not even the Ghostbusters.

Chaos is not a bug, chaos is a feature of all ecosystems where individual choice is (mostly) unconstrained. If you want simplicity and efficiency, you give up a lot of individual choice and you are opting for a slower pace of evolution.

Chaos is the price of freedom.