Incantations

The danger of using magic without understanding…

(Like this article? Read more Wednesday Wisdom!)

Note: This edition of Wednesday Wisdom comes with a soundtrack: The album Incantations, by retired English multi-instrumentalist Mike Oldfield. 

Many years ago I found myself in a situation where we couldn’t login to a local HP-UX machine through a terminal that was connected through that timeless classic: The RS-232 interface lead (with getty, remember getty?). I didn’t quite understand what was going on and I asked one of my associates if he knew what was happening. He looked at the error and said confidently: “Oh, I know what to do about that.” Next, he fired up vi, opened the /etc/utmp file and changed a 1 into a 0 (or something like that). And lo and behold, after he saved the file, the error went away. Surprised, I asked him what he had done and why that worked. “I don’t know”, he said, “I just know that whenever I get that error, I make that particular change in that particular file and then it works…”

Also many years ago, I sometimes asked SRE candidates who had professed expertise in Unix and Linux, how they would remove a file called “-f”. This question used to work quite well and often led to interesting discussions. However, after a few years of asking that question, I started seeing candidates who immediately answered “rm –- -f”. When asked why that worked, a shocking number of them didn’t know and couldn’t speculate about it, but they did know that this was the way to remove that file.

You can still ask that question, but you need to do it the right way. Read this article on interviewing.io that (among other things) covers the concept of “digging” and that uses the “remove a file called -f” question as an example.

One of my dearest friends works for an organization that has very strict space quotas on their file servers because, well, I don’t know, hard disks are expensive or something. My friend regularly gets emails threatening that they are going to remove all files that haven’t been accessed for six months, which used to prompt her to spend a day clicking on every file, opening it, thus resetting the date and time the file was last accessed, and thereby safeguarding it from deletion. Once I heard this, I gave my friend a few lines of Powershell and whenever it is that time of the month again, she types in these lines, which recursively walk the directory tree and reset every file’s LastAccessTime attribute. My friend has no idea what she is doing, but she knows it works. “It’s magic!”, she exclaims every time she uses it…

And indeed it is. These are all examples of incantations, a series of words used as a magic spell or charm. It might not be actual magic, but it is as good as magic for the people who use them. The beauty of incantations is that they work, regardless of whether you understand how they work. The danger of incantations is that without understanding, you do not know whether they will work in your specific situation.

From your incantation you might understand that “-f” is a file that rm, under “normal” circumstances, has some challenges removing. “*” is also a file that rm has some problems removing. If you know the incantation, but have no understanding, you might surmise that “rm -- *” is the way to fix that. So why does “rm \-f” not work to remove a file called “-f”, but “rm \*” does work to remove a file called “*”? Only knowing the incantation does not provide that answer…

This example is not entirely made up. I once assisted the system administrators of a Unix system at the Dutch Ministry of Foreign Affairs to recover some files from their backup tape. The files had been deleted because the system administrators had tried to remove a file called “*” using the rm command. The next time I came over for something else, there was a book with a yellow and black cover lying next to the console…

Remember Harry Potter? I love the Harry Potter books because they paint such a wonderful world full of magic and because of the timeless motifs about the struggle between good and evil. But what I don’t like about Harry Potter is that the only thing that Hogwarts seems to teach the children is incantations. There is never any understanding, never any theory of magic, never any investigation of why and how a spell actually works. You just need to learn them all and then practice until you get it right. In that world, who is ever inventing a new spell and how do they do it? Do they just throw together some stunted Latin with a swish-and-flick of the wand to see what it does? Or is there a level beyond the O.W.L.s and the N.E.W.T.s where they actually explain how any of it works? Voldemort and Dumbledore regularly perform powerful and exciting magic, but how did they come up with it? Clearly it is not taught at Hogwarts and there is no mention of a magical university or a Magical Institute of Technology or something.

If you, like me, are annoyed about that, but love the world of Harry Potter, I can wholeheartedly recommend Harry Potter and the Methods of Rationality, some stunning fan fiction about an alternate universe where Harry Potter grew up with a smart aunt and uncle and therefore he approaches the weirdness of the magical world with the analytical mind of a scientist.

The problem with incantations is that you don’t understand in what exact circumstances they work. Change the circumstances, and your incantations might work, might not work anymore, might do something else, or maybe worse, might do lots of damage. It is not safe to rely on incantations, you need to move to understanding.

Nobody escapes incantations though. Whenever you are ramping up on a new system or a new programming language, you really cannot get away from using incantations as a way to get something done before the understanding is there. Recently I was involved in an all-weekend work related project where, courtesy of being on the east coast (best coast!), I was very well placed to perform some critical maintenance. During prep I had written down the incantations to be used. Because these incantations were designed for this specific situation with the help of the actual wizards in charge, they worked, but I’d be a fool to use them in any other situation because there is no telling what might happen. It’s not bad to use incantations, what is bad is using incantations and leaving it at that.

One of my problems with many tutorials is that they never move beyond the incantations, or at least make it very easy to just stay with the incantations. I recently took the PyTorch tutorial and it is totally possible for me to breeze through that tutorial (and others like it), using the incantations provided, but never learn anything. 

For me the way to move beyond the incantations is to take the time to build up understanding and solve an actual problem. This involved the process of me working to understand the incantations I known and then turn them into other incantations that work for other situations. Sometimes, when I don’t have the time to do the research, I try to change the incantations I know in order to make them work for a changed circumstance. This typically involves changing flags, changing an argument, or setting an environment variable. Sometimes this works, and when it does, I gain a little bit of understanding. But quite often it does not work and I have to admit defeat and schedule time to do the work to build up an understanding of what is going on.

And understanding is what it is about. If you ever want to gain mastery on any topic, you have to move beyond incantations. I don’t care if you know dozens, or maybe even hundreds, of incantations; I want to work with the wizards who can create the spells based on an understanding of how the universe works.

Remember Harry Potter? He came across an incantation that he didn’t know and didn’t understand and then almost killed Draco Malfoy. All things considered that might not have been a bad outcome, but still, it’s dangerous to wave your wand and shout some things unless you really know what you are doing.