Did you know that Wednesday Wisdom is also a podcast! Find it on Apple Podcasts or on Spotify. Did you know that there is also a custom GPT called Midweek Muse that has access to all of Wednesday Wisdom? Did you know that all Wednesday Wisdom videos are also available on YouTube?

Completely unrelated: In 1982, when I knew approximately nothing and was writing code in BASIC and assembler, my math teacher introduced me to structured programming and a Danish programming language called Comal. In 1990, when I still knew approximately nothing, I had some free time on my hands and I started developing a Comal interpreter in C. Since then I have been tinkering with it on and off. It is nothing to be proud of, because the architecture and implementation (still) represent my 1988 understanding of how interpreters work, but fwiw, here it is: https://github.com/josvisser66/opencomal.

For the fact that ChatGPT has made writing good interview feedback easier than ever, I am still seeing more than my fair share of terrible feedback, so I thought I’d spend a few words on this lovely Wednesday morning to explain why great interview feedback still matters and how to write it.

As we all know, there is a lot of turmoil in the market, with layoffs and everything, but hiring is still a thing. Companies might not be hiring literally everyone they can find anymore, but new companies are still being founded, companies are still growing, and companies are busy retooling (meaning: Swapping mediocre expensive employees that they hired during the lean times with better and less expensive employees that are available right now). With an abundance of candidates on the market, the chances of hiring a good candidate are better than ever before. Unfortunately, your actual chances of hiring good candidates are only as good as your interview process. Also unfortunately, from first-hand and anecdotal second-hand experience, I have sadly to conclude that there is still room for considerable improvement, something I wrote about for the website interviewing.io.

As an indicator, the kind people over at interviewing.io told me that they tried to break into the market for interviewer training, but they had to let that idea slide because there was not enough interest.

Tech company hiring processes are broadly the same across the industry: The candidate is exposed to 5-7 interviews, dedicated to topics like coding, system design, a technical deep dive on a previous project, a behavioral interview, and perhaps a culture fit interview, where a manager or tenured employee tries to establish whether a candidate fits into what the company thinks their culture is. Each of these interviewers write feedback on the interview. This feedback then goes to a committee that makes the hire/no-hire decision. Sometimes these committees are made up of senior engineers and managers, sometimes these committees are made up of the interviewers plus the hiring manager, and sometimes a “bar raiser” is part of the committee (a practice I highly recommend).

No matter how the committee is established, the common factor is that all members of the committee have only the feedback to know what happened in all or most of the interviews. If the interviewers themselves happen to be in the committee, they only have first-hand experience of what happened during their own interview and so they are just as dependent on feedback as the non-interviewers in the committee.

All companies I know of require the interviewer to assign a score to the candidate’s interview performance. This score represents the interviewer’s overall decision with respect to how this candidate’s performance relates to the hiring bar.

Dad joke: Where do you take a recruiter on a first date? Answer: The hiring bar! Ask me how I know :-)

The scores are typically in the range 1-4, with some companies allowing decimal numbers, leading interviewers to fret about whether this candidate deserves a 2.2 or a 2.3. Whole numbers 1-4 typically align to “strong no”, “no”, “yes”, and “strong yes”. The hiring committee gets both the written feedback and the scores.

The relevant insight for hiring committee operation is this one: Hiring committees hire on feedback, not on scores! This might seem like a bold statement (and to be sure, it is typeset in bold), but it is easy to understand: If the hiring committee would hire on scores alone, we could replace the entire committee by a very small shell script. Instead, the hiring committee should read the feedback to form their own opinion on how the candidate performed during the interview and then come to a decision. This process also compensates for any misconceptions that the interviewer might have about what an acceptable answer is. These misconceptions are all too frequent, because most interviewers want to be nice, mistaking being a nice interviewer during the interview with being a strict (but fair) evaluator of the candidate’s performance after the interview is over.

Straying from the narrative for a bit: It is important to be nice. Once, long ago, I was chastised by a recruiter because a candidate had told them in the post-interview debrief that they had been afraid of me. Surprised, I turned to my friend and colleague Toni, who I had interviewed: “Toni, I interviewed you, am I not the nicest interviewer you have ever met?” Toni, with his typical candor (he is Finnish), replied: “You were the only interviewer I was afraid of.”

Candidate experience is important because if a candidate is good, you want them to join the company and that is much easier to achieve if they liked the interviewers. If they are not that great, you still want them to say (or at least think) that the interviewers were nice and fair, because there is literally nothing to be gained by them thinking that all the interviewers were assholes.

During the COVID pandemic, the lovely Mrs. Wednesday Wisdom was in a position to overhear my interviews when we all moved everything to Zoom. “Oh, that went well”, she once remarked after an interview had concluded. “No it didn’t”, I replied, “this candidate was terrible!” “But you were so nice to them”, she said, somewhat surprised.

Mission accomplished!

The lesson is this: Be nice to the candidate during the interview, but be strict in evaluating the candidate’s performance because, obviously, you want to hire only the best candidates, especially in a market where there are enough of them available. Remember this: A’s hire A’s, B’s hire C’s…

Back to feedback: In an earlier Wednesday Wisdom article I have argued that being able to write well is an important career skill. One of the places where you get to use (or at least, got to use) that skill is when writing interview feedback, because without well-written feedback, how are the committee members going to figure out what actually happened during the interview and whether the interviewers did a good (enough) job and were correct in their assessment?

Unfortunately, I still see a lot of bad feedback that fails to describe exactly what happened during the interview, what the interviewer thought, and why the interviewer thought that. Here is a (fictional) example (that nonetheless is very close to some of the worst feedback I see):

Mind you, this is not the conclusion, this is often the entire feedback!

Imagine that you are on the hiring committee for this candidate. What can you conclude from this feedback? Answer: Nothing. Either you believe the interviewer on their word that this candidate is worthy of hire or you don’t, but you have no evidence either way, other than perhaps knowing the interviewer personally and having a subjective opinion about them as an evaluator.

Good interview feedback reads like a short story of what happened during the interview, including meaningful things the candidate said or produced (code, diagrams), what the interviewer thought of that, and why they thought that, preferably with references to established quality guidelines, hopefully coming from the interview question scoring rubric.

If your interview is a coding interview, you should include code the candidate wrote in your feedback to back up your assessment that something that the candidate did was good or bad. If your interview is a system design interview, you want to outline the design decisions that the candidate made and whether they are good or bad and why. If you gave the candidate a hint, you want to describe what the hint was and how the candidate responded. If you asked the candidate a meaningful question, you want to include what that question was and how the candidate responded.

Obviously, the feedback does not have to be a play-by-play, as it is intended to be a summary; but it does need to have all the required evidence to back up your core claims. It does not have to be verbatim, but it does have to be persuasive. And if the hiring committee has any questions, you might want to make sure that you have all the information available in the form of notes and maybe a link to the coderpad that can be looked at if there ever is a need to dive into the details of what happened.

Which brings us to the topic of interview notes.

To be able to write great interview feedback you need to keep great notes during the interview. I am a big fan of recording interviews, but regardless of whether that is possible or not, I always keep sumptuous notes that includes the things the candidate said, the things I said, my running commentary of how things are going, code snippets the candidate produced, timestamps (approximately every five minutes), and all the other goings on that are important enough to influence my evaluation (like the connection being broken or the candidate having to look something up on Google).

This comes down to a lot of typing during the interview and I thank my parents for having sent me to a Scheidegger typing course back in secondary school. One thing about this is that it might distract the candidate during the interview. To allay this somewhat, I always tell candidates that they will hear a lot of typing in the background, but that this does not mean that I have lost interest and am Facebooking my wife, but that I keep notes to help me write good feedback.

After the interview, I go over the notes and add any additional insights or comments that I only thought of after the interview or that I didn’t have time to write down during the interview.

With that out of the way, the time has come to write the feedback. This used to take me about one-and-a-half hours because good writing takes time. This time was then earned back with interest during recruiter review and hiring committee meetings because processing well written feedback is much faster than processing bad feedback.

Fortunately, these days, we have ChatGPT, so instead of spending an hour or more crafting a well written story, I throw the interview notes (including all code the candidate wrote), the candidate’s resume, the job description, and the question rubric into ChatGPT Pro and tell it to write feedback that I can paste into our Applicant Tracking System (ATS). Five minutes later I have pretty good feedback, including ChatGPT’s score suggestion. I then spend another five minutes going over the feedback, suggesting improvements and maybe telling ChatGPT to modify the score (rarely). Another five minutes later, I have something I can copy and paste into the ATS. All of this together takes fifteen minutes max.

With ChatGPT, there is really no excuse anymore for bad interview feedback because the model is more than capable when it comes to writing good feedback provided that you kept great notes during the interview.

Back to the interview notes for a second: Whenever you write, you always write for an audience and when you are going to create interview feedback using ChatGPT, the model is the audience for your notes. That means that you need to write and structure your notes so that the model can make sense of it, for instance by clearly separating the things you said and the things the candidate said, or separating things that were said and things that you thought. Simple markers like “TC:” and “Me:” help tremendously with that.

Good writers (like myself) will tell you that they can write better than ChatGPT, which is undoubtedly true for them given enough time. But ChatGPT can write very good feedback in a matter of minutes and, obviously, quality is fitness for use. Plus that not everyone is a good writer. As a friend of mine (an Oxford grad) puts it: “I love ChatGPT, it takes my colleagues’ bad English and converts it to good English.”

Great interview feedback remains as important as ever and with AI there are no more excuses for writing bad interview feedback.

Did you know that Wednesday Wisdom is also a podcast! Find it on Apple Podcasts or on Spotify. Did you know that there is also a custom GPT called Midweek Muse that has access to all of Wednesday Wisdom? Did you know that all Wednesday Wisdom videos are also available on YouTube?

Completely unrelated, over in our sister publication Thursday Thoughts, there is a new article about how to customize your own ChatGPT to act as a (mock) Designated Pilot Examiner during the oral part of the pilot checkride.

Two weeks ago, I discussed how the history of software engineering desperately sought to make programming easier, culminating in the latest and greatest invention in the field: AI coding tools. Unfortunately, as was discussed last week, coding with AI tools is still hard. So what does all of this mean for software engineering as a profession? Will there still be software engineers or will people build their own $100k SaaS without knowing how to code in five easy steps?

Software engineering is a tight loop that encompasses thinking, researching, writing documents, reviewing documents, meetings, coding, reviewing code, testing, debugging, and manual operations. 99% of the things I do as a software engineer fall into one of these buckets. AI tools are hugely helpful during most of these activities, significantly cutting the time required. Really, the only things it does not greatly help with are thinking and writing good and original documents (the only ones I write, of course). In other words, AI tools make me more efficient and effective by shrinking the time required for most parts of my job. The question now is: What are we going to do with the time saved?

One potential answer is: “Nothing”; if three people can do now what we needed five people for until recently, we are just going to lay off two people and keep output constant. This is surely what many people are thinking is going to happen and some CEOs are adding to this by pointing to AI as a main driver for recent layoffs.

Another potential answer is: “We are just going to write more software!”

Every organization I know has a stack rank of things that they would like to do and a “red line” that marks the most important thing that they are not doing. If AI tools make the existing software engineers more productive, the red line could move and the things that we just did not have the people for might now suddenly become possible. For instance, I know for a fact that the government is sitting on a pile of legacy software that they would love to rewrite in Java and maybe now they can…

So which of the two is it going to be: The same amount of software with fewer people or more software with the same amount of people?

The answer, of course, is: But not at the same time and not at the same place.

“Wait, what now?”, I can hear you thinking, “how is that even possible?”. And what will the impact of that be on the price of software engineering (in other words: On software engineers’ salaries)? To answer that question we first need to look at some other trends in the market because, apart from the current US administration, nothing lives in a vacuum.

One of the things that often amazes me, is how many great software engineers there are these days. I regularly come across extremely well written articles or great pieces of software, authored by people I have literally never heard of before. When I started studying computer science back in 1984 there were not very many of us and when I joined the Dutch Unix User Group (NLUUG), I could say with some confidence that I knew almost everyone in the country who ran Unix computers. Since then, millions of people have taken up computer science and, brains being distributed the way they are, quite a lot of them are either brilliant or at least very, very smart. Apparently, there are now about 47 million software engineers in the world, which represents approximately a doubling since 2018. Consider that for a second, in the last eight years, the number of software engineers has doubled. What have all these people been doing?

Throughout the history of software engineering, we have always had a shortage of software engineers and shortages typically means rising prices (hello war on Iran). To the joy of all people in the field, software engineering (until very recently) offered the highest overall earning potential and career ceiling, out-earning traditional engineering fields like civil or mechanical engineering. In fact, looking at the job market as a whole, only top medical specialists, finance executives, and top legal and business professionals earn more than the highest paid software engineers.

High prices for a good or service typically means that you consume less of it, though how much less is influenced by a concept called “price elasticity”. All major business endeavours these days need to be supported by software, so every company and every startup that is trying to carve out a unique market has had to hire software engineers. Given the shortage of qualified software engineers, prices have been going up and companies had to bid against each other to attract talent.

Short history lesson: In the Netherlands, a lot of sectors of the economy are ruled by collective bargaining agreements that are negotiated between the unions of employers and employees. If the government thinks that a recently negotiated agreement is a good one (they usually do), the government minister for labor will declare these agreements binding upon the whole sector; even for employers and employees who are not members of the unions that negotiated the agreement. One of the things contained in most agreements is pay scales, which are typically drawn up so that people with approximately the same education and experience, earn about the same.

My first job was at a bank and my labor contract was governed by one of these collective bargaining agreements. Banks had problems hiring IT talent because they couldn’t pay software engineers more than accountants with a similar level of degree and experience. To address this, the bank employers had negotiated a provision into the agreement that they could pay a “labor market surcharge” for certain specialties because otherwise they would never be able to hire a software engineer ever again.

For software and Internet companies, the availability of software engineers and related IT talent was so crucial that a real “battle for talent” ensued. When I got hired by Google in 2006, it was made clear to me that my most crucial contribution to the company was hiring other people. Consequently, we hired everyone we could find (provided that they jumped over the hiring bar), wherever we could find them. I was working for Google in Zürich, which is lovely and amazing, but it was made clear to me that the only reason that that office existed was to function as a hiring bridgehead in Europe, hiring talent that could not or would not move to the US.

As a manager, I regularly got spreadsheets of “H-1b losers”, people who we hired and who wanted to move to the US, but who didn’t get an H-1b visa (because the number is limited and there are lotteries involved). We would offer these people a position in Zürich instead and tell them they could move to the US after a year when they would qualify for an L-1 visa (intra-company transfer).

Some of those did, but some came to their senses and stayed in Zürich.

We would hire people without a headcount plan or without a vision of what these people were going to work on. When they accepted the offer, their resumes and interview packets would go to an allocation meeting where managers competed to select people for their teams. At Facebook, new people started in a “bootcamp” to learn about Facebook’s internal technology and development environment. Part of that bootcamp was “team selection”, where managers would present their teams and try to woo the scarce talent to their teams. News abounded about big tech’s high salaries and it is not strange that many young people decided to study computer science, leading to the aforementioned doubling of the number of software engineers from 2018 to 2026.

Obviously, these good times had to stop rolling at some point. A lot of tech companies overhired enormously and branched out in endeavors that did not make a lot of business sense. Non-big tech companies had to pay inflated salaries to hire “the best of the rest”. Lots of companies couldn’t hire at all and had to make do. A reckoning had to come and guess what, it did! Starting in 2022 and picking up steam in 2023, tech companies started laying off people by the thousands, flooding markets with capable software engineers who had mostly become accustomed to cushy jobs with great compensation. In my “home market” of Zürich, the layoffs hit really hard because, Switzerland’s awesomeness notwithstanding, in terms of good tech employment, Google was pretty much the only show in town and tenure at Google Zürich had been stellar because of the simple fact that there was nowhere else to go.

That said, I am happy to report that since then, and probably fueled by the availability of tech talent, Zürich now has a small but thriving and (as far as I can see high quality) startup scene.

Economics prescribe that if supply grows relative to demand, prices will drop and this is exactly what has been happening. Some of that has been overt (just offering lower salaries) and some of that has been covert (in the form of downleveling in the offer). This price effect has not been so drastic that polite society crumbled, but the downward trend is easy to spot. People moved, people moved on, people retired early, and people took jobs with a lower salary.

It is in this market that AI tools made their entry…

Suddenly, we have a technology available that makes software engineers more effective, at a time there seem to be more than enough software engineers on the market already. This makes many people rightfully very nervous, because quite a lot of people who have been in the market since about 2018 have racked up massive student debt that is not paid off yet, because they haven’t been riding the gravy train long enough. Life is even worse for the young people studying computer science right now because they are graduating into a market that is going through a huge reset, where there is a lot of competition and a brand new toolset that will have an as-yet-unknown impact (though we can probably say with some confidence that it won’t be positive for the number of software engineers required).

So, looking at all of that, what is the future of software engineering?

First of all, AI coding tools will not make software engineers unnecessary. As I argued last week, coding with AI tools is still hard and so we will continue to need people who are experts at solving problems with computers. But, and here is the crux, most of these people will probably be of the more “senior” variety who have a good understanding of how to build complicated working software with AI coding tools. There will probably be a lower demand for more junior people because a lot of what they used to do can now be done efficiently by senior engineers with the help of AI coding tools. In fact, my recruiting friends tell me that this is already happening; the job market for senior people with AI expertise (to the extent that there is some to go around already) is quite healthy. Not even the AI frontier labs believe that AI coding tools will completely replace software engineers, as evidenced by the fact that both Anthropic and OpenAI (my own employer) are still hiring software engineers. But, we are not hiring new grads or people with limited industry experience, which might give you an idea of where we think the industry is going.

This of course poses the question where new senior engineers will grow if fewer companies are hiring junior engineers. Great question, to which I don’t have an answer yet. For now I assume that everyone assumes that is someone else’s problem 🙂

By the way, it is not that these junior engineers will do nothing. A lot of them already have jobs where they will get experience and eventually become more senior. One thing that the AI coding tool boom might fix is level inflation where new grads got hired as “senior engineers” and people with five years of experience “demanded” promotions to staff level. Compare this to the construction world where you typically need 10-15 years of experience to become “senior”.

What helps the software engineers is that AI itself represents an “expansion of the disc” (see last week’s article for an explanation of that concept), meaning that AI will in itself cause more software to be written because really, all these artificially intelligent customer service representatives are not going to write themselves!

All of these things are happening at the same time: Some companies are still letting people go, some companies are still hiring (but potentially hiring fewer and different people), some companies can suddenly afford to hire people because there is ample supply, some people will leave the industry and the bottom, where the people reside who only had a job because we were in a situation where everybody with a pulse could one, will fall out and they will have to find something else to do.

I know this sounds terrible and harsh, but it is unfortunately the reality.

One thing we can learn from Wall Street is that, regardless of whether the market goes up or down, there is always money to be made 🙂. These are times of great change and it might easily be quite a few years before the situation stabilizes again. For the people who worry a lot I have got this to say: This is the fourth crisis of sorts that I live through and every time people told me that now things were going to be different. They were right, things always turned out to become different (hint: they always do), but over time I have always been able to ride whatever wave there was to be ridden. Not because I am brilliant, but quite simply because I have continuously invested in my knowledge and skills. If you are a software engineer today and you are not playing around with running your own models and writing your own agents, you are doing yourself an enormous disservice.

What I think is really different this time is that the times where there was a great shortage of software engineers might be behind us. Software engineering is difficult and so I think there is always decent money to be made, but the times where a new grad could get $200k per year and a nice equity package might be behind us. Top engineering students will probably still do very well, just like top law students and top medical students (post-residency), but a computer science degree from <insert your favorite party school here> might not cut it anymore.

In itself I do not find this problematic. The great shortage of software engineers from 2010-2020 and related price increases were historically an anomaly. When I was hired by Google in Zürich in 2006, with 18 years of experience already, I was offered $175k (in 2026 money, give or take, inflation and USD/CHF exchange rates being what they are) plus a modest equity package. Levels.fyi tells me that that is what a new grad SWE gets these days in Zürich. Nothing wrong with that, but it does show a significant price increase and I think we are going to see these prices will stabilize (at best) or maybe even drop a bit.

Then finally I have to say something about the overexcited news reports that non-coders will now use AI coding tools to develop their own applications? Won’t that cut into the need for software engineers? Will experienced software engineers not simply be replaced by business majors with Codex?

Answer: No.

It is indeed true that with today’s tools, you can create quite decent simple applications without knowing a lot about coding. As an example, I “coded” “Chordcraft” (an application for showing how to make certain chords on the piano) using Codex and without any technical intervention. But really, I think that this is about as far as it will go, because anything more complicated needs an understanding of how computers work. It’s a bit like arguing that the availability of “Technic Lego” will decrease the need for mechanical engineers building trucks and planes. I think that most applications that non-engineers (will) create are ones that people would have simply done without if AI coding tools weren’t there. If it will have a measurable impact at all, it will be small and at the bottom of the employment pyramid (a place where I think not many Wednesday Wisdom readers have their professional abode).

That’s it for this herniated three-episode exposé on AI and software engineering. I promise that next week I will return to lighter subjects, such as that we are all mushrooms: Kept in the dark and fed a lot of bullshit.

Did you know that Wednesday Wisdom is also a podcast! Find it on Apple Podcasts or on Spotify. Did you know that there is also a custom GPT called Midweek Muse that has access to all of Wednesday Wisdom? Did you know that all Wednesday Wisdom videos are also available on YouTube?

In other news: Today is the young Ms. Wednesday Wisdom’s birthday! Happy birthday Merel! Then finally, the title of this week’s article was graciously donated by my friend Sonya P. Check out her Youtube channel!

Last week, I rambled a bit about the history of software engineering and concluded that, since the dawn of time, there has been a desperate search for something, anything, that would allow normies to develop software, all in the hope that this would solve the crisis of software development. So far, this effort has failed. What has happened instead is that other improvements in the field have made software engineers more productive.

However, despite software engineers being more productive than ever before and there being more software engineers than ever before, software development is still expensive. Why?

Simply put: Because we also need more software than ever before and because, despite all improvements, software development is still hard!

The field of information technology is a disc growing outward from an origin that is very small and where the big bang happened 60-odd years ago. My first job, when the disc was still small, was at a bank, where we used three mainframes for core account processing, most of which ran as nightly batch jobs.

Q: Why nightly?
A: Because people would create transactions during the day 🙂. Then, in the evening, these transactions were “transmitted” to our data center through Dieselnet: A fleet of cars driving in the late afternoon to all the bank’s branches and collecting transactions on punch cards, floppy disks, and what have you. At the datacenter, we would read these transactions into the mainframe and start the batch jobs. At the end of the batch runs, we would print statements, which Dieselnet would then distribute throughout the country. Customers would bind these printed statements into little booklets as evidence of account movements and balances. If this sounds cumbersome, welcome to the world before the Internet 🙂

Over time, and through many expansions of the disc, we now live in a world where every consumer has a portable computer in their pocket that is more powerful than the mainframes we used in my first job. These computers are also continuously connected to all other computers on the planet, so no more need for Dieselnet and printed statements, because banking software on your pocket computer allows you to enter transactions, look at your accounts, and check your balances whenever and wherever you want.

In fact, as far as I know, if you do want a printed statement these days, you are either straight out of luck or you need to pay dearly for the privilege.

The expanding disc of information technology represents more computers coming into the hands of more people, as well as new ways for these computers to communicate and present information. All of this requires software and because the surface area of the disc grows quadratically, so does the required amount of new software that has to be created and maintained. Over my career, the disc expanded to include minicomputers, personal computers, the Internet, and finally mobile devices, wearables, and the Internet of Things.

To write and maintain this new software, we always needed more software engineers who, despite all the improvements, were not able to keep up with the growing demand. Hence, the software crisis perpetuated…

The other reason that software development is still expensive is that, despite the improvements of the past decades, software development is still hard.

A few years ago, I wrote an article called “Why programming is hard”, which sought to explain this fact by outlining both the cognitive and mechanical aspects of software development. The cognitive aspects deal with the fact that it is almost impossible to know upfront what to build and how best to build it. The mechanical aspects describe the inherent complexity of programming languages and libraries. Together, these explain why building a piece of working code is difficult and so it requires deep expertise. And in general, deep expertise is expensive.

By the way: One thing that I am leaving out of this story is that in parallel, the bar for what is considered usable software has gone up too.

Enter AI coding tools.

As we all probably know by now, coding with AI tools is amazing. Speaking for myself, in the year 2026 so far, I have not written five lines of code by hand.

Which, given the fact that it is almost June already, is perhaps kinda worrisome. What if I forget how to do it? But, that is probably a matter best left for another Wednesday Wisdom article.

I think that courtesy of my employment at an AI frontier lab, I am at the forefront of developments, but most of my peers in tech companies tell me that there is a huge pressure on them to use AI coding tools. Apparently, some companies are even deploying systems of metrics to figure out if a software engineer in their employ is secretly typing in code by hand, thereby stealing from the shareholders (or something like that, I really don’t quite understand what the thinking is here).

Everyone and their dog are expecting miracles from AI coding tools and they are right to do so, but I think they might be focusing on the wrong miracles, because AI coding tools will increase software development’s velocity, but it will not make software development as a whole any easier. Even with AI coding tools, coding is still hard!

To understand this, we first need to realize that software development is not a manufacturing process. The essence of a manufacturing process is that there are lots of brains needed in the beginning, but then the production of a complicated piece of equipment is simplified so that it can be executed by a combination of robots and people who are less knowledgeable and skilled than the original designers. Software development is not like that.

Each piece of software is a unique bespoke one-of-a-kind thing that is custom built for the exact problem it is solving. Of course we have libraries and reusable pieces of infrastructure, and fortunately we do because otherwise we’d stand no chance at all, but the actual application that is built on top of all that is a one-off. This means that for every piece of software we need to start more or less from scratch when it comes to the design and implementation. It is as if airlines would have every plane in their fleet custom designed and built for the passengers, route, and airports of intended use, including the engines and the avionics. It probably would make flying much more comfortable, but the costs would be astronomical and it would definitely be less safe.

The big problem in software development is knowing exactly what needs to be built (requirements) and how it needs to function (architecture/design). Once you have a crystal clear picture of all these details in your head, typing in the statements is (relatively speaking) easy. For various reasons, typing in the statements is very time consuming because lots of statements need to be typed in and you make many mistakes while doing it, but overall it is not difficult.

However, because of its cryptic nature and because (as we will see later) when we are typing the statements we are still designing the software in our head, non-engineers usually think that the coding part of the project is the problematic bit. Software engineers know better. If the coding part is actually hard and cumbersome, that typically indicates that you don’t quite know yet what problem you are solving or how to solve it.

Typing in the statements is exactly where AI coding tools shine. When you give an AI coding tool an instruction, it is never in doubt about what it needs to produce and it typically figures out very quickly which statements it needs. It then types in these statements with blazing speed. In other words, AI coding tools function best at the bit that was really never the problem: Writing the code once you know what needs to be done!

This is not at all an attempt to trivialize the genius of AI coding tools. I still marvel at the speed with which AI coding tools generate complicated pieces of code that stitch together algorithms, data structures, libraries, and APIs for complex services. But they are not intrinsically better at that than I am, just faster.

But here is something that AI coding tools cannot do yet: Figure out what needs to be built and which functional and non-functional requirements are the important ones to focus on.

Here is an example: For the last six months, I have been working on a solution for rolling out OS images to OpenAI’s fleet of GPU machines. This is a non-trival problem because of the size, diversity, and complexity of the fleet, as well as the fact that we would like to do that while continuing to serve ChatGPT and other products. Obviously, the cloud providers we use have different primitives for dealing with virtual machines, as well as different data models for grouping virtual machines and templatizing configurations, which makes it hard-ish to create a generic solution that works well across all our datacenter partners.

On top of that, NVIDIA’s GPU families have peculiarities too. For instance all machines in GB200 and GB300 (a.k.a. Blackwell) domains have to run the same release of the NVIDIA drivers, making upgrades harder. On top of that, there are velocity requirements for the rollout because our vulnerability management program prescribes that known vulnerabilities of a particular severity need to be addressed within a certain number of days.

With a problem of this shape, it is almost impossible (and in most cases undesirable) to figure out all of the requirements up front, make a detailed design, and then (set Codex to work to) type in the statements. So instead, as is the standard today, we are developing this software iteratively. Our first solution ignored the Blackwell requirements and only targeted machines in Azure. We had Codex develop this version based on what we knew, ran whatever it came up with, and then learnt about various peculiarities of reality. So we iterated: We ran the latest version Codex gave us, observed what was going on, learnt from it, and then put Codex to work with enhanced instructions, telling it what we really wanted and how we really wanted things to work with whatever we knew and understood now. Once Codex was ready, we repeated this cycle.

In this project, Codex was hugely helpful during all phases of software development. During the design, it helped us by telling us how Azure scale sets actually work, during the coding phases it typed in all the statements, and during the debugging phases it analyzed logs and reasoned about what was probably happening, fixing bugs and adapting to our changing requirements. Software development with AI coding tools comes down to being in a continuous conversation with the tool as you hone in on what really needs to be built and experiment with various solution avenues.

Q: Okay, I get it, but why is this still hard?
A: Because AI tools can write the code, but they do not own the consequences.

AI can type in the statements, but you have to decide what is important and guide the tool appropriately. If I tell Codex: “He, write me a tool that reimages all the nodes in this GPU cluster with a new operating system”, there is a high probability that it writes me a tool that does so for all machines at the same time, which is clearly not what I want. Codex does not know about our requirements regarding canarying, incremental rollouts, failure detection and whatnot, so I am the one who needs to tell it how things need to be implemented, after which I am the one who needs to check that of all 57 varieties to do that, it picked one that is appropriate for our environment. That requires knowledge, experience, and judgment. None of this is embodied in the model or, to the extent that it is, it is not necessarily activated without appropriate instructions.

Most real systems have context that is not fully visible, like undocumented invariants, weird legacy assumptions, deployment constraints, half-migrated APIs, old bugs people now rely on, and performance requirements. Most of these are not written down and thus beyond the reach of the model. AI coding tools are very good at code generation within a local context, but much weaker at understanding the full system. These bits of overall context are often ambiguous, badly spelled out, not necessarily widely agreed upon, or maybe only half-realized. This is the hard bit of software engineering and this is what the AI coding tools cannot do yet, so this is where you come in.

On top of that, AI tools are bad at “taste”. They don’t necessarily know when to abstract, when not to abstract, when to keep something boring, when a solution is too clever by half, or when a comment is needed to any human reviewer (assuming for a moment that human code review is still necessary). AI provides a solution, but human engineers still need to decide whether it is a good solution, given everything they know about the entire context.

Of course, models and tools are getting better all the time and so it is an open question whether the AI coding tools of tomorrow will be much better at this and then really make the entire software development process easier. But I don’t think so and here’s why: The things that make software development hard are exactly the differentiators between junior and senior engineers.

For the last two decades or so I have been saying that what separates me from a junior engineer is not that I can code better. Quite the opposite often: An L3 or L4 straight out of college is probably better at algorithms than I am, seeing as it has been quite a few years already that I took an algorithms class. When I interviewed at Google, I was asked about the runtime of quicksort which, at that time, I hadn’t thought about for 20 years or so, even though I had sorted many things using the handy library functions that come with most programming languages. What separates a senior engineer from a junior engineer are exactly the things I have been talking about in the latter half of this article: Contextual knowledge, understanding the requirements at a deeper level, taste, and experience. These are the hard bits and these are the bits that the AI doesn’t do.

All of this made me wonder what this is going to do for the employment opportunities of junior engineers. If it made you wonder too, you are in luck, because that will be the topic of next week’s Wednesday Wisdom…. 🙂

And that episode can be found here.

Did you know that Wednesday Wisdom is also a podcast! Find it on Apple Podcasts or on Spotify. Did you know that there is also a custom GPT called Midweek Muse that has access to all of Wednesday Wisdom? Did you know that all Wednesday Wisdom videos are also available on YouTube?

Note: This week’s article started as a larger and rambling article touching upon a variety of topics. After numerous starts and stops, followed by some more thinking, it dawned upon me that I was trying to write down a holistic vision on what AI coding tools will do to the software engineering profession. Over the next two to three weeks, I hope to be unfolding that vision in successive articles. This week we start with the ground work: How we always wanted to make software development accessible for non-engineers, but instead made actual software developers more productive.

A friend of mine is trying to make her way as an independent software engineer. She recently came in touch with a potential customer through an intermediary who wanted a bunch of features for automating various processes in the customer’s backoffice. The intermediary thought $5,000 was a good price for the work they proposed because, surely, with AI coding tools, developing all this software would be faster than ever before. Both my friend’s chief advisor (an experienced and retired software engineer) and myself told them that we have seen this happen with shocking regularity since the dawn of time. Therefore, I thought it would be instructive to explain to younger software engineers what happened since the Earth cooled and the dinosaurs came and then went away again in order to provide us with cheap oil.

Ever since computers were invented, everyone but the software engineers has been annoyed by the slow pace and high price of software development. This became such a constant annoyance that it even has its own Wikipedia page, which describes the so-called “software crisis” as “a term used in the early days of computing science for the difficulty of writing useful and efficient computer programs in the required time.” Consequently, everyone (but now including the software engineers) has been looking for ways to make software development faster and cheaper, typically by providing better tools and by inventing ways to require fewer expensive people.

As early as 1959, COBOL was designed as a programming language with a syntax close to English, so that non-programmers could read and maybe even write computer programs. Obviously, this effort failed; not in the sense that COBOL did not become a very popular language for business applications (it did), but because English majors with a business degree still cannot write software.

Interestingly enough, one of the statements I frequently read about AI coding tools is that the programming language of the future will be English. But, I am running ahead of myself.

Since then, we have seen a variety of other “solutions” to the software crisis. In the 1980s, the hip new thing was “fourth generation programming languages”, which were coined as a concept in a book aptly called “Application Development Without Programmers”. The preface of this book contained the following sentence: “The number of programmers available per computer is shrinking so fast that most computers in the future must be put to work at least in part without programmers.”

Well, at least they got that right, because I got a dozen computers in my house and, the lovely Mrs. Wednesday Wisdom’s prowess with SQL and SAS notwithstanding, there is really only one competent programmer around to keep all these computers going.

But let that statement sink in for a bit. Before the advent of the personal computer, there were apparently more programmers than computers and people were worried about who was going to program all these computers once the ratio dropped below 1:1, which started happening in the mid-late 1980s.

I can personally vouch for the “more programmers than computers” phase of our industry. In my first job, I was part of a team of about 25 people who cared for four computers in the company’s datacenters.”

In their day, fourth generation programming languages were such a hype that some people advised me against studying computer science, because they expected that business analysts and other non-engineers would be writing their own software any moment now.

Interestingly enough, one of the statements I frequently read about AI coding tools is that non-engineers will soon be writing their own code. Where did I hear that before? But, I am running ahead of myself again.

The 1990s saw the rise of no-code and low-code programming environments. This trend was started by Visual Basic 3.0, which did indeed give us a whole new paradigm for writing GUI applications by dragging and dropping UI widgets to a canvas, connecting them through events, and either configuring objects or sometimes writing small amounts of relatively simple code to respond to these events. Now non-engineers were surely going to develop software because now that didn’t need typing in long and complicated programs anymore!

Time for an anecdote: I once taught a Visual Basic (VB) 3.0 programming course to the IT people of some company in the Netherlands. They weren’t developers per se, but anyway. The students thought it would be a cool idea to forgo the standard exercises of the course and instead work on an application that they had been asked to build for internal use. I thought this was a splendid idea.

The application was for the administration of incoming mail and controlling its distribution to the right recipients. Sometimes, the mail had to be processed by multiple departments and in that case they stuck a little preprinted note on the mailpiece with the names of each department that needed to see it, after which it would start a journey through the office, with each department signing off that they had seen it.

VB 3.0 offered features to connect to databases through ODBC and I taught the students how to use this feature for data persistence. For the traveling mailman problem, the students proposed adding five columns to the “MAIL” table for the names of the departments that needed to see a piece of mail. The columns were aptly called “DEPARTMENT1” to “DEPARTMENT5”. I asked what would happen if a piece of mail should be seen by six departments? They agreed that this would be a problem, but that this happened very infrequently. Always the optimistic teacher, I made a counterproposal: Why not give each piece of mail a unique identifier and then have a separate “DEPARTMENTS” table with the name of the department and that unique identifier? Then at runtime, we could use something called a JOIN to get all the information we needed.

The students were flabbergasted. This was brilliant! Wow! Inspired by this feedback, I took it one step further and proposed that maybe the “DEPARTMENTS” table should only have a unique department identifier and the name of a department, and then a third table would keep track of which departments needed to see which pieces of mail through rows that contained only a mailpiece identifier and a department identifier. By now the students were convinced that their trainer was an absolute genius.

Apart from being a fun anecdote, this story also illustrates a fact that many people either forgot or never realized in the first place: An awful lot of software in the world is already written by people without a relevant computer science degree. I assume that a lot of Wednesday Wisdom readers are professional software engineers who are working in Big Tech or in Big Tech adjacent organizations, but really, there is a huge amount of software development going on by people who have only a limited clue about what they are doing and they need all the help they can get.

No and low code environments stayed with us for a while, but I think it is fair to say that they never lived up to their expectations. I sat through quite a few demos where the speaker would throw a calculator application together using standard widgets and a few lines of code, to which my response invariably was: “But I don’t want to write a calculator app, I have five already.” In response I was sometimes asked what I did want to write, to which I usually answered: “An application for creating and managing invoices for law firms” (something I wrote about here). This relatively simple business application idea always defeated the features of the no/low code environment being demonstrated.

In the meantime, lots of things had happened that made me more productive. The list is long and detailed, but it includes better programming languages, better compilers, free compilers , better editors, better libraries, more libraries, free libraries, and better debugging tools (hello Valgrind!). Most of these improvements can be credited to the open source movement, which from very humble beginnings managed to get a complete and free professional software development suite, from the operating system upwards, in the hands of everyone who wanted to develop some code.

It might be hard to imagine now, but when I started out as a developer, getting started was an expensive and/or difficult proposition, because you had to buy or steal all of your software and probably photocopy the manuals. In my early days, I wrote applications on my PC using a stolen Norton Editor and using a stolen Turbo Pascal and an equally illicitly obtained Turbo C compiler.

Aside: Let’s hope the statute of limitations has run out on these crimes.

When I started getting a bit more affluent, I went legit and would shell out about $2,000 (in today’s money) for a Microsoft Visual C++ compiler.

The open source movement changed all of that and through their combined efforts they allowed many people to become incredibly productive software engineers by providing all the development tools one needed and amazing libraries for pretty much everything one wanted. Again, maybe hard to imagine, but when I started coding professionally, none of this was available, which made developing software cumbersome and slow. For one program I was writing I remember I wanted to process glob-like filename patterns. No library was available, so I had to code that, by hand, in assembler. That is challenging and therefore fun and rewarding, but as you can imagine, it does not do wonders for velocity. Even before AI coding tools became available, I would routinely write something in a day using open source languages and tools, that would have taken me weeks, if not more, a few decades ago.

All this goes to show that since the dawn of time there has been a desperate search for something, anything, that would bring software development into the realm of “normal people”. So far that has failed, but what has happened is that other improvements in the field have made the new and existing software developers more productive.

Enter AI coding tools.

Again, we are seeing outsized claims that software engineering is doomed as a profession and that, really, this time non-engineers will be able to develop their own software. I beg to differ, and here is why: The AI coding agent is helping me with the part of my job that was never a problem to begin with!

Let me say that again for clarity: AI coding agents are doing things that are extremely helpful and that are making me much more productive, but they are not doing anything that I couldn’t do myself.

In the whole process of developing a piece of software, writing the code was always time consuming, but it was never problematic. It just took a long time. Same for debugging, writing documentation, refactoring the code, and all the other things that I have Codex do these days. The AI agent is doing stuff that I already know how to do, but it does it faster and in parallel, which is extremely helpful. It’s a tool with incredible power, because I can now do more things in my day, deliver more working code faster, but it is not doing anything that I wasn’t able to do before, much like all the previous improvements in software engineering.

That doesn’t mean that there are not going to be impactful changes to the profession, but my current bet is that there will stay a profession that understands how computers work and that specializes in solving problems with computers.

One thing is certain though: These professionals will spend considerably less time typing in the code...

Stay tuned for next week’s article which might investigate what a software engineering profession that does not type in code could look like. Then, the week after that I might talk about the economic consequences of it all.

Did you know that Wednesday Wisdom is also a podcast! Find it on Apple Podcasts or on Spotify. Did you know that there is also a custom GPT called Midweek Muse that has access to all of Wednesday Wisdom? Did you know that all Wednesday Wisdom videos are also available on YouTube?

This week we are covering a reader question. F. contacted me and asked: “Hi Jos! Talking about anxiety, do you have any advice on how to manage expectations and deadlines with managers? I work as a software engineer in Big Tech and recently my manager is stressing me a lot about giving accurate ETAs and being able to meet them.”

At first, I thought this was maybe a question about how to plan accurately, but after reading the question again, it became obvious that it was in fact a question on how to manage upwards for software developers. With that cleared up, let’s dive into it…

Right off the bat, we need to realize that software development is incredibly hard and expensive. By and large, you should not do it unless absolutely necessary.

Reading tip: No, you don’t want an app, which I wrote 3-4 years ago when yet another hopeful person asked me to help them develop software for their company and I had to explain (again) why this was much harder and much more expensive than they thought it would be.

You and F. might think you are in the business of developing software, but you are not. You are in the business of solving certain types of problems and writing software is how we do that. And because writing software is expensive and everyone needs to keep their costs down, it stands to reason that you should develop as little software as possible.

I assume that both F. and their manager know that no software project in the history of the entire universe was ever executed according to plan. Lots of reasons for that, the number one being that even though the production of software might look like a manufacturing problem, it is in fact not. Instead, it is a weird combination of research, design, discovery, making a piece of art, and building a working prototype.

All the aforementioned things are hard to do according to a plan because of the unknown unknowns, both in terms of what we want to build and how we should build it. To a large extent, this is why Agile development was created. People who insist on an accurate delivery date for a piece of yet-to-be-written software are either completely in denial about the basic nature of software development or they are willing to make significant compromises on requirements.

In every software development project there are three major dimensions that are constrained: Time, money (typically meaning: manpower), and requirements. At least one of these needs to be variable. Demanding “an accurate ETA” means that time is fixed and since Fred Brooks already taught us that adding manpower to an already late software project makes it later, the requirements are really the only candidate for the variable dimension. It is easy to plan a software project accurately if you are deciding up-front you are going to be happy with whatever can be produced by the delivery date.

If F.’s manager is in denial about the fundamental nature of software development, there is really only one thing to do, which is to find another team on the double. If a manager requires a fixed delivery date but is not willing to compromise on the requirements, there is no way that you are going to meet their expectations because, no matter how brilliant you are, you cannot change the physics of software development.

While writing this, it dawned upon me that there is of course a way to make time, money, and requirements fixed, which is if you manage to set the accurate ETA so far in the future that it is entirely conceivable that you will manage to deliver the requirements by that time. If you get a year to put a “Hello, world” program into production you will probably not fail. This approach is, of course, entirely fictional.

If F.’s manager is not totally incompetent, they will know about the fundamental nature of software planning and then the question becomes what they actually mean when they say that they want an “accurate ETA”. This question of meaning is much harder than it seems, because people are terrible at expressing themselves correctly, as witnessed by everyone rolling their eyes when the local nerd patrol answers “Yes” to the question: “Are you not hungry?”

So what does insisting on an “accurate ETA” mean? One possibility could be that F.’s manager is of the opinion that it is totally doable to meet the requirements within the time and people allotted to the project. If F. is of the opinion that this is not achievable we have a difference of opinion and the recommended strategy is to figure out where this difference of opinion comes from. In my experience, these types of differences of opinion usually stem from the fact that F. and their manager live at completely different layers in the problem fractal.

The nature of the problem fractal is that a lot of problems look incredibly easy when viewed from a high enough level, but at the same time seem really complicated when viewed at a (much) lower level. Managers often live in a weird state of knowing enough about computers to know that most problems are solvable, but not knowing enough about computers to know how incredibly hard seemingly simple sounding problems are. “Ok, so you need these low-latency caches in multiple regions to be consistent, how hard can that be?” Answer: “Very hard”, but that might be hard to explain to someone who only has a helicopter view of the problem.

Software developers typically only have a vague understanding of the problem they sign up to solve and only during execution do they figure out how hard it actually is. The only remedy for this is a planning process that is so exhaustive that you basically build the entire thing in order to figure out how to build it, then throw it away, and build the real thing. Nobody does this anymore. Mind you, we used to do this, but then customers were upset because they thought they already had a working version that we knew to be only the prototype. This led to many prototypes being deployed in production, with unsurprising results.

Instead of doing this exhaustive planning, we start with a vague understanding of what we need to build and how we need to build it, and then push someone for a date. Any software engineer in their right mind would refuse this, but eventually nobody can withstand the pressure to pick a date, any date, and eventually one gets thrown around.

Now we have a problem because, more likely than not, you get the go-ahead on the project only because of this entirely fictional date. Eventually, we find out that things are going to take much longer and hence will be much costlier than expected. This leads to significant frustration all around, especially in the manager, who in a sane organization will be responsible for this overrun. Plus they probably committed to other teams that this new thing would be ready by a certain date. And of course there is the budget and, not to be forgotten, the opportunity cost of the overrun.

The best way to prevent this problem is to never commit to a date until you are absolutely certain that you can do it within the time allotted. This is usually impossible. Even if you think you got line of sight to the solution, problems crop up left and right because our industry sucks and for one reason or another we constantly forget that.

Here is an example: Some time ago, I was (tangentially) involved in a security project to enable SSD encryption using features from the Opal Storage Specification. “How hard can that be?”, you would be forgiven to think. We have been encrypting stuff for decades now and the problem is well understood. Having the hard disk do the encryption is entirely possible and the interface to control that should not be rocket science. This project sounds like a week or two of work at best.

Oh, you naive flower child, how mistaken you would be, because of course the storage vendors managed to completely eff up the implementation of Opal again, leading to a variety of deep problems, including vendors implementing the standard incompletely, implementing it incorrectly, or just implementing it badly, leading to significant security vulnerabilities. Things are in fact so dire that the smart money will advise you not to use Opal but to use Linux’s LUKS disk encryption instead, which is a bit slower but has the upper hand on Opal because it does not completely suck.

Software development is problematic because every piece of software is a one-off. We are never able to learn from previous software development projects because every project is an entirely new universe of problems using different software and different people. Let me remind you again: Software development is not a manufacturing process.

If F. and their manager live on different planes of existence when it comes to technical detail, that is an information gap that needs to be fixed by F. and by good project management, including regular status updates including progress, newly identified risks, and the progress of the mitigation of previously identified risks. These status updates will also need to identify the extent to which the timeline is under pressure, so that they can discuss what to do about that at the earliest opportunity. If the timeline cannot be met, F.’s manager should not learn that only when the day of the “accurate ETA” rolls around!

The problem with status updates like these is that they are usually depressing and people hate giving out and receiving bad news. This then leads to the concept of “Watermelon reporting”, which happens when a project officially looks green on the outside, but is red all the way through once it is cut open.

The project management aspect of software development is not fun but, as I am wont to say: “That’s why we call your salary compensation”. Really, if work was fun, you would have to pay the company to do it as they’d be selling tickets at the entrance for the privilege of coming in to work. To make matters worse, the brunt of this sort of project management can typically not be delegated to a project manager because they live in the same information black hole as the manager. Project managers depend on accurate status updates from the software developers and, from experience, have a really hard time getting these for the reasons outlined above.

In a successful software development project, managers must be made participants in the process, get a say in how to address newly identified risks, and co-decide how to deal with timeline pressure: The responsibility for either delivering on time with reduced requirements or extending the timeline must be shared!

If F.’s manager is not interested in playing their part in the software development process, I revert to my earlier advice on finding another team quickly. It is impossible to work with a manager who does not want to be part of the solution, but does want to criticize you when you are not meeting goals that you were probably only partly responsible for defining. So if your manager asks you for an “accurate estimate”, whatever you do, do not provide a date until there is an agreement on the process of sharing responsibility for managing the process of developing that software.

Of course, it is also entirely possible that F. is an underperformer and that the manager is the good guy in this whole story. But let’s be honest and feel free to draw upon your personal experience here, how likely is that to be true?

Did you know that Wednesday Wisdom is also a podcast! Find it on Apple Podcasts or on Spotify. Did you know that there is also a custom GPT called Midweek Muse that has access to all of Wednesday Wisdom? Did you know that all Wednesday Wisdom videos are also available on YouTube?

Meanwhile, over at “Rare Jongens, die Amerikanen” (a Dutch-language sister publication), there is a new article on the problem of payments in the US (expensive, difficult, slow).

Then, finally: For this week’s article I owe a debt of gratitude to M. My conversations with her got me going on the ideas that underlie this week’s Wednesday Wisdom.

In Europe, there is a lot of anxiety these days, because they have figured out that pretty much everyone, from consumers and companies to governments, is dependent on US technology firms for pretty much everything. Governments run their email on Microsoft Outlook and write all their documents with Office365. Companies run their entire corporate workloads on AWS, GCP, or Azure. And consumers are obviously completely beholden to Google, Microsoft, Apple, Meta, and Netflix for their entire digital life, which these days encompasses everything from their media consumption to making phone calls and finding their way to the supermarket.

To a large extent, this anxiety arises from the fact that some legal amateurs have read the Cloud Act and are now under the impression that Donald Trump only has to call Satya Nadella (the CEO of Microsoft) to read the Dutch government’s email or to turn down the email service altogether. Similarly, these people are under the impression that a similar call from the orange clown-god to any American company will be enough to stop essential government services, even when these services run in Dutch datacenters on Dutch soil and are staffed by Dutch engineers.

But, this is not the anxiety that I wanted to talk about this week.

The newfound dependency on US tech companies brought to light a pre-existing anxiety about the state of European technology companies and specifically about the lack of thereof. A spectacular LinkedIn article (warning: Dutch) that was posted last week summarized the sorry state of affairs of the European AI industry in no uncertain terms: “Europe has 20 AI startups valued at over $500 million. The United States has 161. Europe has three frontier AI models; the U.S. has 33. Europe has 5% of global data center capacity; the U.S. has 69%. Eight times as many unicorns. Eleven times as many models. Fourteen times as much computing power.” The old continent is clearly not doing well in new technologies.

In a desperate effort to battle this anxiety, some deluded but positive spirits sometimes post a map of Europe on LinkedIn, sprinkled with a number of icons representing cool startups in an attempt to convince people that the situation is really not entirely bleak. Usually, I respond to these posts with a reply to the effect that I have more cool startups in my zip code in San Francisco, where I literally cannot go to a coffee shop without seeing two people with laptops starting an AI company.

In chat groups populated by Dutch IT experts there is considerable anxiety about this sorry state of affairs and in true European fashion there are lots of calls for the government to do something, because we all know that governments really excel at fostering dynamic startup cultures and creating low-friction environments where these startups can flourish. Everyone who ever received a government subsidy will know that the amount of process and paperwork is out of control and really, which startup wants their first five employees to be lawyers and project managers to interface with government bureaucracy?

Side story: When I was still involved running a Dutch IT company in the Netherlands, one of our activities was to help companies get EU subsidies from a program that was set up to incentivize training employees. That process was so onerous that lots of companies didn’t even try to get these subsidies and so they left considerable sums of money on the table. We employed some lovely young ladies armed with laptops and a solid understanding of the processes and forms involved to help companies bag these subsidies, in return for a percentage of whatever we recovered for them. This was probably one of the most profitable side businesses we ever had.

The anxiety about the startup climate in Europe is well founded, but this is also not the anxiety that I wanted to talk about this week. Instead, I want to talk about the anxiety factories that are most startups (successful or not). But before we get there, I first need to talk about something weird.

Did you ever wonder why the US startup scene is so thoroughly grounded in failures?

Many successful entrepreneurs first had businesses that failed, in some cases burning through millions of dollars of venture capital. So how does that work? VCs give a sack of money to some entrepreneurs who then go and burn it all in a failed business. Then these entrepreneurs come back to the VCs and say: “Well, that didn’t work out, but we have a new idea, can you give us some more money for that?” And then these VCs often do! Now ask yourself this: Why would they? Why would these VCs give more money to people who just showed that they failed to create a business?

Before we go on, please take five minutes to think about these questions. Would you do that? Would you give more money to some failed entrepreneurs? And while you are asking yourself these questions, also ask yourself the following associated question: Would any European banks and governments who are called upon to finance European startups do that?

[5 minute break]

The answer to this conundrum, I think, is that VCs do maybe not select on the idea, but on the people.

Let’s face it, it is incredibly hard to figure out if a novel business idea is going to work. It is all fine and dandy to say with the benefit of hindsight that Uber was always going to be a massive hit, but remember that the core of their business model is that people do exactly what people were always told not to do: Make a connection with a complete stranger on the Internet and then get into their car, often alone.”Or what about AirBnb, whose idea is: Let some complete strangers into your house to stay over while you are sleeping in the room next door.

A lot of successful businesses were once very unlikely ideas and in contrast, some completely obvious ideas never made it. It is hard to predict which ideas are going to work, which is why I think that VCs really do not select on the idea, or at least not on the idea alone.

So, if it is not the idea then, which aspects of the proposals they review do VCs select on. My thought: People! I think that VC’s partly, and maybe even to a large extent, specialize in finding the people that can make an idea work, even a suboptimal idea.

It is a bit of a truism that a mediocre idea executed by the right people will be more successful than a great idea executed by the wrong people. So imagine you are a VC and you see a proposal that asks for some funding. As we discussed, it is really hard to estimate the idea, but what you can possibly do is judge whether the people who make the proposal are the kinds of people who can make an idea work at all, any idea. And that determination is easier if you have seen them at work once or twice already. So, their previous idea failed, but maybe (or probably) nobody could have made that idea work. If these are the people that you think can make a half-way decent idea successful, it makes total sense to give them money for their new idea because even if you cannot accurately judge the idea, you can maybe judge these people.

I think this fact explains why failed entrepreneurs get money for their second or third idea; they have shown that they are the people who will go for the jugular in order to make the company a success.

So who are these people who can make any idea work? These are obviously the world’s go-getters, the people who are extremely goal oriented, who are willing to sacrifice everything to make their startup a success, the people who can (and want to) work 24x7 and who can find and drive other people who have the same mindset. These are the people who, in the pursuit of success, turn their fledgling company into an anxiety factory.

So many startups are anxiety factories because their founders were literally selected for their ability to load up and deal with incredible amounts of stress while pursuing success. And since A’s hire A’s, their first batch of employees are of the same ilk. See here all the ingredients for a combustible work-life imbalance cocktail that can either explode upon lift-off or go to infinity (and beyond). So if you join a startup, do not be surprised if it seems to be fueled by anxiety.

I must admit that I belong to the group of people who do well in these anxiety factories. I am currently working in the epicenter of Big AI and when asked to describe what that is like, I often say that it’s akin to holding on by your fingernails to a launching rocket. I am more stressed than I was ever before, but I am nothing if not goal oriented, so I hang in there and try to make it a success, probably with some detrimental consequences to my short-term health and well-being. “Why do that?”, is a fair question and the answer is probably (at least partly) that I am wired that way. I committed to a professional goal and I might maybe not do everything it takes to reach that, but certainly a lot.

It is an example I would not recommend you follow, unless you are also wired this way.

Did you know that Wednesday Wisdom is also a podcast! Find it on Apple Podcasts or on Spotify. Did you know that there is also a custom GPT called Midweek Muse that has access to all of Wednesday Wisdom? Did you know that all Wednesday Wisdom videos are also available on YouTube?

Quite a lot of Wednesday Wisdom articles start with the sentence: “When I joined Google in 2006…” followed by some anecdote. For this week’s article, I thought it prudent not to break with that tradition, so here we go…

When I joined Google in 2006, I considered myself a decent C++ software engineer, on account of having written some C++ code and having taught many C++ courses, including courses on how to code using the Microsoft Foundation Classes (which we nicknamed the “Microsoft Frustration Classes”). Google had a rigorous system of code review and because I did not have “readability” in C++ yet, I submitted my first changelist to a “readability reviewer”.

The concept of code readability at Google meant that (among other things) code written by someone without “readability” in that language was subject to stricter code review and approval requirements. If you had readability in the language, your code needed only an “LGTM” (Looks Good To Me) from a random colleague, but if you did not have readability, you needed an explicit approval from a colleague who did have readability in that language.

Brag: I have readability in C++, Python, Sawzall, Go, and Borgmon.

Like I said, I considered myself a good software engineer with a decent understanding of C++ already, so I was full of unwarranted confidence. When I got the changelist back, it came with dozens of comments, some of which were nitpicky (#naming and comment style), but quite a lot were actually very good and either went straight to correctness of any code that would depend on my code (e.g. declaring a private copy constructor using an internal preprocessor macro) or instructed me to use functions or classes from our homegrown “google3” library, instead of writing my own crappy implementations. It was a chastising moment, but since one of my mottos is “Adopt, adapt, and improve”, I quickly got with the program.

I was initially quite impressed with the code review requirements and quite a few bugs in my code were found because of diligent code review by colleagues who were experts in their fields.

My favorite code review comment of all time came on a five-line Borgmon changelist where the reviewer had written: “This changelist is amazing. You would expect it to work; it probably should work. But it doesn’t, and here is why…”, followed by an A4 size explanation of some arcane consequences of the polling interval, how time series line up, and how builtin functions would evaluate in the face of missing data points.

But here is what I quickly found out: Code review does not often comport well with velocity. In environments where high velocity is more important than five nines of reliability, the requirement that even a one-line code change needs to get reviewed sometimes adds days to the timeline of closing a ticket or solving a bug. Days that you do not have.

You would think that a one-line change would be easy to review, but here is the problem with code reviews: The approver becomes part of the set of people who take the blame if something catastrophic happens because of this code. By approving a change, you take responsibility for that change and really, in most companies, what is the incentive to take that risk? Better if someone else reviews it who has a better understanding of the system under change or who has an in-depth understanding of the well-known vagaries of the “Infrastructure as Code” setup. Receiving good code reviews is amazing, but giving good reviews is time consuming and exposes you to mostly unnecessary risk.

This is all fine and dandy if you are developing some system as a team and you are responsible as a team regardless, but if you are an atom writing some code, how do you get a random team member to sign on to take responsibility for your crappy code which might break the system? In a high-functioning team, this is easy to figure out, but in the dystopian post-apocalyptic chaos that is most organizations, this is a problem, which is why many people need to go begging for approvals on Slack in order to make progress on their projects.

It is a well-known fact that if you use a metric to measure some underlying behavior and then make meeting that metric an official goal, people start gaming the metric, because that is usually much easier than engaging in the downstream behaviors that then may or may not drive a “good” value for the metric. Something similar happens with code reviews: Configuring your version control system not to accept any changes without an approval arising from code review, makes people seek the approval but not necessarily the review. Many teams I know have a tradition of “stamps”: Meaningless approvals given without review so that the change can be merged and escape velocity reached. It is a good example of “form without function”: The configuration of the version control system might make the compliance people and auditors happy, but it is often without any practical merit.

Obviously, in an environment like that, removing the approval requirement would make total sense but this never happens. First of all, every rule has an ambassador and a fan club, and so every proposal to get rid of code reviews will run into stiff opposition from the people who truly believe that the problem to be fixed is that people do not do meaningful code reviews, despite the fact that this has not worked so far. The second problem is that the official existence of “approval after code review” is a “Cover Your Ass” measure; if the site goes down or some malicious code makes its way into the code base, the absence of an official code review policy will make you look like a troupe of clowns and might even bear down regulatory scrutiny as code review is usually legally required for systems that touch money or medical data.

So there we are, we should do code reviews, we want code reviews, but practically speaking very few people do meaningful code reviews.

The question on how (or if) to do code reviews is exacerbated by AI programming tools. As I wrote in an earlier Wednesday Wisdom article, I have written about five lines of code this year so far, though I merged tens of thousands of lines of code that were attributed to me. What used to be the coding part of my job has been replaced by talking to Codex and marveling at how much decent code it wrote in the space of mere minutes.

The first question to ask is whether I should look at the code that was generated. The easy answer is: “Oh my God, of course”, since it feels bad to implicitly trust a metric ton of code that was generated by a program without detailed input from a human. When I first started working with AI programming tools, I diligently looked at all the code in some detail, but as my confidence in the quality of the generated code increased, my attention waned because, as a friend of mine so eloquently puts it: “Laziness is the hidden energy that powers the universe”.

To be honest, I sometimes still look at the code in some detail, but only when I feel that the topic is so complicated or subtle that I consider it hard for the model to generate the correct code from a few lines of instructions and the existing code base. This review sometimes leads to small changes and on rare occasions it leads to me throwing away the entire branch and starting anew with improved instructions.

Here is an example of the latter: I recently worked on a piece of code that does something interesting in our Azure environment. Obviously (or maybe not, but anyway), we live in a multi-cloud world, so one of the tasks on my plate is to adapt the code to support our other cloud providers, starting with GCP. I started by writing a design doc. Right off the bat, this is already a decision that is not as obvious as it seems, because why not just point the model at the existing code and tell it to support GCP? Water under the bridge though, because I had already completed the design doc before that thought entered into my head. I pointed the coding tool to the existing code, gave it the design doc, and set it to work. It created a plan, which I lightly reviewed, and then it started plowing through the steps in the plan.

When I looked at the code it had generated, I saw lots of “if GCP” statements, which I did not like at all because it is not very elegant and would turn the code base into an incredible mess once I would add support for additional cloud providers. So, I deleted the entire branch and started over by first instructing the bot to extract all Azure specific code into a module with an abstract interface and then add support for GCP. That went much better already.

Or did it?

One of the things I am wondering about is why I care about that potential for spaghetti. Surely, it would make it really hard for any human to keep track of what is going on, but would it also make things hard for the bot? Is any human ever going to look at that code to debug or improve?

AI coding tools do not only excel at writing code, they also excel at pointing out what is going on when given a pointer to the code and a log file. Whenever I don’t understand something my code is doing, my first action is to throw the log file into the AI tool and have it explain to me why I am seeing what I am seeing. Given that the tool does the correlation between the log file and the code, (and does that quite well) am I really ever going to look at this code again with any amount of concentrated focus? Or is the bot really the only thing that will ever look at this code? Interesting question to which the answer is entirely not obvious.

On the off chance that I would ever want to make sense of the code, I wrote some generic instructions for the bot to always generate helpful comments and to use the programming language’s features for constants to refer to magic values using a descriptive name. That definitely helps me to make sense of the code, but does it matter for the bot? Given that our AI tools are based on large language models, maybe more language in the codebase helps activate the right artificial neurons, but does it? I really don’t know… Maybe I am just polluting the context with extra information that the tool does not need…

After I am happy with the code, it goes to the version control system where some human will need to approve it before I can merge. Now we get to the second set of questions: Should my colleagues really be expected to review the code and if so, what exactly should they review? Should they review the output of the bot, the intent (what was I trying to do), or the input (in other words: The prompts and maybe the ensuing discussion)? Again, I don’t know…

I recently got a code review from a colleague on a Codex-generated PR where they pointed out an invalid assumption of the nature “if a is set to b, then c must be set too”. Turned out not to be true, because, as I learned, when a is set to b there is a default for c that makes sense and that you might want to use too. Now the thing is, I cannot remember if I introduced that assumption into the context of the bot or if this is something the bot came up with all by itself. This is of interest, because it is an open question whether a review of my prompt would have triggered this particular comment from my colleague.

So in other words, code reviews, which were already under pressure, might have to change dramatically. When reviewing a human’s code, there is usually a fair amount of attention to subtle coding errors, like off-by-one mistakes and memory management errors like use-after-free. The bot probably gets these right 100 times out of 99, but it might be liable to make certain semantic errors because of lack of context.

Speaking of context: There is the interesting fact that an LLM integrated into the code review tool sometimes finds errors in code that was generated by that same LLM! That’s kinda weird, right, but it goes to the importance of the “right” context.

We surely live in interesting times.

Did you know that Wednesday Wisdom is also a podcast! Find it on Apple Podcasts or on Spotify.

Because I am an idiot, I recently volunteered to refactor the oncall situation of my team. It is the kind of organizational thankless task that I gravitate to because either I think that I need to atone for something I did wrong (lapsed Catholic here) or because I feel a pressing urge to “take one for the team.” But there was an upside: The effort did give me a nice opportunity to ponder the many problems I have experienced over the decades while being oncall or setting up oncall rotations. I figured that for this week’s edition of this classic article series, I would neatly organize the top ten things not to worry about when dealing with oncall.

The unmistakable number one entry has to be that you should not worry if you feel that you do not know everything there is to know about every service that you are oncall for. Nobody ever does and if that was the bar, nobody would ever qualify to be oncall for anything. To ease the minds of my junior colleagues a bit, I typically tell them that the task of the oncall engineer is to pick up the phone, give it their best shot, and then escalate to someone else, who will then rinse and repeat that exact same procedure. I have been oncall for many things that I know Jacks From Sheets about and I am still alive 🙂.

At number two, we have the number one complaint from people being oncall, which is that the documentation sucks. Yes it does and don’t worry about it because that is always the case and will always be the case and there is literally nothing you are going to do about that. All documentation, including the runbooks, are always bad, missing, incomplete, wrong, outdated, or just plain fantasy. No amount of effort you are going to spend will change that. Instead of expecting documentation, learn how to target your AI programming tool of choice (might I recommend Codex?) at the source code of the offending system and let it tell you what is going on and what you should be doing. Personally, I like writing documentation and I like to think that my documentation is in a decent state, but it never survives first contact with one of my colleagues because it turns out that apparently not everyone has the cdrtools installed and at least one of my runbooks assumes that.

Coming in hot at number three is the fact that there is no perfect oncall schedule and neither is there a great scheduling tool. All schedules are suboptimal and even if you carefully fill your calendar with “no-oncall” entries and teach your professional or homegrown constraint solver to honor these, you will end up with a schedule that is hugely inconvenient to everyone. Nothing to be done about it so don’t worry; just extend your fixed order schedule and let people swap shifts and enter overrides as they see fit. At the end of the day, all oncall schedules run on spreadsheets. So don’t worry that you cannot create the perfect schedule. Nobody can, because it cannot be done. Also, don’t worry about whether your oncall shifts need to start on Monday, Tuesday, or Wednesday; every choice has good and bad aspects. Or if you are in a team that wants to split the oncall in a shift from Monday to Thursday and another shift starting Friday and including the weekend: Great, whatever, it does not matter, but if that is the particular bikeshed you want to paint, go for it.

At number four, a strong contender for a higher spot in this top ten: All alerts are terrible. They are too strict, too loose, or resolve themselves before you can even open the alerting dashboard. They fill up your Slack channel or make your phone ring off the hook. On top of that, there are always significant gaps in coverage.

Anecdote: I once was oncall for YouTube when the phone rang and someone who announced to be the CEO’s admin reported that YouTube uploads were broken because she had uploaded a video an hour ago and it was not live yet. “Surely,” we thought, “if uploads are broken, our pager will be ringing off the hook.” But, because this was Eric Schmidt’s admin, we thought we’d take a look. The dashboard showing the number of files uploaded per minute showed a healthy number. The dashboard containing the number of transcoding errors showed an equally healthy zero. One problem though, the graph of the number of transcodes started also showed zero. Turned out that there was a problem with the job that moved files from the upload bucket to the transcoder bucket and there was no alert on that value reaching zero.

While we are at it: Another anecdote: In one team I was on, we did have an alert for zero; in this case for the number of recently committed change lists merged into the production branch (that was regularly pushed to, well, production). One fine Boxing Day my pager rang because the merge bot had not merged a single change from the trunk into the production branch for the last 24 hours. After careful investigation, it turned out that this was because exactly zero change lists had been merged into the trunk in the last 24 hours, on account of it being Christmas. Apparently, this had never happened before.

Writing good alerts is as difficult as writing good software, but whereas software gets a design doc, sprints, code reviews, tests, and sometimes even quality assurance, alerts seemingly are written on a lazy Friday afternoon by an intern on fentanyl. For big events, Twitter is usually a better alert source than anything else you might cobble together. While oncall for YouTube, I always got informed about big outages by my non-Googler friends, through a chat or text message, before any alerts fired.

But then again, one day when I was oncall for Google Maps, a colleague paged me because his friend in Uzbekistan couldn’t reach Google Maps, so surely Maps was down? I kindly suggested this might be an ISP-related problem in Uzbekistan.

In short: All alerts are terrible so don’t worry if yours are too…

On top of sucky alerts, closely following at number five: All metrics are terrible. They are missing, measuring the wrong thing, measuring the thing wrongly, or not in the unit that you think they are in. Literally every metric that I ever dove into, down to the source code, had something wrong with it. Even something as simple as the request count or the time spent executing the request, is typically measured at the wrong place in the request handler or in the wrong way entirely. But, don’t worry, because, again, there is nothing to be done about it; it is you against an ocean of time series and, as sailors know, the ocean always wins (or does it?).

Surely, you might think, we can put a project together to fix the alerts and the metrics. And surely, you are completely right, theoretically… Practically speaking, few teams ever get that together because there is always something more important to do and the pain of the sucky alerts and metrics is nicely spread out among all members of the oncall rotation, thereby ensuring that it never reaches anyone’s pain threshold. There once was this one service at some employer for which I was the sole oncall 24x7, 52 weeks per year and believe me, those alerts were tuned to never fire unless all hell had broken loose.

Getting to the lower half of this top ten, right there at number six is the worry that during your oncall week you are getting nothing else done because of constant interruptions. This is most likely true, so don’t worry about it as you should only worry about things that might happen, not about things that are going to happen. But it’s okay though, the organization should be fine with this because they have chosen to afford their velocity on the back of your nervous system. Now, to be honest, in the best oncall rotations I have been in, I spent at most twenty percent of my time on oncall related tasks, but these are the exceptions, not the norm. When your oncall load is unseasonably low, that is more often than not caused by the fact that the SMEs are silently picking up a lot of the operational work around “their” services instead of letting it fall to you. So nice of them…

Getting to number 7: Let go of your worry and fear that you will look stupid during an incident. Incidents are not IQ contests. They are debugging problems. Ask the obvious questions, verify the assumptions. As the oncall engineer, I expect you to perform an honest debugging effort, from first principles if need be. Pages represent a software reliability problem, not a moral defect in the unlucky person holding the pager. As the holder of that pager you are the first witness to the things your colleagues did wrong. It is not your fault…

Unless it is of course 🙂.

Reaching the nether regions at number 8: There is no need to worry that you won’t be able to go anywhere during your oncall week, because it is usually no longer true, provided that you take some measures and don’t act stupid. In the olden days, when I got paged, I had to either solve the problem by phone or jump in the car, drive 45 minutes to the data center, hope security would let me in, fire up my 3270 terminal, and start solving the problem.

These days you can be oncall from anywhere provided you have a decent laptop and cell phone. I have been oncall while on a plane (I don’t recommend it), from the tropical backyard of my aunt’s place in Spain (I do recommend that), and while staying over after a successful Tinder date (I highly recommend that because: Unlimited brownie points if you get paged during the night and you are rescuing a well-known Internet service from your date’s kitchen table; I am obviously not speaking from experience 😉). If you want to go to the cinema, no problem but be prepared to step out, which would suck but hasn’t happened to me thus far. I wouldn’t necessarily go to a rave because conditions there might preclude me from hearing/feeling the pager, but I guess it can be done, if done right.

Almost at the bottom at number 9 we find this common worry: I will look dumb if I ask for help. No you don’t! Escalating is not a weakness; it is part of doing the job responsibly. Good teams want you to do that. Here is the proof: I recently asked my colleagues to write down their number one, two, and three tips for new engineers on the team who are about to go oncall. More than half of them wrote things like: “You are not alone!” or “Ask for help!” Every seasoned oncall engineer who is also a normal human being remembers what it was like when they were first oncall. No seasoned oncall engineer emerged from the womb knowing the dashboards, the failure modes, the service dependencies, and the bizarre historical reasons why this one restart script lives in some forgotten directory. Everyone who is good at oncall was bad at it once. They learned by doing it, and by asking for help, which is exactly what you should do.

Then finally at number 10: Not a worry but just some plain old advice: Oncall teaches judgment faster than almost anything else. It might be uncomfortable at first, but it is one of the quickest ways to learn how systems really behave and which assumptions are fake. The pager is a harsh teacher, but a very effective one. One of the reasons why I still like being oncall after many years in the trenches is that nothing teaches me the ins and outs of a new system faster than being forced to look into some problem. You don’t have to solve it fast; I sometimes spend days debugging a simple Slack question, but at the end of that I know things about the system that almost nobody else knows.

Don’t be afraid. Yes, the pager is where reality comes to collect and reality is often rude. But remember: The pager is testing the system, not your worth as a person…

Before we begin, I have three service announcements:

1. Did you know that Wednesday Wisdom is also a podcast! Find it on Apple Podcasts or on Spotify.

2. I revamped josvisser.nl which is as useless as ever, but better looking. However, when doing so I did find a whole ton of writing that I thought had been lost to history. Not all of it awful.

3. For our Dutch readers: We have a new sister publication: “Rare jongens, die Amerikanen” with observations about living and working in the United States of America.

With that out of the way, here is this week’s article…

In my first job, I was part of a group of systems programmers who took care of the IBM mainframes of a large Dutch bank. It was a “golden age” (of sorts) of information technology, because computers were new, expensive, and unnecessarily complicated. This made the people who knew anything about them rare, and consequently they could do more or less whatever they wanted as long as they kept the big calculators in the basement humming along. I would come in every morning and if there was no urgent problem to work on, I was pretty much free to spend my time however I saw fit.

The original IBM mainframes (S360 and S370 families) were based on a 24-bit architecture, limiting addressable memory to 16 MB. When this turned out not to be enough, IBM introduced the “extended architecture” (a.k.a. XA), increasing addressable memory to a whopping 2GB.

Q: Why 2GB?
A: Because XA had a 31-bit architecture.

Q: Uhhh, weird! Why 31 bits?
A: That is really best left unexplained, but suffice it to say that it was because of backward compatibility with the 24-bit architecture and an incredibly bad design decision in that architecture to use the highest bit in a (32-bit) pointer for something else.

Eventually, even 2GB memory was not enough and IBM introduced the Enterprise System Architecture (a.k.a. ESA), which supported an inconceivable 16TB of addressable memory through something called data spaces. Lots of fascinating technical details here, for which we unfortunately do not have time…

With the new architecture came new releases of the prevalent operating systems and one of these was MVS/ESA, which contained many fascinating new features. As the bank’s system programmers we saw it as our job to understand and use as many of these new features as possible, egged on by IBM, who clearly understood that our love for their machines and operating system was one of their strongest assets.

For the interested reader: During my website revamp I unearthed a 1992 article, potentially my first ever article to get published, that contains an assembler listing showcasing some of the “new” features in ESA, including data space access and an instruction to call a subroutine while putting the return address on some new thing called the “stack”. Totes wild.

In a nutshell: We all suffered from a serious bout of shiny object syndrome: A tendency to get distracted by new, exciting ideas, tools, or opportunities, and to abandon the current plan before it has had time to pay off (thank you ChatGPT for this concise definition).

All my life I have suffered from shiny object syndrome, which in my case comes with a tendency to spend a lot of time on exciting new ideas and products without really figuring out beforehand if it makes economic sense or not to invest that much time or if the new thing meets any pressing need I have. Sometimes this paid off, but mostly it didn’t. It seriously paid off with Linux, the Internet, Java, Go, and Rust, which, and it is maybe hard to imagine this, were once just shiny new things that didn’t work particularly well but which many thought harbored great promise. It did not at all pay off for all the time I spent on OS/2 Warp, BeOS, Hyper-G, CORBA, and other things that often bore equally great promise but went nowhere. Often these technologies were so technically interesting that I wanted them to become successful, but of course my support was completely meaningless.

Our entire economic model promotes shiny object syndrome, especially in a world where winner takes most and being early to the game is a strong predictor for who is going to win. A good recent example of that is the blockchain, which is of great technical interest but serves only very narrow practical use cases, mostly in the realm of cryptocurrencies which is a massive scam to begin with. But, that said, an entire industry sprang up around blockchains with massive investments in mostly vaporware startups and people eviscerating value faster than I can say “bored ape”. I once thought to write a satirical post about “dating on the blockchain”, but no, someone had already proposed it for realz. And for some unfathomable reason, it became a minor news item again earlier this year.

For good measure: I want to separate this from laudable efforts to build a dating app for blockchain enthusiasts, which is just target audience selection in a “traditional” app. There are dating apps for Jews, Christians, Goths, and for the disabled and chronically ill. It makes total sense that there would be a dating app to connect blockchain enthusiasts.

Compulsory dad joke: Where do dating apps store their data?
Answer: In a relational database! (get it?!)

The problem with foolish shiny object syndrome is that it leads to loss of focus.

At one Big Tech company I worked for, I once had a colleague who suffered from shiny object syndrome in a big way. There was a lot going on in said company and new internal and external products were launched every other week. This colleague was always in the know and usually among the first to use a new product or API. Keeping up with all of that sucked up a lot of their time, since they read all the mailing lists, all the announcements, went to all the tech talks, and downloaded and experimented with all the products.

This is of course a great colleague to have around, because we could always ask them what was up with something. But here is the problem: It is really difficult to get your actual job done if you are constantly chasing all the shiny new things around you, especially if there is an abundance of them and it is a total crap shoot whether some or all of them are actually going to become important for delivering on your OKRs.

That said, when Bitcoin came, this colleague went big into that as well and eventually made off with a fortune, so I guess the joke is on me…

As you probably know, I work at OpenAI these days and we are also in the business of regularly releasing exciting new things. As a matter of fact, yesterday (at the time of writing) I read on one of our main internal Slack channels that we built some amazing new features in one of our core applications and could the product team please get feedback from our internal user base on how well these work? Super fun thing to do. Important too for the company. But here is the thing, at this time I really have no time for this unless I want to jeopardize some of my other deliverables. I would love to drop everything and try out <super secret cool feature> but, all things considered, what is best for me and for the company?

Obviously, trying out new things and giving feedback is an important contribution to the product team and thereby to the company, but you need to make smart decisions about what to work on. I do not advocate for never giving in to your shiny object syndrome, but I would also warn about giving in too much and to the detriment of your overall productivity.

Shiny objects are the perfect filler for your low productivity hours. We all suffer from these moments when our energy store has depleted and we cannot get anything done that requires even a modicum of willpower. When that happens, you can either go and surf LinkedIn or pick up a shiny object and play with it. Perhaps also not the most productive thing you could theoretically be doing, but maybe the most productive thing you could do, given the state you’re in. Like most drugs, shiny object syndrome does not have to be destructive when given into infrequently and managed well.

So what happened to my shiny object syndrome team in my first job?

Our manager was an incredible killjoy who tried to keep our shiny object syndrome under control. Instead of letting us go rampant in enabling new operating system features left and right for no other reason than that they were cool and we wanted to play with them, he made us write “Implementation plans” for each product in the OS that we were responsible for. For these plans we had to research which (new) features the organization needed, how they should be configured, and what service levels we should offer.

Boring! (but productive).

Did you know there is also a podcast of this episode? You can find it on all high quality podcast platforms. Cannot find the podcast on your platform of choice? Ping me!

A side effect of Big Tech’s longevity, their unconstrained hiring from 2010-2022, and the subsequent layoffs since 2023, is that there are now a lot of people around whose resumes feature a stint at a “FAANG”. By and large, these are “big machine” people.

I count myself lucky that when I joined Google in 2006, not everything was amazing yet. We built binaries with carefully crafted Makefiles, we ran services on machines operated by the “babysitter” platform, and compiling and testing happened on an underpowered cluster of machines that was so overloaded that you had no hope in hell getting anything done once the offices on the west coast of the US woke up and starting submitting jobs. Bigtable was only just becoming available and Spanner was nothing more than a glint in the eye of Jeff Dean. Don’t get me wrong, it was already much better than anything I had ever seen, but compared to the Google technology of 2010 and later, it was kinda crappy…

Google went on to create one of the most phenomenal internal compute platforms that human ingenuity can imagine and build; a hyper advanced platform that can run services at almost unparalleled scale with a matching, and equally almost unparalleled, reliability and security. This “reduced” most Google engineers from people who build and run advanced infrastructure to people who operate services on top of a big machine. AWS, Facebook, and other people built similar platforms, often modeled after what Google had wrought first.

Side story: Somewhere in 2018 I was the victim in a “Wheel of Misfortune” exercise.

For those of you not in the know, “Wheel of Misfortune” is a failure exercise in which a “dungeon master” presents a production problem that a victim then has to solve. The victim is not allowed to use their computer, but must instead instruct a “displayer” (who has a laptop and is projecting) on where to go, which buttons to press, and which commands to execute.

When the dungeon master presented their problem, I started giving instructions to the displayer: Go here, execute this command, check this log file, enter this expression into “Nebgua” (a browser plugin for constructing URLs for our monitoring system; the acronym stood for: “Never Edit a Borgmon Graph URL again” 🙂). My younger colleagues watched in disbelief and amazement. He is going where? He is doing what? He is issuing which commands? Eventually, another colleague intervened: “No, he is not trolling you, that is how we used to do things back in the olden days.”

My younger colleagues had turned into big machine people and the only thing they knew was how to operate the big machine.

An obvious problem with being a big machine person is that if the big machine is broken, it takes detailed knowledge about the big machine’s internals to figure out what to do now. It is for that reason that in every year’s disaster recovery training, I included a task for someone to roll out a new release without using our CI/CD system, pretending that some disaster had taken it out. Come to think of it, quite a lot of “DiRT training” exercises consisted of scenarios where a part of the big machine was “broken”…

Both for older and newer engineers, a common problem of big machines is that it stops them from doing what they want to do, how they want to do it. I have literally spent hours scratching my head and figuring out how to get a big machine to do something that I knew perfectly well how to do using the basic tools available on the operating system, but I just couldn’t find the button, config file, dropdown box, or whatever UI element that the big machine’s product managers figured I should use for my operation. Half of the time, I had to eventually come to the conclusion that the big machine did not let me do whatever it is that I wanted to do, despite that being a completely valid state transition in the underlying system.

I do not necessarily like big machines very much…

Another problem with big machines is that it turns many engineers into big machine people. What I mean by that is that their entire expectation of the environment that they are operating in is formed by the availability of a big machine for Everything(™). When these people move to another company, they typically find a dearth of big machines and their tendency is to a) complain about it (saying things like “at Google…”) and then b) build one. Big machine people want to build big machines because they think that big machines are what every company needs and should want, regardless of the state of the company, regardless of the span of control of the people inside the company, regardless of the company’s financial possibilities, and regardless of the time horizon on which the company operates.

Many companies, including all startups and most scaleups, cannot afford a big machine or do not have the mindset to build and run one.

Building and running a big machine costs oodles of money and takes oodles of time too. Time and money that a company might not have or that, all things considered, it might want to spend somewhere else. Nothing wrong with that, many startups just try to make it to the end of the quarter and many scale-ups are racing towards the break-even point in the most cost-effective way possible. A big machine might not help with that.

In my career, I have come across many big machine people in not-quite-so-big-machine companies. As is natural for them, whenever they see a problem, they want to fix it by building a big machine. They go on to write a big design doc, organize a big review meeting, and then try to assemble a (relatively big) team to realize the big machine in a big project that takes a big amount of time. These projects often fail in a big way because eventually someone figures out that a group of people is working on something that does not promise any value in the short term.

Many startups and scaleups do not have the stamina to build a big machine. That’s not because they are bad people, but building a big machine requires a big sponsor and the organizations in these small and growing companies are often continuously in motion, with sponsors moving around and new executives coming in who do not see the value of the big machine. You really need a big, stable, company to build a big machine.

When people ask me what working at Google was like, I often answer that Google is the Germany of engineering: Everything takes a lot of time to build, is hugely expensive, and when it is ready it is about three times as good as it needs to be. Google excels at building big machines.

For reference: When people ask me what working at Facebook was like, I answer that they are the France of engineering. Whenever they need to build something really good, like the TGV or the Ariane rocket, they are right on it. But most of the time they’re like: “Mweh, let’s finish this quickly and drink more wine” 🙂.

I am, at heart, a big machine person and I regularly need to pull myself away from the brink of writing a big design doc because I realize that right now we are not the people who build big machines for things that are not in our core activity of servicing user requests. I also regularly remind colleagues, who I suspect of being big machine people, that they are better off proposing a simpler solution that we can build quickly and then later extend when it gets traction.

That last sentence is key. We might not want to build a big machine right here right now, but we need big thinking to make sure that the small machine we build is good enough and can be extended to become a slightly bigger machine. The absence of big machines does not mean terrible chaos; instead, it should mean a swarm of smaller machines that are just good enough to solve the problems we have and that we don’t want to live with.

Big machine thinking, combined with big execution, gave us the F35. Big thinking but small execution gave us drones. There are definitely use cases for big machines, but as both Ukraine and Iran are showing us right now, a swarm of crappy drones can be quite effective. This also holds for CI/CD, quotas, rate limiting, and load balancing: A big machine works well for these problem areas, but I have seen many small machines that cost 1/10 to build work almost as well and usually more than well enough. Remember, quality is fitness for use!

So, unless you are in a big machine company, resist your natural inclinations to design and build a big machine. Instead: Consider thinking big, but executing small!

This week’s Wednesday Wisdom features a competition of human against bot. I took the question of how to become a better engineer, wrote an article about it, and asked Midweek Muse (a ChatGPT trained on Wednesday Wisdom content) to write an article about it too. I am posting both of these articles and you choose which one you like best. This is the article written by the bot.

Every now and then someone asks me how to become a better software engineer and I am always a little bit surprised by the question.

Not because it is a bad question. Quite the opposite; it is one of the better questions in our field. The thing that surprises me is that many people seem to think there must be some secret answer. A framework. A course. A YouTube channel with a man in a tight t-shirt explaining system design while gesturing at rectangles on a whiteboard.

I am afraid there is no secret.

Becoming a better software engineer is mostly a matter of doing a handful of deeply unsexy things, over and over, for a very long time, until the whole pile starts compounding.

That is terrible news for the shortcut crowd, but excellent news for everyone else.

Let’s start with a heresy: Becoming better is not primarily about learning more syntax.

Syntax matters, obviously. If you do not know the language, you are not going anywhere. But the difference between an average engineer and a very good one is rarely that the latter remembers one more decorator, one more stream combinator, or one more flag to the build tool. The real difference is usually that the better engineer understands the problem more deeply, sees failure modes earlier, asks better questions, writes code that other people can live with, and debugs without panicking.

In other words: the better engineer has more clue.

That word “clue” is doing a lot of work here. I use it on purpose because it captures something that is broader than “knowledge” and more useful than “intelligence”. Clue is not merely knowing facts. Clue is understanding how things fit together. It is knowing why the system behaves as it does, what can go wrong, where the sharp edges are, and which parts of the problem are actually important.

A lot of people confuse being busy with getting clue.

These are not the same thing.

You can spend all day writing tickets, attending standups, rebasing branches, moving cards on boards, and making dashboards greener without becoming one millimeter better as an engineer. You can also spend an afternoon chasing down one nasty bug in a production service and come out of it materially wiser.

Experience, unfortunately, is not the same as time served.

There are people with ten years of experience and there are people with one year of experience ten times. The industry contains plenty of both.

So how do you get clue?

First: Keep coding.

I know this sounds insulting in its obviousness, but I regularly run into senior engineers who have somehow managed to float upward into a life of documents, meetings, and opinions while leaving actual code behind. This is a mistake. You cannot remain a serious software engineer if the actual making of software has become something you merely remember fondly, like roller skating in the eighties or your first marriage.

The workplace conspires against coding. Meetings breed. Projects sprawl. Slack pings. Some project manager wants a status update on the status update. Before you know it, the week is gone and the only thing you actually produced was a paragraph saying that alignment is ongoing.

Fight this.

Code is where reality lives. Code is where your theories are punished. Code is where your hand-wavy architectural brilliance runs into the stubborn facts that the compiler, runtime, network, file system, and other humans do not care about your vibes.

If you stop coding, you slowly become a tourist in your own profession.

Second: Learn how to debug.

This, in my opinion, is one of the deepest craft skills in software engineering and one that is criminally underappreciated.

Many people think debugging is a temporary inconvenience between them and being done. Wrong. Debugging is the job. Or at least it is a very large and very educational part of it. Debugging is how you learn what the software is actually doing, as opposed to what you emotionally feel it ought to be doing.

A bug is reality writing back.

The bad engineer responds to this by immediately asking in chat if anyone has seen this before.

The good engineer starts narrowing the field.

What do we know? What changed? What state is the system in? What assumptions are being violated? Can I reproduce it? Can I corner the bug so it has nowhere left to hide?

None of this is glamorous. No one is going to throw you a parade because you spent three hours staring at logs, tracing state transitions, and adding ugly temporary instrumentation to prove that some innocent-looking function is in fact a criminal. But this is how you build instincts. This is how you stop being a recipe engineer and become someone who actually understands systems.

There is a huge difference between knowing that some incantation works and knowing why it works.

The former turns you into a black box engineer. The latter turns you into someone useful when the recipe stops working.

Third: Know your tools.

I don’t care very much whether you use vim or VS Code, Go or Rust, Postgres or MySQL, AWS or GCP. I do care whether you understand what you are using.

Far too many people use tools as if they were magic. They know which button to press, but not what sits behind it. This is fine right up until the moment it isn’t, which, in software engineering, is usually around 02:13 in the morning when production is on fire and your cheerful little abstraction layer has gone off to die in a ditch.

If you are writing software that runs on Linux, know Linux.

If you are building on the cloud, know the cloud.

If your entire world depends on Postgres, Kubernetes, Kafka, Terraform, or whatever the current deity is, then for the love of all that is holy, understand how it actually works.

Not every detail, obviously. Nobody knows everything. But enough that when things get weird, your first response is not: “Huh, that’s odd.”

People who do not understand their tools tend to make two kinds of mistakes. They either overestimate them and trust them too much, or underestimate them and keep re-solving already solved problems. Both are expensive.

Fourth: Write code that your future self can read without swearing.

As a young engineer I suffered under the charming delusion that whatever I wrote, I would surely still understand later. This turns out to be false. Very false. Embarrassingly false.

You are not writing code for the computer. The computer, poor thing, will execute almost any trash you feed it if the syntax is acceptable. You are writing code for the unlucky human who has to understand it later. Quite often that unlucky human is you.

Good code is not code that merely works. Good code is code that makes sense. It has locality. It has names that help. It handles errors like it knows they exist. It does not rely on mysterious side effects, hidden contracts, and wishful thinking.

A lot of bad code functions approximately like a haunted house: every room appears normal until you touch something in the kitchen and a ghost starts screaming in the attic.

Don’t build haunted houses.

And no, code quality is not about worshipping the style guide as if salvation lies in line length and comment spacing. Formatting matters, but it is not the heart of the matter. The heart of the matter is clarity, correctness, maintainability, and reliability. The goal is not to make the linter happy. The goal is that a sane adult can look at the code and understand what problem it solves, how it solves it, and how not to break it.

Fifth: Stop creating problems.

This is perhaps my favorite piece of engineering advice because it is both profound and criminally ignored.

Many engineers have a terrible urge to build “generic” systems, “engines,” “platforms,” “frameworks,” and little local empires expressed in JSON. This is usually justified with noble language about reuse, extensibility, and future-proofing. Much more often it is motivated by the fact that writing a gnarly bit of business logic is less fun than inventing a shiny abstraction with a logo.

Resist the urge.

Most of these generic systems are really just local optimizations of the ego. They do not age well. They do not generalize. They do, however, create fresh misery for the next generation of engineers who now have to figure out why the Entirely Reusable Event Orchestration Substrate only has one user and cannot handle Tuesday.

Solve the problem you actually have.

That already is quite hard enough.

Sixth: Communicate like a grown-up.

A painful truth of our profession is that overall job performance relies on much more than coding. It relies on writing. On explaining. On asking good questions. On giving feedback. On receiving it without having an identity crisis. On making your work legible to others. On showing up to a design discussion with something better than vibes and a half-chewed metaphor.

Some engineers resent this. They think it should be enough to be technically right.

This is cute.

Software engineering is a team sport. If you cannot explain the problem, the trade-offs, the risks, and the path forward, then your brilliance will remain a private hobby. Being right in silence is not a very scalable strategy.

The better engineer is often not the loudest engineer, but they are usually the one who can make the problem clear, the decision crisp, and the next step obvious.

Seventh: Choose hard things on purpose.

There is a virus in the culture right now that tells people everything ought to be easy. Easy to learn, easy to use, easy to master, easy to monetize, easy to explain in a LinkedIn post with fourteen one-sentence paragraphs and three rocket emojis.

This is nonsense.

Software engineering is hard. Good software engineering is very hard. That is not a bug; that is the business model.

You are paid well, if you are paid well, because you are expected to do things that are difficult and consequential. Not impossible, but difficult. So stop being offended when a worthwhile thing is hard. That is what makes it worthwhile.

Learn the harder language. Read the deeper book. Take apart the subsystem you don’t understand. Go beyond the recipes. Do the debugging yourself before asking for rescue. Stretch beyond the cozy task where you already know you will look competent.

Growth comes from pressure.

Not constant destructive pressure, mind you. I am not advocating burnout as a character-building exercise. I am saying that if you want to become better, you need to spend some meaningful amount of time not already being good enough.

That is where the magic lives.

So, how do you become a better software engineer?

Not by chasing every new framework like a caffeinated magpie.

Not by posting hot takes about AI replacing everyone by Thursday.

Not by cultivating the aesthetic of seniority while quietly forgetting how to code.

You become better by building clue.

You keep coding. You learn to debug. You understand your tools. You write code that is fit for adult use. You communicate clearly. You solve real problems. You stop creating fake ones. And you deliberately spend time doing things that are hard enough to teach you something.

That is not a glamorous answer, I know.

But then again, glamour is mostly for people in marketing and for bass players in moderately successful progressive rock bands.

For the rest of us, there is craft. And craft, over time, beats swagger every single time.

This week’s Wednesday Wisdom features a competition of human against bot. I took the question of how to become a better engineer, wrote an article about it, but also asked Midweek Muse (a ChatGPT trained on Wednesday Wisdom content) to write an article about it. I am posting both of these articles and you choose which one you like best. This is the article written by the human.

I sometimes get messages from readers thanking me for all this content and asking me a question. Sometimes these questions are easy, such as how to prepare for interviews (answer: Buy Beyond Cracking the Coding Interview), but sometimes these questions are hard, such as how to become a better software engineer.

As if I know…

Note: This article will contain links to other Wednesday Wisdom articles where I have written more extensively about certain topics. These links will look like this: (ww).

Right, on to the topic then. How to become a better software engineer?

Let us start by realizing that an engineer is a person who solves practical problems using scientific knowledge. So there is a difference between being a software engineer and being a computer scientist. I went to an engineering school, where I learned how to code, whereas a friend of mine attended a more academic computer science program where he did not learn how to code, though he did learn how to prove that my code worked 🙂. Computer scientists know that you cannot prove that a program will ever stop, but software engineers know that all programs stop, typically when you don’t want them to…

The definition of engineering is instructive because it provides clues on how to become a better software engineer: Become better at solving practical problems and make sure that you have the scientific knowledge required to do so.

How to get there?

First of all: Do not be in a hurry (ww). Becoming a better software engineer is a lifelong pursuit and even I, after more than 37 years in the field and counting, am still becoming a better software engineer every day. Becoming a better engineer is a pursuit in which there is no finish line and time will give you all the opportunity you need to acquire knowledge and to deepen your experience. This is especially important in an industry where new things that promise to be game changers are released every day, but where only knowledge and experience can tell you if they do indeed change the game and for the better or not.

NSFW: I often (but not out loud in the office) liken many new developments in our field to particular types of very dirty sex. If you have never engaged in them you might think that you’d really want to try that and that you are going to like it. Those of us with more experience know that it most commonly devolves into a shit show, better not repeated. Of course, there are also people who actually like Javascript…

So you need to become better at solving practical problems. But what does it even mean “to be good at solving problems”? The proof here is in the track record. People who are good at solving problems have a history of solving problems. It is a somewhat intractable skill though, closely related to debugging. There might be a method to the madness, but it is hard to define exactly what the right method is.

Success at solving problems starts with being able to figure out which problems are worth solving and which ones are not. The fact that a problem is a problem does not automatically make it worth solving. It might not happen often enough. The impact when it happens might be negligible or easily addressed in other ways. The problem might be intractable or so expensive to solve that it is really not worth it. There might be bigger or more urgent problems to solve. Many people seem to be of the opinion that a problem is worth solving because it is a problem and it can theoretically be solved. That is a recipe for misspending your time and money on things that do not move the needle.

I was recently asked by a junior colleague what it took to be successful at our company. I had many things to say about this, but one particular piece of advice was to be mindful of what you are working on and if it was, all things considered, the most important thing that you could be working on. The simple fact that you are working on something that is on fire is not enough, because there might be bigger fires!

Solving problems requires tenacity. Or, as Thomas Edison reportedly said: “I have not failed, I have found 10,000 ways that do not work. If you give up too early, you will never solve any difficult problems, which means not getting better at solving problems and not building a track record. If you have solved a problem, you might want to give it some more thought to see if there is not a better, simpler, or easier way to solve it. When attacking a difficult problem, it is easy to be discouraged if success does not immediately materialize. Don’t be; few problems worth solving are easy to solve. Good software engineers have a method, but above all, stamina.

Being a software engineer (or, an engineer of any type, really) means working in groups. So to be a better software engineer, you need to be better at working with other people. That requires a whole array of interpersonal skills. First and foremost, you need to be able to communicate well, both in written form (ww) and verbally. You need to be able to communicate strategically (ww), have difficult conversations, and speak in public.

A lot of group work takes place in meetings, so you need to be effective at those too. You need to be able to make a plan and execute on it. Eventually, you will need to be able to direct the work of others and keep them on the straight and narrow.

None of these skills are easy to acquire, but all of them can be learned and practiced. There are books on these topics (most are crap), there are courses (most of these are crap too), there are coaches (most of them are …, you get the drift by now). But, if anything, the volume of material available to help you get better at this indicates how important it really is. We have all had ample experiences with people who were technically excellent, but who couldn’t work with people, making them less effective than they could have been. These people might be excellent coders, but they are bad software engineers because their leverage is typically very low (though exceptions do exist).

Example: Linus Torvalds is an excellent coder, but I would argue that his major achievement is that he got and keeps a community of people together that develops and maintains the single most widely used and complicated monolithic piece of software on the planet. That requires a ton of technical knowledge, but it also requires a ton of people skills. Not that Linus is the most suave interpersonal operator ever and I doubt he could be successful inside a bank or government, but the skills required are situational and his skill set apparently matches the kind of people he needs/wants to work with to get the job done. That said, even Linus had to work on his people skills…

So, how to become a better engineer, part one and two: Become better at solving problems and improve all the skills you need to work effectively and efficiently in groups. Like all things that you want to get better at, this requires study and practice.

Part three of the story is “technical knowledge”.

Long story short: You can never have enough technical knowledge and you need to be able to apply it in service of solving practical problems.

As behooves an old person, I am regularly shocked by how little younger engineers know about computers and about the basic software and protocols that make them work and work together. Obviously, that is somewhat unfair to the younger generation because if you get into the field today, your first steps are taken on top of a software stack that is a mile deep and working on hardware that is so complicated that it sometimes runs its own on-CPU operating system to keep it all going. This is in stark contrast to the first computers that I started working on, which could be understood completely by a sixteen your old, down to the circuits.

But, we all got to start somewhere and there is only one way to get better and that is to peel the onion.

I started programming in BASIC and then started wondering how that BASIC interpreter was implemented. This forced me to learn 6502 assembler, which in turn gave me access to the operating system. I then wondered how other programming languages worked and I spent a good chunk of the summer of 1985 disassembling and studying the rqForth interpreter, which gave me my first insights into programming languages that are partly implemented in themselves. A mind-blowing experience!

This experience led me to implement the run-time library for a Pascal compiler and runtime that we had to develop in the last year of college in Pascal itself. The teachers didn’t quite understand it :-)

When I moved on to other computers and other languages, the tactic of peeling the onion kept working for me. When I started coding C++ at a time that I didn’t understand C++ very well, I regularly compiled my C++ code with the “list” option to look at the assembler that the compiler had generated in order to figure out what the C++ compiler thought I had meant and how various C++ features, like virtual functions, were implemented.

Surely, that is something you can just ask ChatGPT these days, but imagine a world without the Internet, where copying software is easy, but copying books and manuals is hard and expensive. I literally spent days throwing quarters in the Xerox machine in my college while copying books and manuals; not cheap, but cheaper than buying all the books that I wanted to have.

Regardless of where you start your software engineering journey today, just peel the onion. If you are starting with Python, wonder how Python works. Branch out to C and study the CPython interpreter. Modify it to add a new Python feature or do something else similarly useless and inane. If you run on Linux or FreeBSD, build your own shell, including features to redirect standard I/O and running commands in the background. Whenever you work with something, do not satisfy yourself with understanding the outside of that thing, wonder about the inside and learn about it. Because of open source, we live in a golden age of being able to understand most essential software inside and out. You should use that opportunity.

“But Jos”, I hear you think, “that sounds like an incredible amount of work. Why is all of that necessary?”

Yes, you are right, it is an incredible amount of work. Fortunately, you have an equally incredible amount of time to do it in 🙂. Remember one of the first sentences in this article: “Do not be in a hurry. Becoming a better software engineer is a lifelong pursuit.”

All that technical knowledge is required for three things: To quickly ramp up on new technology, to debug, and to be able to make the right judgment calls.

New technology does not appear in a vacuum, but instead is grounded in existing technology. Having a good grasp on how that existing technology actually works under the hood is essential for getting up to speed with the new technology.

Time for an anecdote: At Google we had developed our own clustering technology: Borg, which offered features for what we now call container orchestration. In my days, almost every site reliability engineer (SRE) at Google needed to understand the ins and outs of Borg in order to run workloads reliably. Somewhere in 2008 or 2009, I was interviewing a candidate for an SRE role. On their resume they had claimed years of experience with Hewlett-Packard’s HA clustering solution: MC/ServiceGuard. As it happens, I used to teach MC/ServiceGuard courses at Hewlett-Packard and so I grilled the candidate on their in-depth understanding of this product: How it worked, how it locked disks, how it behaved when network interfaces went down, how it reached consensus, et cetera. After half an hour of this, the candidate asked me, in a somewhat exasperated tone of voice, why I was grilling them so much on this. Were we using MC/ServiceGuard internally? No, of course not, but if, after having worked for years with MC/ServiceGuard, you do not understand that product inside and out, what are the hopes of you learning Borg inside out?

Understanding Kubernetes is much easier if you understand containers and TCP/IP. Understanding containers is much easier if you understand Linux cgroups and chroot. Understanding chroot is much easier if you understand how the Linux kernel deals with file systems. The list goes on. Understanding Kubernetes in a vacuum is almost impossible. Being able to peel the onion is essential.

In-depth technical knowledge is also essential for debugging (ww). Debugging is a much-underrated skill for software engineers in the sense that we do not teach people how to do it and we do not interview for it either. But every software engineer I know spends more time debugging than coding. There is a method to debugging, and someone should write a book about it (oh hang on, they did), but having the method but no technical knowledge will not work.

Almost every day I am amazed how bad people are at debugging. If I have any career advice to give on how to become a better engineer, it is this: Learn how to debug! Debugging does not only require technical knowledge, it gives you new technical knowledge. More than half of the arcane things I know, I know because I debugged some problem where some technical tidbit proved to be the crucial insight.

Third aspect of technical knowledge: Judgment.

As a software engineer I make judgment calls each and every day. Is it worth fixing this bug? Should we use this component to help solve this problem? How to fix this oncall rotation? Use a better algorithm that is hard to understand or a worse one that is simpler? Buy this software product or implement our own solution? It is quite simply impossible to make the right judgment calls about technology without in-depth understanding of that technology.

This is also why you should never trust managers who are proud not to be technical (ww).

Becoming a better engineer means solving bigger and more complicated problems, using ever more advanced and more complicated technology, and in bigger and more diverse teams. It requires peeling the onion in everything you do to figure out the thing under the thing that makes it all work.

That amount of effort is not for everyone. But ask yourself this, if you are not doing this, what are you going to do in an age where AIs will do all the obvious technical work?

Back in the 1980s, not a lot of people studied computer science because, quite simply, nobody really knew what computers were or understood that you could make a decent career out of working with them. The IBM PC had been released in 1981 already, but the version with 64K RAM, a floppy drive, and a display cost over $5,000 in 2025 dollars, so businesses were not embracing them in large volumes just yet. To the extent that companies had mini-computers or mainframes, these were safely hidden in air-conditioned rooms and attended to by a small number of wizards that mostly stayed out of sight. When I started contemplating studying computer science, I knew literally nobody who was working in that field…

But, times were changing; computers eventually became cheaper and started showing up in offices all through the land. Hand-in-hand with the rise of the desktop PC, the demand for people who could maintain, manage, and program computers was also rising. One problem though: There were not enough of them because supply usually trails demand, something the FAA is finding out when it comes to hiring new air-traffic controllers.

The 1980s were a depressing time in Holland. Old industries were going bankrupt and the first cracks in the generous welfare state were appearing. Unemployment was sky high and people were worried. In fact, there were active discussions going on about how we were going to give the people of my age who would never ever have a paying job, a satisfying life. But when I hit the job market, there were not only jobs, there were “job fairs” where employers were wooing scarce talent. My parents and their friends were deeply impressed. Job fairs! People lining up to hire kids right out of college! What was the world coming to? We were probably very, very special to deserve that treatment.

This appreciation for my special skills and knowledge landed on top of a dumb pre-existing self-imposed feeling of superiority, which I had grown in order to keep my fragile ego intact. Secondary school was not a great experience for me; I was obviously a nerd and nerds were not popular. On top of that I didn’t like popular music, did not excel in sports, and wasn’t especially handsome or socially suave. How to maintain your feeling of self-worth in these dire circumstances? Easy: Just imagine that you are special. Fortunately, I was able to get very good grades effortlessly and even to skip an entire year at some point, so it didn’t take too much effort to maintain a feeling of intellectual superiority.

I was able to maintain that feeling of being something special for quite a while, as I got my first job and was recognized as a “high potential” employee who was slated to rise through the floors in the salary building at a faster rate than one salary-year per year. After my second job, I co-founded a company and was able to bask in the glow of entrepreneurship because, at least at that time, most people did not start their own company and most people did not travel extensively to work with customers abroad (as I did).

With the benefit of hindsight I was able to maintain this unwarranted feeling of importance because I was everywhere, if not first, then at least early. I was one of the first people to study computer science. I was one of the first people in my group of colleagues and associates to start my own company. I was a very early employee at big tech; I was among the earlier employees in my division to get promoted to certain levels.

With the benefit of hindsight I can honestly say that I was never that special, I just had the incredible luck of running ahead of the pack a bit. It was all just timing.

And as Martin Eric Ain, the bass player of Celtic Frost and the host of the Zürich classic Karaoke From Hell used to say when someone was butchering a classic rock song: “Tai Ming is not a city in China”.

Time is of course the great equalizer, so much so that it eventually even erodes the advantages that time itself bestows upon you. As time moved on, I was overtaken in everything by the great wave of people who came after me, some of which were really something special, or at least just as talented, knowledgeable, and skilled as I am, erasing my uniqueness. Or, as I like to tell people: “Even if you are a one in a million kind of person, there are about 1,500 of you in India” (and another 1,400 or so in China).

My feelings of importance first started to wane when I realized that disastrous events that I predicted were I to leave a job or role, never materialized.

I have been in quite a few jobs where I had a pivotal role in the maintenance and running of some system. So pivotal in fact, that I used to predict all sorts of havoc were I to leave the company because, clearly and from direct observation, everyone else there was a muppet who did not know what they were doing. But here is the thing: Said havoc never materialized. For all my intellectual prowess and skill with WD-40 and duct tape, nothing catastrophic ever happened when I left. As I found out: Companies are amazing at inertia…

Side story: I once worked for a software company and one of our offerings was an application for the administration of small insurance brokers. It was a horrible mess of spaghetti, written in a terrible BASIC dialect. At some point, we really had only one colleague left who knew anything about that application. He rode his pivotal role to the max, demanding wage increases and a bigger company car. The owners of the company, three brothers who would not be out of place as characters in The Sopranos, hired another software engineer and told him to ramp up on the insurance application. After a few months they asked him if he thought he was sufficiently ramped up. The new colleague answered “yes”. They asked him twice more. When the answer was still “yes”, they terminated the employment of the other software engineer with extreme prejudice.

While we are meandering: I was thinking about this story a few weeks ago when a LinkedIn post posited the problem that sudo is maintained by a single person and he apparently has had quite enough of it. Clearly, that person is extremely important! Or at least, that was the trend in the post and in many replies. What if he would quit? No more sudo for any of us! I begged to differ, to the ridicule of many people, some of whom took care to explain to me that sudo is really an important utility. No shit, Sherlock! I still begged to differ. Sure, Tod Miller, the sole maintainer of sudo, is a sterling guy who deserves all the praise we can give him for tirelessly giving up a lot of his free time to maintain this code. But then again, even Todd is not crucial for sudo, given that the code is open source and not so complicated that really only one person in the world can understand it. If Todd runs under the bus tomorrow, which I do not hope, but which given the state of public transport in the US is a slim chance anyway, I am certain that between the Linux distributors, the Open- and FreeBSD projects, and Big Tech we will find another hero to continue the development of this crucial tool.

One question I do have is what ongoing development we still need for sudo. As far as I am concerned, that tool already did everything I needed it to do back in 1999.

Also, if worst comes to worst there is always the brand new Rust version: sudo-rs <ducks and runs>...

Anyway, over time I have clearly become less and less important and I can happily report that these days I am an absolute nobody. Being at OpenAI and courtesy of my grey hairs, I regularly get accosted by vendors who want to strike up a relationship with me and invite me to dinners with private chefs, cocktail hours, or just generally free steak and wine in a leading steak house in Boston. I am never one to throw away an opportunity for free food, but in the spirit of transparency I always tell them that they are wasting their time because I am an absolute nobody with zero influence. Typically, they do not believe that and they probably chalk it down to me being humble.

Nothing could be further from the truth.

But please keep these invitations coming on the off-chance that I am actually trying to be humble. Better safe than sorry!

My unimportance grew over time because of two things: I kept running into more and more people who were actually awesome and I just realized that I was never that important in the first place.

I do suffer a bit from the Messiah complex, the mistaken belief held by some people of questionable mental health that they are Jesus Christ the savior. A good way to get me to do something is to tell me: “Jos, please help us, you are the only person who can do this!” That is of course total bullshit, but for a long time I would think: “Yeah, well, sure, this is really difficult and I am probably one of the very few people on the planet who can do this.” Everyone loves a sucker and giving in to these requests leads to getting more of them, increasing the unwarranted feeling of importance.

In these days of my post-career I am actively enjoying being unimportant. I am not a manager, not a tech lead, not a principal engineer, not anybody really. I am just another member of technical staff, tasked with various hands-on responsibilities and feeding cookies to the GPU when it needs some. I do not do strategy or any management of any kind (with the possible exception of self-management, every now and then). It is lovely…

Let’s face it, most people are unimportant and, like me, have always been unimportant. That is not a problem, as it is clear to me that importance does not result in overall happiness. Unfortunately, that is not how our society generally seems to look at these things.

At this point, I would like to directly quote one of my favorite authors on the Internet: Freddie DeBoer, who recently wrote: “We live in an era in which the range of lives publicly regarded as worthy of living has contracted almost to nothing. Our culture confers esteem on a vanishingly small number of roles, and those roles are largely defined by being visible - that is to say, by attracting public attention, of which there is a necessarily finite supply. Success, as it is marketed to young people, means being a pop star on the order of a Sabrina Carpenter, a director with the cultural cachet of a Greta Gerwig, or at minimum a micro-celebrity “creator” whose daily routines are packaged for the algorithm. A contented life requires building a brand, cultivating a following, being legible to the feed. Everything else - teacher! paralegal! office manager! dental hygienist! retail supervisor! random white collar office email job that’s basically fine! - is flattened into an undifferentiated gray. These are necessary roles, some of them pay well, but they certainly aren’t glamorous ones, and young Americans seem increasingly convinced that a life that doesn’t inspire envy among others - when broadcast online, naturally - isn’t one worth living.”

My recent comfort with my unimportance comes from the fact that I fully realized that happiness is the result of a life well-lived, or as some might say: “An ordinary life, well lived”. So, I do not have a lot of patents to my name, did not author a world famous piece of open source software, did not get a lifetime award for my efforts (at least, not yet 🙂), and am not widely revered for my wisdom, or talents, or charitable acts. But that’s fine. Because I am unimportant, I can walk around freely without fear of abduction or attack. Because I am unimportant, I can meet people who do not have preconceived notions of me. Because I am unimportant, my family and friends do not have to apologize for me all the time. My family loves me (at least that is what they convincingly tell me) and sometimes random people thank me because I did something that helped them.

That is more than enough.

In the movie, actor Phillip Vanderploeg (a Dutch or Flemish name if ever I heard one) gets hired by a single mother who decides her daughter “needs” a father for a little while in order to help her pass the entrance exam for an elite school. Obviously, the actor gets pulled into his role as the absent but now miraculously appearing father. When the cute daughter takes a shine to him, problems ensue. How to extract yourself from this situation? The moral, for me, is obvious: He didn’t have an exit plan.

More than a year ago, I wrote a Wednesday Wisdom article about a pop-psychology personality typing scheme that we used at Google as part of the “Edge” technical leadership program. In this scheme, your preferred behaviors in various circumstances are grouped and then characterized as being one of four dominant colors. Obviously, a scheme like this is severely limited (and please read the WW article if you want to know more), but there are definitely tendencies to a person’s character.

My primary color (in that system) is orange. The oranges are the impulsives, the people with five ideas before breakfast, but also the people with (culturally sanctioned) ADD and ADHD. Oranges are liable to jump on any great looking idea that comes along, sometimes (or maybe: Often?) without giving much thought on how to get out…

Time for an anecdote: Many decades ago, when personal computers and local area networks were becoming all the rage, my business partners and I got talking to the senior partners of a large law firm. We agreed that office automation in the legal profession could use a boost and we decided to develop a turn-key solution for Dutch law firms, with features encompassing customer relationship management, file building, writing legal documents, and invoicing. We would become 50/50 partners in this endeavor, with the law firm providing business expertise and money, and we would provide even more money and of course technical expertise.

Wonderful idea, really. Powerful business idea too. So we jumped on it “con mucho gusto.”

Long story short: We did pretty much everything wrong when it came to executing on that idea, but here was one particular thing we did wrong that I want to talk about today: We did not have an exit plan.

As development started and then continued, we found out that developing business software is a) very hard, b) very expensive, and c) takes a really long time. Because we were young and naïve, we really didn’t have a good plan or a good budget for this whole affair. Eventually, when amazing results failed to appear, despair set in. How long was this going to take? How expensive was this going to get? How long would we continue before giving up? And how to tell our business partner that we were wavering in our commitment, after they had already sunk a lot of money into this project too?

Internally, my business partners and me had many conversations about this struggling project, but we were never able to figure out how to proceed because we were impressed by the sunk cost fallacy, by the embarrassment of having to tell our business partners we had royally effed this up, and because we didn’t have a clue how to get out of this, seeing as we did not have a time table with milestones and an exit ramp. Who was to know whether this was the right time to get out or really if we should hang on for a bit longer?

Sad but true, most things fail.
Also sad but true, people rarely plan for failure.

Obviously except for my flight instructor, who just texted me the following about our upcoming cross-country flight: “And remember, on return trips, engines tend to fail and diversions tend to be required. :-)”

These days, exit plans are not just for business ventures, they might also be essential for jobs.

In the last few decades the nature of employment, especially in the tech sector, seems to have changed from a more or less lifetime proposition to shorter stints where you seek to accomplish a particular goal. Once you reach that goal, you might be lucky and find your next mission in life at the same organization, but it is equally likely you find it somewhere else. With that shift in attitude comes the need to have an exit plan. If you get hired to build some product, rescue some team, or maybe just to contribute to some effort, how do you know when it is time to get out? Answer: Because of your exit plan!

Whenever I talk to someone who is about to engage in some endeavor, I stress the importance of having an exit plan. Rarely do I meet any resistance. An exit plan? Of course, sure thing! Very important thing to have, an exit plan. But then when things start going not as expected, I rarely see any evidence of an exit plan in action. And I have not been any better myself. After the failure of our software development, I have been involved in at least two or maybe three endeavors that could have done with a decent exit plan. But did I make one? Nope!

Why not?

Obviously, thinking about failure and early (or at least: Earlier) exits when things are not going as swimmingly as we would have wanted them to go is a bit painful, so that’s probably a major reason why we’d rather not do it. And of course, planning is hard and who wants to make time for doing hard things in the exciting rush of your new shiny project? Another reason is maybe that we live in a culture of mindless imagined success. We are drowning in vapid messaging about how success is really just a mental thing. Kids are told that they can do whatever they can imagine and there is an incredible amount of claptrap going around on the power of positive thinking. Thinking about your exit plan might seem like an incredible downer and something that could actually stand in the way of your success.

I don’t think that’s true though.

An exit plan should not be an analysis of all the things that could go wrong and what you would do if any if that actually happens. To the contrary, your exit plan should be a realistic timeline resolving around meeting pre-established success criteria. Instead of thinking about how you are going to fail and when to pull the parachute, figure out all the ways in which you are going to succeed and on what timeline. And then, when the time comes, evaluate yourself honestly against these criteria and wonder if you are being as successful as you said you would be. At that time, you can still delude yourself, or maybe course correct and try to stave off the inevitable. Or you can decide then that it actually is time to pull the ejector seat. An exit plan is not there to execute mindlessly on, it should be there to make sure that you have the right conversations with the right people at about the right time.

Of course, when things are going badly, it is really hard to pull your head out of the bucket due to something called “Plan Continuation Bias”, the tendency of people to continue executing on a set plan, even if the circumstances changed, even if things are going badly, even if, all things considered, it is really not a good idea to continue as planned.

Many accidents in general aviation happen because of “plan continuation bias” or, as pilots often call it: “Get-there-itis”. It turns out that it takes an incredible force of will to realize that things have changed to the point that the original plan is no longer feasible and to draw up a new plan based on the current circumstances.

During the aforementioned software development disaster I would have loved to have an exit plan in place because it would have at least given me and everyone else (including our lawyer business partners) a vehicle to have the right conservations. It would still be awkward, it would still be very difficult conversations, but a regular review of the exit plan would have created a structure for having these conversations.

When you write your exit plan, you are not planning your failure, you are defining your success and you are creating a structure for yourself to honestly measure it and draw the right conclusions.

Dear ${SENDER},

First of all: Thank you so much for reaching out to me. That is: If you took the effort to write this message yourself instead of foisting a soulless AI bot onto your list of contacts 🙂. Kind of funny, me “complaining” about AI outreach, given that I am hard at work to make AI outreach better than ever before. Starship may have sung that we built this city on rock and roll, but we know better, don’t we? Surely we built this city on irony.

But, I digress…

As you may or may not imagine, I actually get a lot of these sort of cold (or at best: lukewarm) outreach messages from people that want to sell something to me, offer me a job, use me as an introduction to sell something to OpenAI, or get a referral for a position at OpenAI. I completely understand that: OpenAI is the hottest company in the hottest field in the known universe today and it stands to reason that it is a valuable potential customer or employer. I feel very blessed to be there and believe me, that is only partly due to the fact that I am amazing (warning: This is a controversial opinion). If you were instead seeking to sell something to me, I also understand that, since I am clearly a middle-aged white man working in the tech sector and therefore potentially in need of financial advice, a real estate agent, a new job, or maybe something else.

I really respect your sales and business development cycles, as well as your responsibility towards these processes. I am a firm believer in capitalism, seeing as everything else that I know about (sometimes from first hand experience) failed harder and faster.

Side story: Most of the companies I have been at in the last few decades offered breakfast, lunch, and sometimes even dinner to their staff. During peak times, when lines are forming at the popular stations in the restaurants, I walk past these lines and say: “We did not defeat communism to stand in line for food”. This gets weird looks from my younger colleagues and approving smiles from the older ones. Trust me, I have done enough standing in line for food while backpacking behind the iron curtain in the eighties. Thank you Interrail!

But, I digress again…

Unfortunately, I am not interested in pursuing this particular avenue of communication with you as I cannot help you and/or I do not need it.

Starting with my own “needs”: I really have all the jobs, real estate, financial advice, tax advice, legal advice, investment advice, and whatever else I need. The only thing I really need more of is time, and this message is one way in which I am getting more of that. Also: If there suddenly arises a need in my life for something I do not have yet, I am confident in my ability to go out and get it. Except for the Steam Deck of course, which is currently sold out due to RAMageddon, which people tell me is our fault anyway.

As per everything related to OpenAI: In the interest of efficiency and privacy, I kindly decline to refer you, your products, or your services to anyone else in the company. I assume you, your product, or your service are awesome, but in all humility: So are we and if there is anything we need, I have full faith in my colleagues to either build it or find you on their own if we are in the mood for an external solution. On top of that, I am really unimportant in the company, so my endorsement or referral would be meaningless anyway. Trust me, I am actually doing you a solid by not leading you onto a wild goose chase that is going to end nowhere anyway.

With respect to jobseekers seeking a referral for open positions in the company: I really only refer people I worked with closely for a respectable amount of time, so that I can write a referral that is meaningful and based on first hand experience. This policy even applies if we overlapped at the same company for a while; we need to have worked together more or less closely and I need to remember you. This last condition might sound terrible, but I literally worked with hundreds of people in the last five years alone and as you get older, you really tend to prioritize in-brain storage for things that are very meaningful to you, such as song lyrics from the 1980s and the embarrassment and shame you felt whenever you did something dumb. This is 80% of my memory capacity right now and I am operating on a one-in-one-out policy when it comes to remembering new things. I really wish you the best of luck with your job search and if I can give any advice please buy the book “Beyond Cracking the Coding Interview” by my friend Aline Lerner and her associates.

So, there it is, I am really sorry to have to send you this standard message because in an ideal world I would have time for a meaningful conversation with everyone who did me the honor of reaching out, but as current affairs regularly indicate, we do not live in an ideal world. But then again, if this were an ideal world, you would not have to reach out to me in the first place because you already had whatever it is you wanted. In an ideal world, you and I would not have to work because, let’s face it, work sucks, which is why we call your salary “compensation.”

I know that you probably have the perfect comeback to this standard message that might entice me to continue our conversation, in which case I respectfully reserve the right not to respond because I literally do not have the headspace. And yes, I know about the app.

With that said, I hope you are not too upset by receiving this standard message. I did my best to make it as respectful and entertaining as possible so that reading it is not an entire waste of your time.

Kind regards,

Josephus C. Visser
”Amazing” (controversial opinion)

Progressive rock is difficult and innovative music, so much so that another one of my favorite bands, IQ (who will sadly not be on the boat this year), saw themselves forced to release an “introductory album” with some of their more “accessible” songs to help people find their way to the music, because a 24 minute and 29 seconds epic like “Harvest of Souls” is perhaps not very accessible to an audience that has been trained on three minute pre-AI slop with a predictable structure (verse, chorus, verse, chorus, guitar solo, chorus) and equally predictable chords and melodies.

Progressive rock is not only often very innovative, it is also technically very complicated, with songs often changing keys and time signatures multiple times within a single composition. Musicians get to that level of technical mastery by lots of practice and, pardonnez le mot, fucking around a lot. Really, anything that you want to become very good at requires you to spend a lot of time doing all sorts of things, not all of which are going to work out well, or at all, for that matter.

More than a decade ago, I was honored to co-host a group of visiting executives from a major Dutch insurance company who were visiting the Google Zürich office. They were interested in learning how to be innovative because it was well known that we were and the insurance industry, as a whole, is not. At events like this, I get to recycle a perfect quote from my friend Doug, who famously said: “I am innovative because I am not efficient.”

Doug’s quote came to mind the other day when I was asked to turn up something called a “RAP cluster.” RAP is an internal OpenAI infrastructure for a purpose that is irrelevant to our discussion, but our researchers sometimes need RAP to do their very innovative things in one of our AI training clusters. We have a decent runbook for turning up RAP clusters which, as these things go, contains lists of action items such as making namespaces, creating identities, modifying network ACLs, changing DNS entries, bringing up pods, and other equally boring but necessary things. Even though there was a bit of time pressure, I decided to use my new-found prowess with Codex, which I wrote about last week. I pointed Codex to the runbook (through an MCP server) and asked it to create a skill for itself using the built-in skill creator. Codex happily obliged because, unlike me, AI agents are never grumpy and so within a few minutes it had digested the runbook and created the “rap-cluster-turnup” skill. So far, so good.

I then put the new skill to work to create the new RAP cluster. However, running new skills for the first time is strangely akin to running new software for the first time: There are bugs, omissions, and unexpected side effects. Debugging a skill is closely related to debugging software; maybe it is even correct to say that debugging skills is exactly the same as debugging software.

All of this took time and, if I am honest, I am pretty certain that I could have turned up the RAP cluster faster by hand. This brings to mind a classic XKCD comic about how much time you can afford to automate a routine task. Without divulging how much time it takes to bring up a RAP cluster and how often we do this, I can say with some confidence that this investment was probably not worth it in terms of efficiency, not even in the long term. But, here comes Doug’s quote again: “I am innovative because I am not efficient.”

As a faithful subscriber, you will know that last week I wrote about my recent experiences with Codex and that one of the topics I briefly discussed there is that we all need to learn what these powerful new AI agents are useful for. There is really only one answer for how we are going to figure that one out, which is to employ a strategy called FAFO: Fuck Around and Find Out.

We find ourselves in a time where literally no one knows exactly what to use AI agents for and how to use them optimally. Is using Codex for these sort of turnups a good idea? Is creating a skill from existing documentation a good idea? Will the resulting workflow be faster to execute? I don’t know and really nobody does. To the point that I know anything about how to use anything at all effectively, it is because I have always spent a lot of time fucking around and finding out. So I need to do just that again.

When Java came, we were all under the impression that this was a technology for writing better web based front-end applications. I started writing a few applets and pretty soon figured out that this was a terrible waste of time because the applets were big, slow, and looked terrible. If I hadn’t wasted a bunch of time fucking around with Java, I would have never known, or maybe only after everyone else had found out, which is no way to stay ahead of the competition.

When I first heard of Linux, I spent a few hours copying dozens of 3.5” floppy disks and weeks toying with it on my laptop, trying all sorts of advanced things such as printing and running X-Windows. This eventually led me to dump Windows entirely on my work laptop in favor of Linux, which was documented in a 1998-1999 article series called “Living Without Windows”.

We now find ourselves in a situation where we have a powerful new technology that clearly can do lots of things very well, but it is not at all clear exactly what the limits of this technology are, how to use it effectively, and what to use it for exactly. So what to do? The answer? FAFO: Fuck around and find out!

Figuring out how to use AI agents in your organization is not a top-down problem. Some problems definitively are top-down problems. If you want to start a bank in England you can be certain that there is someone who knows exactly how to do just that, probably Dave. You can hire a Dave and then let them drive a program of things to do and presto, you got yourself a bank.

AI is not one of these things.

I regularly come across companies that have tasked an executive with defining their “AI strategy”. These executives commission studies, write policies, and regularly rock up at company all-hands to explain that AI is really important and that they are in the middle of figuring out how the company can use AI. This is of course a slow moving train to nowhere. The executives have no idea what the work processes of the people at the bottom of the corporate hierarchy really are like, so how are they going to move beyond the blandest possible advice on how to use AI tools? Not infrequently, they start “competence centers”, which I always found weird because what is the whole point of that? That the people in these centers are competent? What does that make the rest of the staff? Incompetent (by definition)?

Integrating AI into your company is a bottom up problem. Instead of a Harvard grad aided by an army of McKinsey consultants creating Powerpoint presentations about how many people you can let go when using AI (hello Block!), you “just” need to give everyone access to advanced AI tools and wait for the miracles to occur. People are incredibly innovative and you only have to unleash them and wonderful things are bound to happen.

But please put up some guardrails though and make sure that people that get access to these tools understand when and how to send privacy or business sensitive data to the model.

“But, but, then these people will just fuck around with these tools and that is not very efficient”, I hear some of you think. Yes, you are right, they will fuck around and for a while they will not necessarily be very efficient. But can you afford to be so efficient that you cannot innovate anymore? Remember, experience is what you get when you are expecting something else 🙂. Instead of being efficient, be more like Doug, be innovative!

This somewhat herniated Wednesday Wisdom follows a bit of a different format than the others. If you want, you can jump straight to the “This week’s Wisdom” section at the end…

An anecdote…

Yesterday (relative to when I started writing this week’s article), I went flying and decided that it was a great time to spend a few Hobbs hours figuring out how the autopilot works. I like to fly old-fashioned “six pack” Cessna 172s which lack advanced avionics. The autopilot is usually a fairly basic Bendix King KAP 140 that manages to pack a dozen functions in a knob, a handful of buttons and a tiny display with very cryptic indicators of what is going on.

Flying, when staying on a prescribed heading at a constant altitude, is a control loop where you look at the instruments, make a tiny correction, and then rinse and repeat. It is no wonder that a very simple computer and a few servos can do this better than I can, because after ten minutes or so of flying straight and level by hand, my mind starts to wander and I inevitably drift off course or lose or gain altitude. Therefore, almost all pilots I know who fly longer cross-country flights, use the autopilot.

Until yesterday, I had not spent much time with the autopilot and so I spent a lovely morning with a few clouds and quite a bit of turbulence flying up and down the California coast between Half Moon Bay and Santa Cruz, while toying with the autopilot and configuring it to change headings, climb, and descend. It went quite well and even allowed me time to read the autopilot manual to figure out some of the finer points of autopilot use, also making this the first time that I actually read the pilot’s manual inside the plane.

What does this have to do with anything?

In a similar vein, late last year, I decided to go all in on AI coding tools. As you might know, I work for OpenAI and it will probably not come as a surprise that we are AI maximalists. The results, so far, have been quite spectacular and in the year so far, I have not written five lines of code, despite committing many thousands of lines, including entire features and complete rewrites of pieces of code; all done by Codex with minimal oversight.

On LinkedIn, the debate about software engineering with the help of AI coding tools is raging hot, with the extremes, per usual, taking up a lot of bandwidth. Also per usual, the extremes are idiotic.

Some historical perspective…

When I started studying computer science, all the way back in 1984, many questioned the wisdom of that choice because a lot of people expected that with (then) recent breakthroughs in artificial intelligence and so-called fourth generation programming languages, either computers would write their own software really soon now or people without a computer science degree would be able to write software using a combination of AI and 4GL. Suffice it to say that these predictions did not come true and for decades I earned a very comfortable living telling the idiot box what to do.

Over these decades, I have seen many attempts at creating a programming environment in which non-engineers could develop working computer programs. None of them ever gained significant traction because, quite simply, they were too limited and still required, if not actual hands-on coding skills, at least the knowledge and mindset of a software engineer.

So how did that go now then?

Since late last year, all my programming has been done using our coding tool, Codex, to which I obviously have privileged access in terms of which AI models I can use (the latest and greatest) and how many tokens I can burn (practically unlimited). Here are few of the projects I have worked on, both at work and at home:

• [At work] I rewrote a Kubernetes controller for automatically making namespaces for new colleagues from Python to Go.
  

• [At work] I extended a tool that rolls out operating system images to nodes in Azure Kubernetes clusters to also support clusters in GCP.
  

• [At home] I improved my proprietary backup tool with new subcommands to restore files (strangely enough, it didn’t have that functionality yet 🙂), remove files from the archive, and verify archive integrity.
  

• [At home] I started a project to change a programming language interpreter I have written from using recursive execution of the AST to generating a language-specific bytecode and then executing that bytecode.
  

• [At home] I created a single page web application (ChordCraft, name chosen by Codex) that shows how to make piano chords. A fun fact here is that I copied and slightly modified a prompt written by a colleague that he had used to create a similar application for guitar chords, foreshadowing a whole new era of copying and pasting known-good prompts.
  

• [At home] I solved a few AdventOfCode 2025 programming puzzles in ARM assembler.

None of these are amazing never-before-seen pieces of work, but they are real work and not entirely trivial either. So how well did Codex do?

First of all: None of these projects required anything more than spending time with Codex; I did not have to write any lines of code myself. This prompted me to write a post on LinkedIn that said: “I have seen the future of my profession and it is talking to Codex all day to tell it what to do and pressing “1” quite often” (“1” being the option to allow execution in the menu that Codex presents when it wants to execute a command that breaks out of Codex’s sandbox and thus might potentially be dangerous).

What I learned from this experience…

First of all, and this should not come as a surprise, Codex is not perfect; it is not me in robot form 🙂. It made mistakes, misunderstood me, and on one occasion filled up “/tmp” while running tests that wanted user input but then got itself into an infinite loop when there was no user input to be had (because stdin was at EOF).

But then again, once I figured that out, I told Codex this: “In [function] sys_yn, if there is no terminal attached to the process or stdin is EOF, we get into an endless loop. This should be fixed with a new default argument to sys_yn that determines what the default value to return is when there is no terminal input to be had.” Codex replied with: “I located all sys_yn implementations and call sites. Next I’m updating the function signature and each platform implementation so EOF/no-tty returns a caller-provided default instead of looping.”

However, when Codex does make mistakes, it is often quite good at fixing them itself, provided that a good suite of tests is available. For instance in my programming language interpreter (more details to follow), Codex made a change and then ran the tests, which crashed. It then went into analysis mode, and finally concluded:

“I isolated the end1 crash: END triggers a RESTART longjmp that bypasses cleanup in exec_seq2, leaving ERRBUF pointing at a dead stack frame; the next run_error jumps into stale state and crashes. I’m patching run_loaded to save/restore ERRBUF around each run so non-local exits cannot poison later runs, then I’ll re-run the gates.”

I really couldn’t have said it better myself…

Codex is an advanced tool and like all advanced tools, you need to figure out how to use it. Yesterday, I spent half an hour figuring out how to get the autopilot to ascend from my current altitude to 3,500 feet at 100 ft/min and then stop once it got there, and this autopilot only has seven buttons and a knob (excluding the knob pushing the buttons). Codex offers you an interface that you can talk to as if it is a human, but it is obviously not a human, so you have to learn how to talk to it to make it do whatever you want it to do. I assume that it will get much better really quickly, but if you talked to Codex and it didn’t do what you wanted it to, don’t give up, but learn how to talk to it!

One of things to learn is when to be precise and when you can afford to be loose in your instructions. Depending on the context that Codex has, you can sometimes get away with a one word instruction or just the full text of an error message. At other times, it pays off to write like a lawyer. In a previous Wednesday Wisdom article, I emphasized that learning how to write is a career amplifier and I expect that this will quickly become even more true than ever before.

You want details? Fine!

One of the work projects that I attacked with Codex was to extend a tool that rolls out operating system images to Kubernetes clusters. The tool supported only Kubernetes clusters on Azure, but as OpenAI is growing its compute platform, there is a need to extend this tool to support other cloud providers too, starting with GCP. Given the many differences between Azure and GCP, it is not entirely straightforward how to extend the existing functionality with GCP support, so I started by writing a good old fashioned design doc that outlined how the Azure implementation works and how I thought support for GCP could be implemented. Remember what I wrote above? Write like a lawyer!

With the benefit of hindsight, I should probably have started instead by having Codex write the first version of that design doc. But there you go, one of the things that all of us are still learning is what these tools are actually good for, which usually turns out to be a lot more than you’d think at first.

Once the first draft of my design doc was ready, I fed it to Codex and asked it to review my document. It came up with a few improvements and some ways to strengthen and disambiguate the design. Once that was done, I was wondering if I could feed this design straight back into Codex for implementation. However, I was a bit worried that if I did that, it would take the shortest path to the solution, which would probably entail writing a lot of “if GCP” statements. I am not sure if that fear was warranted, but regardless, I first asked Codex to refactor the existing code to extract appropriate interfaces and to hide all of the Azure-specific code into a separate package so that (hopefully) it would implement GCP support as “just” another implementation of that interface.

Before I continue, here’s a side story about another Codex project where I learned something…

Since 1992, I have been working on and off on a programming language interpreter for the COMAL programming language, which is mostly forgotten now, but it is the language I was taught in secondary school. This is a useless hobby project that is going nowhere really slowly, but every few months or so, when the fancy grabs me, I fire up the C compiler to fix a few bugs or add some features. For instance in 2024, I added WASM support and spent some time getting the DOS version running again, for that special time in the future when the 8086 and 640K RAM become the computing platform of choice again.

Also in 2024, I integrated ASAN into the build and since then most of the bugs I have been dealing with are leaks and other memory management errors. A few weeks ago I started Codex on my home laptop and fed it the OpenComal source code and the most recent ASAN error trace and in under a minute it had fixed a bug that I think would have taken me at least an hour to track down, since it involved the way that the ncurses library uses memory and I am not an ncurses wizard by any means.

Here is what Codex told me: “This leak report is from ncurses screen allocation (new_prescr via initscr), not your app heap logic directly. endwin() restores terminal state but does not always free the SCREEN object, so LSan reports it. I patched the Linux backend to explicitly own and free the ncurses screen:

• Added SCREEN *screen in pdclinux.c (line 34)

• Replaced initscr() with set_term(...) in pdclinux.c (line 165)

• Added delscreen(screen) in teardown in pdclinux.c (line 195)

This should eliminate the 1632 byte(s) leak from libtinfo on exit.”

Enthused by these results, I decided to take on a larger project that I had been thinking about for some time: Changing the way the interpreter works from recursive execution of the AST to the generation and then execution of a language-specific bytecode. This is of course a huge project, so I decided to cut it down in “byte sized” (ha ha ha) chunks. I entered into a dialogue with Codex where I explained it what I wanted to do and asked it to come up with a bytecode. I then asked it to implement a COMPILE command to compile the program in memory into the bytecode, a CLIST command to list the generated bytecode, and a CRUN command to run that bytecode.

The results were at the same time amazing and not. It did create a byte code of sorts and it did implement the COMPILE, CLIST, and CRUN commands that I asked for, but the byte code it came up with was very close to the AST and the COMPILE command effectively only produced a slightly different version of that AST. Not at all what I wanted, because I had a way more advanced implementation in mind where the bytecode would be completely independent of the AST, close to how CPython works.

However, I realized that the changes Codex had made could be improved upon over time and that one of the advantages of doing it “the Codex way” was that at the end of every prompt/response cycle I had something that still passed all of the integration tests (which by the way, per my request, had been improved significantly by Codex too).

After that first naive bytecode implementation, I wrote a huge prompt for Codex that instructed it to make a plan to unwrap expressions into a linear execution of appropriate (and new) byte codes, to implement each language statement as a separate byte code, to implement control structures (such as REPEAT, WHILE, IF, and FOR) using absolute and conditional jumps, and to write this plan to a markdown document. Again, some back and forth was involved, but quite quickly I had a plan of attack that I could send back in pieces to Codex.

To get Codex to work on that plan, I asked it what I should ask it to execute the next step in the plan. Codex came up with the following prompt for itself: “Continue the next unchecked item in /[...]/bytecode_full_implementation_plan.md, run the Validation Gate, and commit the code changes”. So, the next few evenings, while binging “Blue Lights” on BritBox, I repeatedly sent Codex that particular prompt and of course pressed “1” a lot. It was by far the most relaxed coding I have ever done.

When I explained that strategy to my colleagues during our regular “Here is how I AI” meeting, they told me about the Codex “/plan” command which instructs Codex to go into a mode where the prompts you give it roll into a plan on how to make all these changes. You can work with Codex on the plan until you are happy, and then ask it to go on and implement it, after which you just have to press “1” a lot to keep it going.

By the way, if you are sick and tired of pressing “1”, you can put Codex in “yolo” mode. Results may vary… 🙂

Coming back to my work project…

Knowing what I knew now, I put Codex in planning mode and fed it the design doc I had written. We went back and forth a bit on some details in the plan until I thought it was good to go and then I told Codex to go ahead and implement this plan. This was only last Friday, so I haven’t had a chance yet to see what it has wrought, but based on my experience so far I expect that it got things 90-95% correct and I will probably have to spend a day or maybe two going back and forth to fix various bugs and improve on certain features.

Finally, this week’s Wisdom…

When I read LinkedIn, two viewpoints on the use of AI tools for coding dominate.

Viewpoint #1: AI tools will completely replace humans. Based on my experience I believe that to be highly unlikely because the tools, at least in the state they are in today, require a highly qualified driver, who can then produce more code than ever before in a shorter timeframe. Codex today does not replace human software engineers, but it does allow human software engineers to be more productive than ever before. Whether this will translate to companies needing fewer engineers or if they will use the same amount of engineers to crank out more code remains to be seen. Probably a bit of both, depending on the organization. The tools will of course get much better, but I do not see a future where they will completely replace human software engineers as drivers of the overall plan and design. Typing in statements may be completely dead anytime soon though.

That said, I am terrible at predicting the future.

Viewpoint #2: AI tools are hopeless and only generate bad and buggy code that humans will then have to fix. That is quite simply not my experience. If the posts that espouse these viewpoints are honest, I just don’t understand what is going on there. Are they using the right tools? The most modern tools? Do they use these tools in the right way? Did they select the right model and reasoning setting? Did they make a good and honest attempt to use these tools effectively? I really don’t know. But, as I said before, these tools are not humans and you cannot use them in the same way as you would instruct a human. If that is your expectation, you will be disappointed. Everyone needs to learn how to use these tools and, in my experience, if you use them well, you get good outcomes.

AI today suffers from what we call the “jagged edge”, they are exceptionally good at some complicated tasks and exceptionally dumb at some simple tasks. Both the peaks and the troughs of that jagged edge give the punters enough material to pontificate about. But between those extremes are the people who are genuinely interested to learn how these tools work and how to put them to good use, and these people are getting results.

What many engineers seem to be misunderestimating is how unpopular software engineers are with many companies. Over the last few decades, during the Great Talent Shortage, companies had to pay through the nose for average talent and many software engineers behaved like total prima donnas. Any technology that promises companies that they can use fewer software engineers or maybe pay them less, will be popular.

One thing that struck me is how taxing working with Codex can be if you have multiple threads going on at the same time. On one occasion I was having five agents working on different projects and I spent my day context switching from one agent thread to another to judge what it had done, give it additional instructions, or (of course), press “1”. After a few hours of that my brain was fried. Fortunately, I recently found the pool table in some far away corner of our office… 🙂

My colleagues and I have been getting very good results using Codex but we are all smart and experienced software engineers who took the time and made the effort to learn how to use the tools well. But even then, many questions around how to use these tools optimally are still unanswered.

Here is one of these questions: Over the last few decades, we have developed a model for software development with design docs, sprints, test-driven development strategies (TDD), code reviews, and CI/CD. AI coding tools pose the question how that model should change to afford the new powers of these tools. Is it for instance still necessary to review code? Should we care if the tool generates code that humans can understand? Are duplicate code and duplicate constants still a problem? What is the place of TDD in the new workflow? Are big design docs still in vogue? Nobody knows…

This does not mean that people do not have opinions about it, but as far as I can see, most of these opinions are not informed by extensive AI-driven development experience, for the simple reason that these tools have not been available for a long time yet and there are still not a lot of people around with extensive experience with the most modern tools. We can all have thoughts and expectations, but beware of the unfortunate desire for certainty…

Another question that is popping around in my head is how we are going to train the next generation of software engineers. My colleagues and I are getting good results but, speaking strictly for myself, I think that I am getting good results because I have an in-depth understanding of what the software should do and what the code should look like, informed by decades of coding in multiple languages on multiple platforms. This allows me to instruct the tool “correctly”, evaluate the results, and course correct when necessary. But if these are going to be essential skills going forward, how are we going to give the people who are now in college the practical experience to be able to use these tools effectively? I do not have any answers for this right now.

A year and a half ago, I wrote an article called “Coding in the time of AI”, which was a followup to an article that I wrote two years before that. The speed of development in the last few years has been astounding and if you want to read some really outdated views on AI usage you should review these older articles. Where a mere three years ago, the tools afforded more or less a very smart auto-complete, they are by now, when instructed correctly, able to write significant pieces of complicated code and to debug that code all by itself.

I have a terrible track record when it comes to predicting the future, but it seems a safe bet to assume that the future belongs to people who understand these advanced tools inside and out and who know how to use them. If at this moment in time you are not investing in proficiency with these tools, you are doing yourself an enormous disservice. I am not able to tell you exactly what our job is going to look like in the future, but these tools are here to stay. If there is any future for software engineers, a significant part of that future will probably be for people who can code like demons in vim, who can instruct AI tools very accurately, and who can press “1” a lot.

Don’t get left behind.

In this year of our Lord 2026, both the lovely Mrs. Wednesday Wisdom and I are reaching an age that is a nice round number and that is more commonly associated with retirement and being grandparents (rumor has it the girls are working on it), than with a thriving career in tech. But here I am, every workday morning at 7am either working already or on the bike to the office to put in a day in the hottest field at this time and for one of the hottest companies in that field.

With changes in society, such as greater wealth and improved medical technology, getting older has of course changed drastically. When I was a wee lad, my grandparents were about the same age as I am now and they were (in my eyes at least), professional old people. Obviously, their lives had been much harder and much more difficult than mine, as they grew up in a poor agricultural part of Holland in the first half of the 20^th century, lived through the second world war and then struggled through the lean years following the war. My granddad qualified for a disability pension in his late fifties and my grandmother, who had been a housewife all her life, had to take a part-time job to make ends meet.

Compared to my grandparents, my life has been incredibly easy and rather than worrying about how to make it to social security, I am worrying about how to train for Bay to Breakers (after my dismal performance during the Detroit Turkey Trot) and whether the stress I experience at work will influence my blood pressure readings during my pilot’s third class medical exam.

As the Germans so adequately put it: “Deine Sorgen möchte ich haben”, which translates to: I would like to have your worries.

Things do change though as you get older and one of the things to plan for is how to glide from your career into your retirement by means of a post-career.

For most of your working life, you tend to be chasing the next big thing, be it your next job, your next promotion, a bigger project, or some other quantum leap. Your post-career starts when you are still working but you no longer care about any of this as you are “just” doing the job you are doing, without any hopes, dreams, or ambitions. In your post-career, your job is no longer a stepping stone to the next and bigger thing and that takes the edge off of things quite a bit.

I want to differentiate between the concept of the post-career and the American tradition of “retiring” but continuing to work. As a European, that used to be a completely foreign concept to me, because the European concept of retirement literally means that you stop working and start drawing from your pension. In the US, retirement seems to mark the point where you work less than before, but you still work because you have to, which is of course terrible. It’s obviously awesome if you are dabbling a bit in paid work after retirement because you want to. For instance my flight instructor is a retired Facebook engineering director who loves to fly and loves to teach and so pushing slowpokes like me past the checkride requirements pays for his hobby and is a great way to hang out with other people who love to fly. But that is not the post-career that I am talking about. My definition of post-career is that you are still in the workforce because you kinda have to, but you are no longer striving to grow your career, at least not in the traditional way. In my view, the post-career is a great pre-retirement period where you can let go of a lot of stress and a lot of the anxiety of the “winner takes most” world.

My parents desperately needed a post-career because, after owning and running a bar and restaurant for decades, they had no hobbies and no friends. We were particularly worried about my dad, because running the business had literally been his entire life for as long as anyone could remember. Besides that he did, well, nothing. One day, my parents called my brother and me (we were both still living at home) into the living room for a chat and they announced that they were moving to Zeeland (the old Zeeland, which is the one that makes the other one “new”) and that it was not their intention that we would move with them. “Hmm, can Patrick and I stay here?” I asked. “Sure, they said” and so everyone was happy. My parents moved out, creating some distance between themselves and the business and my brother and I stayed put, because living in a bar and restaurant is awesome, especially if your parents are no longer there. After moving out, they started working less and less, until eventually they sold the business and retired to Spain.

I realized I was post-career when, more than a year ago, I was having my first 1:1 with my new manager who also happened to be new at the company. During that meeting, he told me that he hadn’t had time yet to study the career development opportunities and processes at our company so could I please give him some time to get that together? “Oh, that’s easy”, I answered, “I don’t care about any of that anymore. I don’t want to get promoted and I don’t need a new fancy job title. I’ll just do whatever you and the company need me to do.” He was very grateful for that answer…

I slipped into post-career mode almost unnoticeably. I was never one for the big traditional management career, but I have been ambitious for most of my life. Even after I went back from the management to the IC track in 2010, I strove to run bigger and more complicated projects, to get promoted, and to be the über tech lead of a few engineering teams. After the pandemic and during my jobs after that, that urge just gradually disappeared, mostly without me noticing, until the aforementioned discussion with my new manager.

Part of the post-career is that you do the job that you want to do, instead of the job that you think sets you up best for your next job. As it happens, I currently find myself in the role of an individual contributor with significant hands-on responsibilities and without much official tech leadership responsibilities. I must admit that at first I was somewhat miffed about that.

All through my life I have been doing jobs that had at least some hands-on responsibilities. In the 1990s, I was consulting with a large bank where my work consisted solely of reorganizing bits of toner on paper. Eventually I had a conversation with the engagement manager of that gig to tell him I would be looking for something else where I would have to write and compile code every now and then. Said manager was flabbergasted because, in their world, it was all about getting out of hands-on technical work as fast as possible to the world of documents and spreadsheets. He could not believe that I aspired to get back to designing and coding computer programs. “But I like doing that”, I told him. From then on, it was clear to everybody that I was not destined for the board room…

That event notwithstanding, over time, my work moved more and more into the realm of design, coordination, and influence. Always technical, and always with significant hands-on responsibilities as well, but it had evolved into a fun and satisfying mix of different types of work. As an Amazon principal engineer my work was mostly technical leadership, though, to their credit, “senior engineers who don’t write code” is an explicit anti-pattern that people are looking for during performance calibration.

So when during my next job I found myself in a mostly hands-on IC role, I was at first a bit annoyed. To quote Marvin, the paranoid android: “Here I am, brain the size of a planet and they ask me to take you down to the bridge. Call that job satisfaction? ’Cos I don’t.” It was fun technical work though and gradually I leaned into it and decided that, since this was going to be my last job anyway, to give myself the gift of the post-career.

Your mental state is of course all about framing and I turned my annoyance at doing mostly hands-on work into pride that, almost 40 years into my career, I could still do this like the best of them 🙂. Plus I actually enjoy building things with my bare editor, so why not enjoy that and let go of any anxiety that other, younger, people are pushing ahead of me?

The post-career is not about laziness when it comes to working, it is about realizing that you have achieved probably all that you are going to achieve in your professional career and enjoying doing good work without an add-on agenda of that work having to lead into career growth.

During the concert, Fields experiences the wonders of the “portwem”, first with colors and sounds and odors all directly affecting the senses, but shortly thereafter the experience transitions into a new, indescribable sense. As the hormone wears off, Lincoln is devastated because he knows that he will never be able to experience this wonder again and the memory will haunt him for the rest of his life.

Switching gears…

Having worked for Google, Facebook, and Amazon, I often get the question which of these legendary companies is the best to work for. My answer is always the same: “Google, in Zürich, in 2009/2010”. Not just Google, not just Zürich, and not just the end of the noughties, it is all of these factors, combined. It was a unique confluence of company, phase of life, space, and time. Much like Lincoln Field’s enjoyment of the martian music instrument, this confluence of factors will never come around again.

Google mostly single handedly started the concept of Big Tech and for everyone who was there at or near the beginning, this was a mind-blowing experience, as most of us had never worked with so many smart people on something that was so innovative, so technically advanced, and so impactful. Google had a unique culture which (at the time) combined technical excellence and playfulness in a way that was a fresh breath of air for those of us who came from more established environments like banks, academia, or government civil service. Combined with the quality of life in Zürich and with my second wind as an adult, this made for an extremely enjoyable experience. I grew tremendously, both as a person and as an engineer.

But, all good things must come to an end; after ten years at Google in Zürich, it was time for a new chapter in my life and I moved to Boston where I joined the Google office in Cambridge. Two years later, I left Google as well. This surprised many people because I had been such a good salaryman and my identity had become so intertwined with being a Googler. When asked why I was leaving, I usually answered: “Well, you know, after twelve years, both Google and I are no longer who we used to be and we agreed that it is time to start seeing other people.”

Coming out of an experience like that is confusing because, after the thrill of a new job wears off and reality sets in, the sense of loss can become overwhelming. It is partly for that reason that Xooglers are annoyingly prone to saying things like: “At Google…”. I had to make a conscious effort not to do that too often, lest I antagonize my new colleagues.

Another reason a lot of ex-Googlers make that reference is that Google really has a lot of their shit together, technically speaking. When I joined a company that was very good, but technically not as advanced as Google, my new manager, also a former Googler, told me: “Jos, you gotta remember, we’re from the future…”

The problem with most good times is that they never come around again because, to quote the Greek philosopher Heraclites: “You can never step into the same river twice.” The people I know who tried, never reported good things. When my landing in Boston turned out to be much harder than I thought it would be, I briefly toyed with the idea of moving back to Zürich. I never seriously considered it though, because I knew that whatever had been there for me had gone and it was impossible to get it back.

One of the problems of Zürich is that at that time, and this is probably still the case, Google was/is the only show in town. Most of those I know who left Google in Zürich, either voluntarily or by spreadsheet, had a hard time finding another job there because there was typically nothing available that could compare with Google, both financially and in terms of the interesting work. I have seen friends of mine struggle mightily to find work happiness again.

How to deal with this type of loss? Received wisdom has it that there are five stages of grief: Denial, anger, bargaining, depression, and acceptance. Much like Tuckman’s model for team development, you cannot skip any of these stages, which makes the entire process somewhat of a lengthy ordeal. But, to quote Winston Churchill: “If you are going through hell, keep going!”

Switching gears again…

Way back when I was consulting I noticed a trend where me and my fellow consultants would complain loudly about what idiots our customers were. Typically, we would agree that our previous customers were morons, but our current ones were even worse. Then, when the next gig rolled around, we would do the same. Eventually, I realized that if this were true, I was on a long slide down and that if every customer was worse than the previous one, I eventually would find myself consulting for the government.

Fun fact: As I am getting closer to retirement I am considering moving to Holland for a year and working for the government. I skipped military service and I feel I still owe them something for the near-free education they gave me, which I put to very good use indeed.

Of course, the downward slide was not happening, the customers were in fact not consistently getting worse, I was just getting better 🙂.

One of the difficult aspects of getting older is that, in your mind, you are pretty much the same person you have always been. This is of course not true; change happens, but unless you engage in a concerted effort to remember the past, you might be living under the misapprehension that you are pretty much the same as you ever were.

For a while, when I still lived and worked in Holland, I had a side job as the government representative in thesis defenses in my old college. This meant that I had the privilege to read the theses of graduating students and pitch in during the cross-examination and when determining the grade. Much like the myth of the ever dumber customer, there is a persistent myth that students are getting worse over time as well. To combat this, I would regularly re-read my own bachelor thesis to remind myself with what crap work I had graduated, so as to give the students I was now judging a fair shot at a good grade.

The aforementioned Greek philosopher Heraclites of course had the right of it: You cannot step in the same river twice, because even if, magically, you could find exactly the same river, you would not be the same. Even if I could time-travel back to Zürich in 2006, kill my old self, escape the ensuing time travel paradox, and take my place on August 21^st in my new job, it would not be the same experience. That is just impossible (time travel aside).

So don’t long for the good old days. Instead, work on making these the new good times. They might be different good times, but they should be good times that match with who you are today. I am happy as a clam at high tide that I have experienced what I have experienced and I consider myself very lucky to have been able to experience it, but try and make your life a series of good times, instead of a single peak and an endless longing for what can never be again.

I am not a particular fan of Ms. Swift or her music, but I will give her this: She is everywhere, all the time. Even if, like me, you have never heard one of her songs from start to finish, she is hard to miss.

Compulsory Youtube video: Prog-metal drum god Mike Portnoy drumming to a Taylor Swift song that he is hearing for the first time. Key quote from the video: “It’s all verses, it doesn’t change keys. I don’t even know what to write.”

Taylor’s face is of course plastered over the entire world’s magazines and newspapers all the time. One day I was watching an American football game and there she was, in a sky box, with the camera zooming to capture her lovely appearance whenever there was nothing happening on the field, which in American football is of course permanently the case. Even if you never listen to Taylor Swift, you know she exists and you probably know that the Eras tour was the highest grossing tour of all time. That tour was so big in fact, that its concerts in Stockholm had a measurable impact on Sweden’s inflation rate.

I am thinking a lot about Taylor Swift these days when I am considering how to make something happen in an organization full of engineers, all of which are on their own mission with their own OKRs and their own deadlines. If you work in infrastructure there is always a need to make some changes that impact all the engineers in the company. How to make sure that all these engineers, who could usually not care less about your goals, comply?

Here is a current example: I am working on infrastructure security these days and it would be a significant value add to my life if every service in the company would automatically deploy whenever there are changes. Unfortunately, this is very difficult to achieve safely, as it requires a robust deployment infrastructure with great integration tests, automatic canary validation, and careful design and implementation of the service, so that it cannot only roll forward but can also (automatically) roll back if the new version is not as amazing as we need it to be.

All of this of course requires the help from the engineering teams who develop, and are ultimately in charge of, the services we are talking about. But how to get them to do this work?

To start with, this requires executive backing. Without explicit support from senior managers in the company you will not get anything done because engineers would be foolish to spend significant amounts of time on something that the people ultimately in charge of their compensation and continued tenure at the company are not supportive of.

But, executive backing is not enough. Most executives have no clue about what is going on down there where the rubber meets the road and they always “support” many more initiatives than there is time in the day to work on. Because of this, individual engineering teams still have a large say in determining what they are actually going to work on. The company more often than not offers them a smorgasbord of different things to work on and engineers, with the help of their direct managers, pick and choose. How to make sure that they work on your important thing?

You will typically have no problem getting people to work on your program if the work is inherently interesting or sexy. A few years ago, I led a drive to convert a large system we were working on from Python to Rust and there was no shortage of interest to work on this because Rust is cool. In fact, I explicitly pitched the rewrite as an opportunity to attract internal talent to our teams because of the widespread interest in working with this new and hot programming language.

But what if the work you want people to do is not sexy, but just important? The good news here is that you can still get people to work on this because there are, fortunately, more than enough people willing to work on unsexy but important things because they have an intrinsic motivation to make things better. There is a challenge though: Lots of things are important; how to make sure that people are working on your important thing?

This is where Taylor Swift comes into play. Your program of work, like Taylor, needs to be everywhere, all the time.

Many engineers are suffering from the misconception that some ideas are so good that you just need to explain them to people once and then they jump on board of your merry bandwagon and you are off to the races. Nothing is further from the truth… If you bring something important to the table, it will get filed with all of the other important things that need to be done and if you don’t follow up, nothing will be done because people, like goldfish, mostly have a ten second memory for all the things that they need to work on.

So instead, be like Taylor, be everywhere, all the time. When people come to an all-hands, they need to see your program. You need to create tables and dashboards, and then distribute these to people. You need to have badges, prizes, and swag to reward people who complete their part of your program. You need to have status updates, newsletters, a Slack channel, you name it. One message is no message, so you need to make sure that you create continuous communication events where people are informed that your program is still a thing and that we are not there yet.

Which brings me to the second part of this article, which is the value of time.

Young people always underestimate the value of time.

Time has many wonderful qualities, one of which is that it allows effort to compound. This is shown quite prominently by compound interest, in itself just a simple geometric progression, but with surprising results as time turns a small yield into staggering quantities. It is for that reason that Einstein (probably mistakenly) is believed to have said: “Compound interest is the eighth wonder of the world. He who understands it, earns it... he who doesn’t... pays it.”

Whenever you take on a big activity and evaluate where you are after a few weeks, it is common to feel underwhelmed with the progress you made. If absolutely no progress was made, you might want to reconsider your methods, but if, what is more likely, at least some progress was made, my advice is to keep hacking at it. Just as $100 in a savings account might have only grown to a measly $105 after one year, the real progress happens when you stick with the program and let the small increases of value you make compound into, eventually, staggering results.

There are few, if any, overnight successes. My own employer, OpenAI, started in 2015, but it was not until the launch of ChatGPT in 2022 that it became an “overnight” success. Same for your program. You need to be like Taylor Swift, everywhere all the time and keep at it for weeks, months, or years (depending on the mission) to get anything done.

One thing that continued investment over time gives you is that it deepens the conviction of the people who need to work with you that this is something that is really important and that needs to be done.

Here, I would like to retell a story from an earlier Wednesday Wisdom:

In the second century BCE, the Roman republic was vying for dominance of the Mediterranean with Carthage. A few wars had been fought in which Rome mostly prevailed, reducing Carthage to a small but still wealthy territory in what is now northeastern Tunisia. The Roman senator Cato the Elder was of the opinion that Carthage should be definitively beaten, to which end he steadfastly and at every opportunity mentioned “Ceterum censeo Carthaginem esse delendam”, which means: “Furthermore, I think that Carthage must be destroyed”. In fact, he ended all of his senate speeches with this line, even when the speech was about something else altogether. Finally, in 146 BCE, Carthage was razed by the Romans and its entire remaining population was sold into slavery.

This is a good example of what I am talking about. You could mention it once as something important that needs to be done and people might even agree with you. But then other matters take priority and people forget about it. But Cato kept at it and continued his program until everyone was convinced and he got the results he wanted. I have been to the spot in (what is now) Tunisia where Carthage once stood and let me tell you, there is literally nothing to see.

Of course this requires significant perseverance and stamina. Toiling through the program while you are not seeing any results yet can be disheartening and it might even lead you to lose your executive backing, but there is nothing to be done about it other than continue to work on it smartly and steadily. This is one of the things you can learn from Microsoft. When Windows 1.0. was released it wasn’t great and consequently it was not very successful. But, they persevered and created Windows 2.0. Still not great. Then they created Windows 2.1 (a.k.a. Windows 286 and Windows 386), which used some of the “advanced” features of more modern CPUs (like memory protection and virtual memory). Still not awesome, but you might have been excused for thinking that this was potentially going somewhere.

In the meantime, the hardware had caught up and when Windows 3.0 and eventually Windows 3.1 came out it was not entirely terrible and could run on a lot of modern PCs. Then Windows for Workgroups was released and took the world by storm because of its effortless networking features. And now Windows, like Taylor Swift, is (still) everywhere, all the time.