Domain Specific Terribleness

We keep shooting ourselves in the foot with terrible domain specific languages

(Like this article? Read more Wednesday Wisdom!)

This week’s article is about the horrible mistakes we have made while we sought to automate manual work, externalize configurations, and prevent repetitive work. In other words, it is about something that I think is among our industry’s worst errors of judgment: using Domain Specific Languages (DSLs) for configurations.

It all started, as these things are wont to do, with a decent idea: “Wouldn’t it be cool if we externalize values that could conceivably change over time or that could potentially be different in different instances of our application, into a separate file?” People agreed and hence, the configuration file was born. In the olden days the configuration file was a simple static text file with some key-value pairs in them. “How much memory should I allocate for this buffer over here? ”Uhh, dunno, let’s check what the system administrator told us about that by reading this file over here and looking for the value configured for “BufferAllocateMemory.” This was a reasonable step up from the situation where all values were hard-coded in the source code and where a change in these values required a modification of that source code, with all the associated problems of rebuilding and restarting the software. This process is suboptimal at the best of times, but out-right terrible when it is your operating system.

From these humble beginnings, things started getting out of hand pretty quickly. Lots of reasons for that, but there are four things that I want to call out: The four horsemen of terribleness:

Size: Configuration files grew and started containing hundreds of settings. To make matters worse, people often wanted the values of these settings to have some relationship to each other. For instance if the number of buffers was set to n, then the number of file descriptors should always be n*2. Of course we should only have to configure n and then the values of dependent settings should follow from that. Or could we have all of the log files in “/var/log/foo” please, and when we change our minds, can we do that by changing only one thing in the configuration file? Simple static text files don’t allow for programmatically determining the values of various settings and so, clearly, some additional power is needed.

Complexity: People started creating applications that required extensive and complicated configurations. Take Kubernetes as an example: Great system (actually not, but I’ll get back to that in a future article), but its configurations are very complicated. Not only do we have many settings that need to be aligned with each other, but we also need a lot more expressive power because we need to be able to configure lists and maps of things and objects of various complicated (compound) data types. A static text file doesn’t cut it, so clearly, more additional power is needed.

Scale: People started running lots of instances of things that were mostly, but not entirely, identical. For instance: “Configure me a database server that is just like this other database server over here but with 2 CPUs instead of 8 and with 4 GB of RAM instead of 32”. Or: “Make me an alert for <this> condition but with a threshold of .5 errors per second instead of .1 errors per second.” Duplication is evil, so we need a reusable template of some sort with arguments. Clearly, even more additional power is needed.

Reliability: The correctness of the configuration files is by now so important to the overall reliability of the system that we need to version control them, test them, canary them, roll them out gradually, and be able to revert them. Clearly, a lot of additional power is needed.

Configuration has become a software engineering problem. The features we need in and around configuration files are identical to the features we need for writing regular software, including the ability to do calculations, support values of different types (including complex ones like sets, hash maps, and records), selection (if/then/else), loop over ranges of values (for, while, map), linting, the ability to make snippets of code available in multiple places in the file under a single moniker (functions, lambda), and the ability to write modules that we can reuse in different files.

In other words, we need programming languages to write configuration files.

Not a problem, you might think: We have programming languages! Many of them. Some are even not terrible. We could use one of these. We should use one of these. Or, alternatively, if we wanted to be dumb, we could develop a whole class of new and terribly handicapped programming languages to develop configuration files. Guess what we did? By now there are dozens of languages to write configuration files. Some of them claim to be generic, most of them are home-grown and can only be used for a particular system, but all of them are terrible. Without exception.

A lot of this terribleness stems from the fact that the developers of the Domain Specific Language did not want to design a programming language. They wanted to design a limited language that could be used to write configuration files only, and without the complexity that comes with using an actually powerful programming language. Then feature creep started. Lots of reasons for that: The language was used in a more complex environment and needed more power, new use cases needed new features, users requested features to make their lives a bit easier, developers got enthusiastic and thought they could add more power easily, and of course there is the constant pressure caused by promotion driven development.

As C++ shows, continuing to add features to what was originally a relatively simple language does not make it easier to work with. On the contrary, new and old features constantly clash and choices made in the past constrict you in annoying ways. Also you need to be aware of exactly which version of the system you are working with to know if a specific feature is available or not.

At the low end of terribleness are domain specific languages that are hacks on top of an existing text format. Helm’s Chart Template language is a perfect example of this. It starts with YAML, which is in itself already one of the worst things mankind has done to itself this millennium, and then throws in a templating feature that is confusing beyond belief and (therefore) very hard to use. 

At the high end of terribleness is something like the Hashicorp Configuration Language (HCL) which drives things like Terraform. At the end of the day, every Terraform “program” semantically needs to generate a series of named and typed JSON objects that it passes to plugins that then do something useful with these objects. Over dozens of versions, HCL has become more and more powerful so that by now it is for all practical purposes a complete programming language, but one unlike any other (with the possible exception of Google’s proprietary but equally stupid Google Configuration Language (GCL)). This means that if you come to HCL with decades of experience in normal programming languages, you are completely lost. Not because you don’t understand the basics of what is going on, but because it is completely unclear how to do what you want to do. HCL is a bit like programming on Windows: Simple things are really easy, something a bit more complicated is possible but already confusing and hard to figure out, but doing really complicated things is mindboggingly weird and difficult.

Compare this to Unix, where simple things are unfortunately a somewhat hard, more complicated things are not that much harder, but really complicated things are surprisingly easy once you understand the basics really well.

Take the following problem: You want to create an HCL resource (object), but only if a particular expression is true. All programming languages have by now figured out the if statement, so you spend hours looking through the HCL guide looking for it. Turns out HCL doesn’t have an if statement, but you can achieve the effect you are looking for by using the “count” meta attribute that you normally use to create an array of objects (already weird). If you set “count” to 0, the object will not be created. Cute solution, inventive, and needs the secret if statement embedded in the ternary operator. Also, this solution is completely not obvious. HCL is full of things like that; it doesn’t matter that you have four decades of experience in programming, even the simplest things are a needless struggle.

By now I have spent a godforsaken amount of time writing configurations in crappy configuration languages that are often Turing-complete and regularly support basically everything that a full-blown programming language offers, but in the dumbest possible way so as to ascertain that everybody has a terrible time of it. Configuration, my friends, is a software engineering problem, so for the love of all that is holy, let’s use decent software engineering tools to generate them: Existing languages with all the features and supporting tools that you need: Unit test frameworks, debuggers, IDEs, modules, and libraries. If you need to generate a few thousand JSON objects that are strangely the same, or different, or related in weird ways, surely Python or Go are the way to do it. Maybe even Typescript if you really have to. But let’s not waste our collective time and effort coming up with more crappy domain specific languages. They are a blight on this earth and should all be destroyed in holy fire.

Thank you for your attention…