I could not agree more. The problem is that the software industry’s workforce has shifted from engineers to power users. The engineer is supposed to do what you suggest: provide a solution for a specific problem. What we have in the industry are power users, which do not want to be engineers anymore. You can become the configuration specialist and reject any ambition to be an engineer. The new idea is that we have either power users, or we have people inventing engines for power users.
Not everyone wants to become an engineer, not everyone succeeds in becoming an engineer, but everyone can become a biological configurator. This reflects the industry's tale: engineering skill is not necessary; it is all just a matter of setup experience. Why don't we count the hours spent on configuring systems, like pilots have flying hours? We could list the hours in our resume. Then we can also tell the tale that systems can be devised by anyone.
Engine-itis makes things look simple without being easy.
What the industry pretends to need at any cost is uniformity and engine-itis is a good response to that. Engine-itis is a social equalizer: the skilled engineer and the talented power user who recalls all config switches by heart look alike, because they struggle in similar ways with the legacy.
I think you have more than one point here, I fully agree, since ever I wish to go back to the roots: let’s be engineers and let’s engineer. Then, so many dysfunctional containers/engines with fancy DSLs (nobody needs in a properly designed system) will melt like snow in spring.
At Google, the #1 reason for writing a configuration DSL (typically protobuf) is to achieve language neutrality for the configuration surface. I would buy most of Jos's argument that all the other reasons are BS, but this one clearly isn't. Sometimes this can be sidestepped by hiding the library behind a service, but that comes with its own costs.
A library offered in 3 languages configured with a DSL (and otherwise nearly-zero-config) is a lot easier to document and maintain than a library configured natively in each of the languages.
Is it wrong for big companies to invest into multiple languages? I don't have the answer, but the article seems written with the tacit assumption that it is indeed wrong, which I doubt is clear.
"I know you fear it might not help you get promoted" this is so true, and I have seen this practice in multiple companies that design and launch a generic platform as a way to get engineers promoted, and it worked. So how to fix this?
I tend to find that the folks that are building the best "engines" are the folks that wrote the application that solved a business problem to start with, and then understood that there was some base engine capability that needed to be abstracted out (often because they found that they had a very similar, but not quite compatible business problem, that didn't quite fit their existing infrastructure!). This kind of tie stops one building generic things that have no applicability.
The point about DSLs is also well made -- people often think that because they can parse some DSL first time around, it's easy to establish one. It is definitely not in my experience. Building a complete ecosystem around a DSL is a huge, ongoing investment.
There is a lot to agree with here, but I also kept getting the uneasy feeling that it was recommending the complete opposite of Steve Yagge's Platform Rant (https://gist.github.com/chitchcock/1281611), which I also agree with.
However, I think both of these arguments are targeting different organizations and projects, and both are right in the right context. And some of these arguments are about the approach and implementation details.
Yagge's Rant is right for Google, because Google is a big infrastructure provider (for both itself and external customers), and thus **is** the kind of organization that should be building engines. For the vast majority of other organizations and projects, you should be using engines, not building them.
It's also a bit unclear what you mean by "engine", and there is the specific case of a very generic DSL driven thingy, but you also mention that the "engine" trend started with linkable libraries. So I'm going to define "engine" as any generic re-usable component, regardless of how it's implemented. This means you can not only argue about whether you should be building an engine, but also how you should implement it.
In all cases, there are some definite approach and implementation red flags that should make you question your approach;
1. Always step back and think about the point of what you are doing. Is the point to build an engine, or is the point to solve a particular problem? Is what you are planning to do addressing the point, and is it the best way to address it? Maybe you really do have multiple similar shaped problems that could be best addressed by building an "engine" of some kind for all of them, but do you really, or are you just imagining theoretical future problems?
2. The undocumented and neglected first step of any design process is **research prior art**. You might be smart, but there is a huge number of people in human history that were (or still are) smarter than you. It's highly likely that someone has already tackled your problem, or something very similar, and maybe they've already done all or most of the work for you. Don't re-invent and re-implement solved problems badly (if I had a penny for every time I've seen this...).
3. If you find yourself inventing a DSL, you are probably doing it wrong. What is wrong with the language you are writing your project in? Maybe you should be writing a library for that language instead? Are you sure there is not already a perfectly good DSL or general-purpose language you could just use? Is the "Domain" really so "Special" that a new language will be better than an existing mature language and its whole supporting ecosystem? Do you really want to waste your life building a mature supporting ecosystem for it?
4. Usually the best time to build some kind of generalized engine is when you find yourself about to implement the same thing the third time. Go back to the two earlier implementations, refactor out the common bit, taking into account your about-to-be third implementation, and then make all three use the common engine/library. Don't speculate about needing an engine before you have proven you need it. Refactoring old "finished" projects is an important part of project maintenance and make sure it's budgeted and accounted for. A project that is not actively maintained is not "finished", its "dead".
5. If you really do find yourself needing to build an "engine", KISS. Do you really need an RPC micro-service that uploads and executes a DSL, or would a linkable library do the job? Think about the support/maintenance requirements. Keep the functionality of it to a minimum to meet the requirements, but try to design it with expansion in mind when/if the requirements grow. Note that engines are not really part of a project, they are their own project.
I could not agree more. The problem is that the software industry’s workforce has shifted from engineers to power users. The engineer is supposed to do what you suggest: provide a solution for a specific problem. What we have in the industry are power users, which do not want to be engineers anymore. You can become the configuration specialist and reject any ambition to be an engineer. The new idea is that we have either power users, or we have people inventing engines for power users.
Not everyone wants to become an engineer, not everyone succeeds in becoming an engineer, but everyone can become a biological configurator. This reflects the industry's tale: engineering skill is not necessary; it is all just a matter of setup experience. Why don't we count the hours spent on configuring systems, like pilots have flying hours? We could list the hours in our resume. Then we can also tell the tale that systems can be devised by anyone.
Engine-itis makes things look simple without being easy.
What the industry pretends to need at any cost is uniformity and engine-itis is a good response to that. Engine-itis is a social equalizer: the skilled engineer and the talented power user who recalls all config switches by heart look alike, because they struggle in similar ways with the legacy.
I think you have more than one point here, I fully agree, since ever I wish to go back to the roots: let’s be engineers and let’s engineer. Then, so many dysfunctional containers/engines with fancy DSLs (nobody needs in a properly designed system) will melt like snow in spring.
At Google, the #1 reason for writing a configuration DSL (typically protobuf) is to achieve language neutrality for the configuration surface. I would buy most of Jos's argument that all the other reasons are BS, but this one clearly isn't. Sometimes this can be sidestepped by hiding the library behind a service, but that comes with its own costs.
A library offered in 3 languages configured with a DSL (and otherwise nearly-zero-config) is a lot easier to document and maintain than a library configured natively in each of the languages.
Is it wrong for big companies to invest into multiple languages? I don't have the answer, but the article seems written with the tacit assumption that it is indeed wrong, which I doubt is clear.
"I know you fear it might not help you get promoted" this is so true, and I have seen this practice in multiple companies that design and launch a generic platform as a way to get engineers promoted, and it worked. So how to fix this?
Here's a 2 min audio version of "Don’t build engines" from Wednesday Wisdom converted using recast app.
https://app.letsrecast.ai/r/4f12195f-fbd6-4308-9701-95bfa8065dca
Well written article.
I tend to find that the folks that are building the best "engines" are the folks that wrote the application that solved a business problem to start with, and then understood that there was some base engine capability that needed to be abstracted out (often because they found that they had a very similar, but not quite compatible business problem, that didn't quite fit their existing infrastructure!). This kind of tie stops one building generic things that have no applicability.
The point about DSLs is also well made -- people often think that because they can parse some DSL first time around, it's easy to establish one. It is definitely not in my experience. Building a complete ecosystem around a DSL is a huge, ongoing investment.
Very well written. Startup founders should read this and enforce it amongst their staff.
No developer should be writing this kind of code, unless he funds it himself.
There is a lot to agree with here, but I also kept getting the uneasy feeling that it was recommending the complete opposite of Steve Yagge's Platform Rant (https://gist.github.com/chitchcock/1281611), which I also agree with.
However, I think both of these arguments are targeting different organizations and projects, and both are right in the right context. And some of these arguments are about the approach and implementation details.
Yagge's Rant is right for Google, because Google is a big infrastructure provider (for both itself and external customers), and thus **is** the kind of organization that should be building engines. For the vast majority of other organizations and projects, you should be using engines, not building them.
It's also a bit unclear what you mean by "engine", and there is the specific case of a very generic DSL driven thingy, but you also mention that the "engine" trend started with linkable libraries. So I'm going to define "engine" as any generic re-usable component, regardless of how it's implemented. This means you can not only argue about whether you should be building an engine, but also how you should implement it.
In all cases, there are some definite approach and implementation red flags that should make you question your approach;
1. Always step back and think about the point of what you are doing. Is the point to build an engine, or is the point to solve a particular problem? Is what you are planning to do addressing the point, and is it the best way to address it? Maybe you really do have multiple similar shaped problems that could be best addressed by building an "engine" of some kind for all of them, but do you really, or are you just imagining theoretical future problems?
2. The undocumented and neglected first step of any design process is **research prior art**. You might be smart, but there is a huge number of people in human history that were (or still are) smarter than you. It's highly likely that someone has already tackled your problem, or something very similar, and maybe they've already done all or most of the work for you. Don't re-invent and re-implement solved problems badly (if I had a penny for every time I've seen this...).
3. If you find yourself inventing a DSL, you are probably doing it wrong. What is wrong with the language you are writing your project in? Maybe you should be writing a library for that language instead? Are you sure there is not already a perfectly good DSL or general-purpose language you could just use? Is the "Domain" really so "Special" that a new language will be better than an existing mature language and its whole supporting ecosystem? Do you really want to waste your life building a mature supporting ecosystem for it?
4. Usually the best time to build some kind of generalized engine is when you find yourself about to implement the same thing the third time. Go back to the two earlier implementations, refactor out the common bit, taking into account your about-to-be third implementation, and then make all three use the common engine/library. Don't speculate about needing an engine before you have proven you need it. Refactoring old "finished" projects is an important part of project maintenance and make sure it's budgeted and accounted for. A project that is not actively maintained is not "finished", its "dead".
5. If you really do find yourself needing to build an "engine", KISS. Do you really need an RPC micro-service that uploads and executes a DSL, or would a linkable library do the job? Think about the support/maintenance requirements. Keep the functionality of it to a minimum to meet the requirements, but try to design it with expansion in mind when/if the requirements grow. Note that engines are not really part of a project, they are their own project.