Not to defend software engineers, but there are plenty of examples of projects exploding in time and cost outside the software industry. For example, Boeing's 777X (which is not even a new design), was supposed to launch in 2019-2020. Today, it is expected to launch in 2025, see https://youtu.be/wyJFnPWe-gs.
Definitely true. For a great story about a civil engineering project that went completely overboard, I can recommend https://www.wgbh.org/podcasts/the-big-dig, which is about The Big Dig in Boston, the benefits of which I enjoy on a daily basis, but which is legendary in terms of budget and schedule overruns. The new airport in Berlin is another terrible example.
What I find interesting in software projects though is that the overruns are typically not due to shortages of resources (concrete, iron), problems in supply lines, or other manifestations in the real world. It is _all_ cerebral: "This turned out to be much slower and more difficult than we thought."
I was having just this debate with some colleagues about two weeks ago. Having had success with leading some projects using a dependency graph, I simply could not, and cannot, fathom the spreadsheet approach.
Of course, as is the norm in our industry, I was completely oblivious to prior art. I knew I couldn't be the first person to have used a dependency graph to manage a software project. But I didn't know specifically about CPM --- even though I knew about CP/M. ;-)
I wonder when this industry will figure this out. In the end, take the best parts of the existing tools (waterfall and agile) and make your own project planning. At _least_ take from either waterfall or agile. It seems we're always inventing our own and they're always worse than either extreme.
<blockquote>We are a species that can build and operate cruise ships and construct safe airplanes with approximately 340,000 parts, but we cannot plan the implementation of a new payments and HR system.</blockquote>
Given the steady drumbeat of stories about Boeing failures, I'm not sure we can construct safe airplanes anymore, either...
Boeing is definitely in a spot of bother, but given what _can_ go wrong once you take off into the air, even Boeings are still remarkably safe. Give or take the odd panel blowing out or wheel falling off :-)
The tale of Boeing's downfall shows that [enshitification](https://en.wiktionary.org/wiki/enshittification) happens not only to online companies. Bean-counter management obsessed exclusively with shareholder wealth causes any company to decline (to the benefit of the shareholders who will sell out just before the collapse). Company-wide culture-damage to morale and resource starvation "cost-cutting" is a whole other reason that contributes to projects going badly.
Project management for any kind of project involving "design" is a tricky beast. By definition, "design" involves unknowns, so any project planning done in advance must be either vague and fuzzy , or full of made-up shit. Planning is also an overhead that doesn't directly contribute to getting the job done. It can help you figure out very roughly what needs to be done, in what order, and how long it will take, but aside from some possible dead-end avoidance and better task-ordering, it doesn't really reduce the total amount of work required, it just adds more work. The more accurate and further into the future it tries to project, the more unknowns it has to predict, and the more work and BS it becomes.
You can expend more and more effort on more detailed/accurate/prophetic project planning, but there is a definite decline in the return-for-effort. From my experience with big waterfall Aerospace projects, accurate plans required accurate requirements and designs in advance to try to minimize the unknowns. History showed "the earlier in the waterfall process that errors are made, the more expensive they are to rectify", so more and more effort was expended on trying to get these right, till the planning/requirements/design was significantly more work than actually implementing it. And they still got it wrong.
"Agile" was a reaction to this; why waste too much effort on planning/design/etc in advance when it's going to be wrong anyway? Let's just iterate our way to the solution! Unfortunately, iteration is not always an optimal way to reach the solution, and you can't iterate your way out of a local-maxima. You can find yourself iterating down every dead-end-path to a sub-optimal solution that can't be iterated better any more, and requires a complete re-write from scratch.
So the sweet-spot is somewhere in the middle. The trick is have a rough plan all the way to the best-guess-end-goal, but don't waste too much effort on it. Beyond a minimum level of effort it will not finish the project any cheaper/faster/better, and will only give diminishing improvements to estimates of what the final cost/time will be. Know that it's just a best-guess for now, and update it as you go. Use it to inform your development iterations to allocate resources, avoid possible dead-ends, and choose a path that can reach the best solution.
Identifying critical paths is definitely one of the useful tools for doing this, but be aware of the effort-vs-return trade-offs in any kind of future projections for design projects, and that they are always varying degrees of wrong.
“[I]t doesn't really reduce the total amount of work required, it just adds more work.”
Y'know what, this would neatly explain why my team is so successful at delivery: everyone pretty much works alone, with the occasional exception for pair-programming on an as-needed basis. Given that we are currently under some kind of quote-unquote “agile” mentorship program, this might be something to bear in mind while arguing with the consultant on how to revolutionize the organization of work... So thanks!
At least in the tools I've seen for dealing with Gantt charts, they obscure what I would consider the most important information: the dependencies.
It seems like Gantt charts can answer the question of "what is our current estimated completion date?", but not the question of "what are the risks in our work plan?"
Every Gantt chart tool I've ever seen/used worked by specifying dependencies between tasks and task durations, with the start/end dates being driven by those constraints. This automatically calculates the critical path which can be highlighted. It's also kinda nice in that it automatically adjusts/recalculates the critical path whenever the durations/dependencies are adjusted. Often they have parent/child task relationships too, allowing you to collapse/expand the view to different levels of task-granularity.
So IMHO gantt charts are a highly useful visual tool for identifying/tracking the critical path. You can generate a gantt chart from anything composed of "tasks" that have durations and dependencies (and optionally parent/child relationships for granularity), including bug-trackers. Identifying/highlighting the critical path is just finding the path(s) between the first task(s) to start and the last task(s) to end with no "slack" between a task ending and it's dependency starting.
Things like MS Project (my experience with this ended in about 2003, so my knowledge is both highly dated and fuzzy) could highlight the critical path and also allowed you to play with things like resources available and required for tasks. So things like "Dave is needed for this and this and this task" automatically meant those tasks could not be done in parallel, or at least could not have the required "Dave hours" part of the required resources in parallel, and you could identify critical resources.
Sure; it doesn't matter if you're using a CPM tool, or a Gantt chart tool, or really any tool at all, if there isn't someone accountable who is going to intelligently break down the tasks and diligently organize and track them in the tool.
I've been known to plan projects with graphviz :)
But yes, then I constantly got questions amounting to "why can't you just use a spreadsheet?!"
Not to defend software engineers, but there are plenty of examples of projects exploding in time and cost outside the software industry. For example, Boeing's 777X (which is not even a new design), was supposed to launch in 2019-2020. Today, it is expected to launch in 2025, see https://youtu.be/wyJFnPWe-gs.
Definitely true. For a great story about a civil engineering project that went completely overboard, I can recommend https://www.wgbh.org/podcasts/the-big-dig, which is about The Big Dig in Boston, the benefits of which I enjoy on a daily basis, but which is legendary in terms of budget and schedule overruns. The new airport in Berlin is another terrible example.
What I find interesting in software projects though is that the overruns are typically not due to shortages of resources (concrete, iron), problems in supply lines, or other manifestations in the real world. It is _all_ cerebral: "This turned out to be much slower and more difficult than we thought."
Yikes --- you must be inside my head some how!
I was having just this debate with some colleagues about two weeks ago. Having had success with leading some projects using a dependency graph, I simply could not, and cannot, fathom the spreadsheet approach.
Of course, as is the norm in our industry, I was completely oblivious to prior art. I knew I couldn't be the first person to have used a dependency graph to manage a software project. But I didn't know specifically about CPM --- even though I knew about CP/M. ;-)
And kudos for knowing CP/M! :-)
There are some halfway decent open source tools that support this: https://opensource.com/article/21/3/open-source-project-management, though none are as explicitly graph based as I would like.
I wonder when this industry will figure this out. In the end, take the best parts of the existing tools (waterfall and agile) and make your own project planning. At _least_ take from either waterfall or agile. It seems we're always inventing our own and they're always worse than either extreme.
<blockquote>We are a species that can build and operate cruise ships and construct safe airplanes with approximately 340,000 parts, but we cannot plan the implementation of a new payments and HR system.</blockquote>
Given the steady drumbeat of stories about Boeing failures, I'm not sure we can construct safe airplanes anymore, either...
Boeing is definitely in a spot of bother, but given what _can_ go wrong once you take off into the air, even Boeings are still remarkably safe. Give or take the odd panel blowing out or wheel falling off :-)
The tale of Boeing's downfall shows that [enshitification](https://en.wiktionary.org/wiki/enshittification) happens not only to online companies. Bean-counter management obsessed exclusively with shareholder wealth causes any company to decline (to the benefit of the shareholders who will sell out just before the collapse). Company-wide culture-damage to morale and resource starvation "cost-cutting" is a whole other reason that contributes to projects going badly.
Project management for any kind of project involving "design" is a tricky beast. By definition, "design" involves unknowns, so any project planning done in advance must be either vague and fuzzy , or full of made-up shit. Planning is also an overhead that doesn't directly contribute to getting the job done. It can help you figure out very roughly what needs to be done, in what order, and how long it will take, but aside from some possible dead-end avoidance and better task-ordering, it doesn't really reduce the total amount of work required, it just adds more work. The more accurate and further into the future it tries to project, the more unknowns it has to predict, and the more work and BS it becomes.
You can expend more and more effort on more detailed/accurate/prophetic project planning, but there is a definite decline in the return-for-effort. From my experience with big waterfall Aerospace projects, accurate plans required accurate requirements and designs in advance to try to minimize the unknowns. History showed "the earlier in the waterfall process that errors are made, the more expensive they are to rectify", so more and more effort was expended on trying to get these right, till the planning/requirements/design was significantly more work than actually implementing it. And they still got it wrong.
"Agile" was a reaction to this; why waste too much effort on planning/design/etc in advance when it's going to be wrong anyway? Let's just iterate our way to the solution! Unfortunately, iteration is not always an optimal way to reach the solution, and you can't iterate your way out of a local-maxima. You can find yourself iterating down every dead-end-path to a sub-optimal solution that can't be iterated better any more, and requires a complete re-write from scratch.
So the sweet-spot is somewhere in the middle. The trick is have a rough plan all the way to the best-guess-end-goal, but don't waste too much effort on it. Beyond a minimum level of effort it will not finish the project any cheaper/faster/better, and will only give diminishing improvements to estimates of what the final cost/time will be. Know that it's just a best-guess for now, and update it as you go. Use it to inform your development iterations to allocate resources, avoid possible dead-ends, and choose a path that can reach the best solution.
Identifying critical paths is definitely one of the useful tools for doing this, but be aware of the effort-vs-return trade-offs in any kind of future projections for design projects, and that they are always varying degrees of wrong.
“[I]t doesn't really reduce the total amount of work required, it just adds more work.”
Y'know what, this would neatly explain why my team is so successful at delivery: everyone pretty much works alone, with the occasional exception for pair-programming on an as-needed basis. Given that we are currently under some kind of quote-unquote “agile” mentorship program, this might be something to bear in mind while arguing with the consultant on how to revolutionize the organization of work... So thanks!
The CPM sounds a lot like a Gantt chart.
At least in the tools I've seen for dealing with Gantt charts, they obscure what I would consider the most important information: the dependencies.
It seems like Gantt charts can answer the question of "what is our current estimated completion date?", but not the question of "what are the risks in our work plan?"
https://teamboard.cloud/gantt-chart-critical-path/ contains a good discussion on Gantt charts versus the critical path.
Every Gantt chart tool I've ever seen/used worked by specifying dependencies between tasks and task durations, with the start/end dates being driven by those constraints. This automatically calculates the critical path which can be highlighted. It's also kinda nice in that it automatically adjusts/recalculates the critical path whenever the durations/dependencies are adjusted. Often they have parent/child task relationships too, allowing you to collapse/expand the view to different levels of task-granularity.
So IMHO gantt charts are a highly useful visual tool for identifying/tracking the critical path. You can generate a gantt chart from anything composed of "tasks" that have durations and dependencies (and optionally parent/child relationships for granularity), including bug-trackers. Identifying/highlighting the critical path is just finding the path(s) between the first task(s) to start and the last task(s) to end with no "slack" between a task ending and it's dependency starting.
Things like MS Project (my experience with this ended in about 2003, so my knowledge is both highly dated and fuzzy) could highlight the critical path and also allowed you to play with things like resources available and required for tasks. So things like "Dave is needed for this and this and this task" automatically meant those tasks could not be done in parallel, or at least could not have the required "Dave hours" part of the required resources in parallel, and you could identify critical resources.
Sure; it doesn't matter if you're using a CPM tool, or a Gantt chart tool, or really any tool at all, if there isn't someone accountable who is going to intelligently break down the tasks and diligently organize and track them in the tool.
Totally true.