The unsolvable problem at the heart of software development

At the core of software development is a problem that will not be solved by new programming languages, tools, or processes. It won’t be solved by AI. You can have infinite scaling for pennies, immediate deployment rollbacks, immediate deploys to production. The problem will still be there. No Silver Bullet by Fred Brooks lays out the problem.

The essence of a software entity is a construct of interlocking concepts: data sets, relationships among data items, algorithms, and invocations of functions. This essence is abstract, in that the conceptual construct is the same under many different representations. It is nonetheless highly precise and richly detailed.

The key term is “conceptual construct”. The conceptual construct mostly exists in the minds of the programmers writing the code, only traces of it make it into the code, comments, or documentation. It includes all the reasoning behind the tradeoffs and hacky workarounds.

How do you build software that must meet continually changing requirements from users, regulators, and other software? There will never be some underlying theory that captures the fundamentals software development that evolution for biology or Maxwell’s equations for electrical engineering. Nature doesn’t keep coming up with new requirements.

In short, the software product is embedded in a cultural matrix of applications, users, laws, and machine vehicles. These all change continually, and their changes inexorably force change upon the software product

Brook’s offers some medicine but no cure. He suggests to buy instead of build, to do incremental development with rapid prototyping, and to find and train great software designers. We’ve mostly done the first two. Open source software, cloud computing and software as a service APIs allow us to buy most of the software we ship. The Agile movement introduced incremental development and rapid prototyping. We still have software that is too complex.

Great designs come from great designers. Software construction is a creative process. Sound methodology can empower and liberate the creative mind; it cannot enflame or inspire the drudge.

We haven’t tried to develop better software designers ⊕ The Mirdin School is one place that I know is trying. I took the courses and though they are hard I think they are worth the effort. . Design compounds. You don’t feel the pain of bad or no design until the technical debt is overwhelming. You don’t feel the benefits of good design until you experience how easy it is to meet some new requirement 5+ years in.

This is why the problem is unsolvable. By the time you notice there’s a problem it’s too late.

The hardest single part of building a software system is deciding precisely what to build. No other part of the conceptual work is so difficult as establishing the detailed technical requirements, including all the interfaces to people, to machines, and to other software systems. No other part of the work so cripples the resulting system if done wrong. No other part is more difficult to rectify later