Data Model Debt is Forever

The team had avoided adding this feature for years. The software was a platform for creators to sell content. It started off as a way for to sell one-off content and subscriptions were added later. The pay once products had a list of content associated with them. A customer paid and got access to a subset of the creators content. The subscription product did not have any direct association with the content. If you paid for a subscription from a creator you got access to all the content made by the creator.

This data model stayed that way for years and was built upon. The service added iOS, Android and built-in TV apps. All of these apps were built with the understanding that subscriptions gave a user access to all the content of a creator.

There were competitors now and they offered multiple subscription products to the creators. The pressure from customers kept growing so the decision was made to try and add multiple subscription products

At first it was the explicit problems. The current subscription product needed to be associated explicitly to all of a creators content so that when a new subscription product was created it could be associated to a subset of the creators content. Then it was the hidden couplings. For example, monthly sales reporting broke because the SQL query that generated the report assumed one subscription product per creator. Then the teams that wrote the iOS, Android and TV apps got involved. It felt insurmountable. The project was abandoned.

No Silver Bullet published in 1986 by Fred Brooks makes the argument that the essential problem of software development is design.

The essence of a software entity is a construct of interlocking concepts: data sets, relationships among data items, algorithms, and invocations of functions. This essence is abstract, in that the conceptual construct is the same under many different representations. It is nonetheless highly precise and richly detailed … I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation. We still make syntax errors, to be sure; but they are fuzz compared to the conceptual errors in most systems.

In that essay he makes the argument that most of the advances in software development solve the incidental problems. Incidental problems are what it takes to implement a solution and get it in the hands of your customers. The essential problem is coming up with the solution, the “conceptual construct”. This is hard because coming up with a solution is not a one time thing. A successful piece of software will have tons of pressure to add more and more features. So you must solve the current problem while making it as easy as possible to change the software when the new requirements come.

In the essay Brooks offers that the essential problems of software complexity can be tackled with

• Rapid prototyping to figure out actual requirements
• Incremental development. Growing, not building software in slices
• To buy rather than build
• Train and encourage great software designers

Most of what he advocated has become mainstream. The Agile movement helped make rapid prototyping and incremental development mainstream. The Open Source and SaaS movements have made easier to reuse other peoples work. What has not happened is the training and encouraging of great software designers.

I think this is because the cost of bad software design takes a while to feel. The decision of letting subscription products implicitly have access to all of a creators content worked fine for several years. The people who made that decision were not the ones that had to deal with trying to add a new feature which that decision made nearly impossible. On the other side it takes a while for the benefits of good software design to be felt. In the early days, the good software design and the bad software design are able to support new features at the same pace. The bad software design might even be faster. It’s only after years of building on the bad software design does the mistake really start to hurt and you wish you had the better design. By then it may require a complete rewrite of the software.

I think the solution to this problem is to have design be a separate phase of the software development process. A phase where different solutions to the problem are considered and the tradeoffs of each are made explicit. I’ll write more about what this phase looks like in a follow up post but I think its going to become more important as the more mechanical aspects of software development gets automated with AI.