software is in the grey areas

It’s all grey – almost nothing in life is black and white. Complexity is all around us. Human as we are, we create rules and mental models to handle the complexity of the world, to reduce it to something we can manage. We abstract and approximate. In our field, that’s resulted in principles and practices such as: DRY, SOLID, SRP, composition over inheritance, TDD, the Agile manifesto, functions must be no longer than 100 lines, avoid premature optimisation, microservices. The list goes on forever. There’s nothing wrong with any of these and in fact, they’ve been invaluable in progressing our field, however, we must take them as they are: tools to manage complexity.

I’ve found framing them as such invaluable. Instead of treating them as gospel, I can just reach for the right principle/practice given the context. That’s liberating. I can ignore them entirely if I don’t think they make sense or I can modify them in any way I please to make them suitable. I have no desire to follow them “in the way they were intended” because they were written with a specific context in mind. Code and practices for writing embedded systems software is wildly different from SaaS which is wildly different from exploratory data analysis. While the spirit of the principles and practices apply across all of them, they can’t be blanket applied as intended in every scenario.

As a concrete example, I worked on a team that was building a machine learning product. Part of the team was responsible for the machine learning work and while the rest was responsible for building the application. The team loosely follow followed Agile principles. We worked in two week sprints and did the whole story points thing. This worked well for the software engineering work, there weren’t many unknowns. That made estimating story points tractable. For the data science team, it didn’t work nearly as well. The work we were doing was exploratory in nature. Story point estimates were effectively useless; it’s difficult to estimate how long or how difficult the work is when there’s no certainty. We could try one approach and get high quality results immediately or we could end up trying 12 different approaches. Who knows! We actually did try our best to do estimates but realised that it wasn’t working. So, we dropped story points from any exploratory data science work. That worked much better for everyone, expectations were better aligned and the working environment for that type of work ended up being much healthier – no more spending 2 weeks on a 3 point task.

One of the better habits I picked up was to repeatedly throw out principles and practices whenever I’m approaching a new problem or situation. I start off thinking about the context, especially the constraints. You can understand a ton about software if you understand the constraints it was built under. Only after I have a picture of the context do I start looking at the principles. I continue to learn as many of them as I can so that I have a depth of knowledge to draw from to inspire my approach. I’m happy to mix and match, modify and discard. It’s been invaluable in making me a better software engineer. But more importantly, it has given me a framework to understand decision making (and just generally be more patient with decisions I don’t understand¹). Ever read code and think to yourself: what in god’s green earth were they thinking? Half the time, the answer is simply: management.

The challenge in continuously evaluating context is that it’s unenjoyable. You throw out your tools for managing complexity and have to start from (near) ground zero every time. In Thinking, Fast and Slow terminology, we’re forced to use System 2 thinking. However, from my own experience, it pays off.

Software engineering is a conglomeration of grey areas, nothing is set in stone. I believe you’re better off learning how to think critically about what applies in your situation rather than running the rulebook.