There are ways through which complexity in a problem domain manifests itself in the complexity of a solution. But how easy is it for an outsider to understand why you chose the approach that was made into software?
And why do you always end up with a solution that looks multiple degrees of more complicated compared to the problem you set out to solve?
There’s a certain charm, I believe, in how few people consider the aspects of their problem domain as part of a structured analysis; and how naively certain problems are ignored only to resurface later.
If you have the nerve to tell me that solution X is the best approach out of three for a given problem without even explaining very briefly the advantages & disadvantages they have and that you know about, I have multiple options:
- I could trust your established opinion, on the basis that you’re a fellow team member or software developer who has more experience with the technology at hand.
- I could verify your claims by implementing different solutions and comparing which of those works better for the use case given.
- We could both ignore the problems that only you, a developer knowledgeable in the framework I’m trying to work with, know about; and then later solve them. This is a good way to establish yourself as an unreliable source of information.
Or, and more favorably: You could convince me with your arguments based on actual facts and past experience that certain approaches work better than others.
In fact, I may never know if the approaches not taken are better or worse: If it is simple enough, can be explained in reasonably simple terms, and works out-of-the-box for my use case, then I would strongly consider it good enough. This needs to hold for the problem we’re trying to solve in general, and not just a handful of test cases.
It’s a bit worrying to see so many people titled Software Engineer to just blindly guess at what a solution could look like, ignoring all sorts of aspects fundamental to building good software. If you were engineering a bridge, you’d not just start somewhere and hope that you’ll have a better understanding of the problem by the time you’re done.
Through some miracle, that’s how software is often built. And yet, after you’ve built a solution, do you have the time necessary to adjust it in a way that better resembles your problem?
Created by Dave Snowden for IBM and aimed at helping in decision making, the Cynefin Framework can offer some guidance, offering five distinct domains:
This can be broken down into what we know about the problems:
- Simple: Cause and effect are understood by everyone: You can establish the facts, categorize them, and respond to them according to straightforward best practices.
- Complicated: Cause and effect are understood by experts or through analysis. Rather than categorizing the known facts, we now analyze them in hopefully the right way. While we had a best practice before, we now have multiple equally possible solutions.
- Complex: You can’t tell the effect of your actions in advance but only retrospectively: Probing will cause effects that can be analyzed in hindsight.
- Chaotic: The effects of any action are unclear, and it is only by acting that you can even remotely find out how things work. The most reasonable act is to try and respond by turning a chaotic problem into a complex one.
- Disorder: We don’t know which one of the above four definitions apply, and it is challenging to categorize the problem domain.
Cynefin isn’t centered around software development, but has been used as framework for making decisions through many different layers, from policies to customer relations. There are things we can take away from it, though.
Simple problems should have simple solutions. If they don’t, that’s cause for concern: What makes your solution so complex? Similarily, if you have a complex problem, are you sure the trivially simple solution you have found is a good fit?
It’s maybe not surprising at this point that different developers will gravitate to different solutions on this scale: If you’re a junior developer looking for a solution that just works when you write it, there’s a high chance you’ll emerge in the complex domain.
If you’re more on the senior end of the scale, there’s past experiences that’ll come helpful: You can more easily determine which domain are appropriate, and solve things in ways matching the underlying problem’s complexity - and in extremely rare cases, find simpler ways of approaching complex problems. In my humble opinion, this very explicitly includes solving complicated problems through an architecture that is just complicated enough for the solution you’re building, but may not be trivial.
Categorizing the problem is something that everyone can and should do. With each attempt, visualize the entire problem, and consider what your previous guess missed that you know in retrospect. Is it applicable here? Do you foresee issues that you, personally, would struggle with? You can learn from what others see as fundamental aspects to solve, and use it as basis for the future.
If the domain and your implementation align closely enough for everyone involved, you can work with it even if it’s not necessarily the exact problem domain according to Cynefin: anyone can only estimate, and any guess can be wrong. You can always contemplate on what to improve that would help either the business or the development team through the course of development. And if you identify parts that are an absolute hassle to work with, but that you think shouldn’t be, you can always optimize for a simpler solution.
If you can wrap the complexity in an approachable architecture that’s understood by other developers, then you’re in a good position to work on it with your team.
Works for me.
There’s a notable difference between works for me and works in general: It all of that falls apart when the complexity of problem and solution don’t align in what you created. On the long road before your product is ready to ship, there are quite a few occasions when you could stumble upon accidental complexity.
One of the weirder ways I found this issue to manifest itself is in the disguise of an agile-themed discussion: Somehow, somewhere along the lines of this whole agile idea, someone inevitably will convince you to strive for the easiest solution possible.
While the exact implementation details is often left open to interpretation, you can make a guess based on who works on a story. If somewhat unsurprisingly, whoever decides to work on that user story decides to build it in the simplest way possible, that’s terrible: without even the slightest hint of consideration towards the fundamentally bigger scope of the solution you’re working on, you’re bound to rewrite it rather sooner than later.
And the developer who’s just made it simplistic has absolved themselves of all responsibility.
Their user story works exactly as requested, and who could’ve known your second, almost identical problem is in no way able to reuse any of it. Would you trust whoever’s made this mistake in the first place to come up with a better solution? Or do you just think they’re incompetent and be forced to rewrite their code at risk of breaking it?
Does your company/team culture even allow for the original developer to refactor their code, or are they now “busy with another story” and unable to help?
Building tomorrow’s technical debt today
Through simple extrapolation of past experiences with your team colleagues, you could probably name at least one person that only solves a user story in a vacuum. It’s only sad when you also consider this person to have been around the company for years longer than you have, while also having a lot more technical knowledge with some aspects of your software – and then still lacking the foresight to build in any way reusable components.
You are certainly doing something wrong if you’re blindly ignoring everyone around you & building excessive technological debt while implementing completely new features.
There are multiple phases where developers get involved with a user story, aside from the one or two people implementing it. Yet through a combination of scrum master and implementing developer, you’re often no longer solving a general problem. You’re solving only as little as you need for one user story.
And when there’s so little consideration for future development, you’re no longer building general solutions. And now you’re back to hacking in the needed parts in multiple workarounds, shifting the complexity from finding a solution to finding a useful solution.
Ah, I would’ve preferred the latter on my first go - but could I have known it would end this way?
Maybe user stories aren’t a good format for preparing architectural building blocks. Perhaps features aren’t the only thing relevant in software development. Maybe you could’ve thought about architecture first & your user story second.
No universal solution to ignoring architecture exists: You need to address it as soon as possible, preferably before you start implementing it. Discuss architecture with your team beforehand and you could save yourself days and weeks building stuff to tear it down shortly afterwards. Discuss it afterwards, and you’ll either be ignored or someone’s work thrown away.
If I know that, at least parts of my solution will be reused within the same sprint/scope/project we’re currently working on, I nowadays refuse to implement the simplest works for me type of solution – even if I am strongly encouraged to.
And if you ignore the big picture & the future entirely, you’re building up architectural debt that will be a burden to you later.
Finding the appropriate level to solve problems
Software is only as good as you make it. You’re bound to involve a variety of people in finding good solutions - and sooner or later, solutions need to be found. All problems have a scope, and the solutions I most often see are as abstract & general as can be, or as specialized & concrete as can be.
Often enough, the problem you are trying to solve in software is on a different layer than the feature you’re implementing. If you have a single user story, you have a very specialized problem at hand. Through expertise, you can extrapolate and abstract it to a level where the problem should be.
There are general problems and specialized problems, but also a wealth inbetween: local to your module, local to the software stack you’re using, local to your business department-specific implementation, and many more.
Remember the complex, complicated, chaotic and simple problem domains from before? Which one applies to the issue at hand, and where should the problem be solved?
Yes, we can solve the most general problem that’s on the scale in the most complex way – one that can be used everywhere, at any time. But is it the smart thing to do? Or do I need a solution for a specific subdomain only? Generic solutions are complicated, but not always necessary. Problems don’t go away, of course. Yet if my solution is at the wrong level, I just place the burden on someone else later down the line.
And your solution doesn’t need to be perfect necessarily. It can be a starting place to build upon and improve. And hey, you could still throw it away if it doesn’t work, but at least you put in the effort to solve it at the right level.
If it’s even the slightest bit reusable, that’s better than the next person starting from scratch, even if months or years down the line you find a better approach.
- While I believe everyone can change for better or worse, being unreliable means I have a harder time trusting yourself. Where I previously trusted your experience, your actions now need to establish that I can trust you. ↩
- Tests can’t ever establish the absence of bugs, after all. They can verify that parts of the software work for known types of inputs, and more importantly, can verify the software still works on a functional level after you’ve made changes to it. ↩
- Oversimplifying things early on can lead to plenty of headaches later down the line. A good approach that I try to follow is to think of people who work with the code after me: with all I know about future problems, can they be solved reasonably with the given architecture, or would they require a complete rewrite? ↩
- This person may or may not be your scrum master, but I’ve also seen it throughout the other members of the team. ↩
- Just pray that your solution ends up at complex at worst, and not chaotic. ↩