Technical debt is on everyone’s mind in the tech industry. According to a recent survey, companies spend about $85 billion annually on fixing bad code. If it were removed, the extra time and effort developers need to spend with it could be poured into something more productive, and that might end up creating quite a lot of extra value.
There could be many reasons behind the existence of “bad code,” but a lot of it is actually technical debt amassed by rushed work to meet deadlines, or lazy engineers unfamiliar with key design principles.
All in all, technical debt causes a lot of trouble.
So, we put together this comprehensive guide on technical debt that covers every important aspect, such as the following:
(Feel free to click on the links below to jump to that part.)
- What is technical debt?
- Why should you care about technical debt?
- Best practices to prevent technical debt
- Be up-to-date
- Code reviews
- Automated testing
- Agile architecture
- Blame-free post-mortems
- Best practices to manage technical debt
- High-interest technical debt first
- Boy Scout rule
- Repay debt while performing valuable customer work
- Just ignore it
- Case study on technical debt
The name is no mistake, as it works in a similar way to financial debt. When you take shortcuts to roll out a product on time, you basically take on debt. If you don’t pay it off before moving on to the next stages of the product, your work will become a lot harder, and you’ll likely be forced to take on even more debt as you move forward.
Basically, the more debt you take on, the more it slows you down, and the more you will have to take on in the upcoming stages of the project.
The term technical debt can basically be applied to any technical solution that is suboptimal in any way. The most important thing to know about it is, it’s unavoidable.
The reason for it is fairly straightforward: whenever there are a number of solutions at hand for a problem, they tend to have different upsides and downsides, so all solutions are suboptimal in some way.
The best engineer teams are not the ones who aren’t stacking up tech debt; such a situation doesn’t exist. The best teams minimize and manage their debt.
We can differentiate between deliberate and unintentional tech debt.
Usually, deliberate debt is better, since that automatically makes the team aware of the situation, while unintentional debt can lurk in a project for a long time, and it could possibly cause serious issues down the line. However, unintentional debt is often unavoidable. This is where you make use of a strong coding culture to manage it.
We can also categorize technical debt as reckless or prudent.
There is no need for a lot of explanation here. Prudent debt is obviously preferable; no one should take on any debt recklessly. Naturally, the two dimensions work together. Ideally, all the debt you take on is deliberate and prudent, while you want to avoid reckless, unintentional debt at all costs.
Developers have to deal directly with technical debt, so while it may make their job easier today, it will certainly make it a lot harder and more time-consuming in the long run. It could be beneficial to take on some technical debt to hit deadlines, but efforts should be made to avoid unintentional and reckless debt.
At first glance, it may seem like clients have no reason to care about the code quality of a product as long as it works, especially when the possible trade-off could be delay and increased development cost. However, a project loaded with technical debt will require far more time, effort and resources to maintain and develop later, the extra cost of which is likely to significantly surpass the initial gains.
The users are also affected by technical debt, even if indirectly. They may not care about the amount of work or money that goes into a software, but they do care about it working reliably, and about new features being added at a rapid pace, both of which could be set back by a lot of technical debt. The happier users are, the happier the client is, and the happier the devs are allowed to be.
The single biggest problem in addressing tech debt is that it’s not really quantifiable. That makes it hard for the developer team to track and for the management to demonstrate to the client how it’s an issue worth spending resources on.
But here are a few things you can do:
It should go without saying that tools, frameworks and libraries should always be kept up-to-date, but it never hurts to state, as it may not be obvious for everyone.
Documenting everything that needs fixing or updating is the most essential step toward making sure they actually get fixed and updated.
If there is technical debt, it’s best to be aware of it and to make sure the team or future developers are aware as well. It reduces the effort required to locate and fix any issues, and if the debt is well-documented, it may even be visible on the business level, causing clients to acknowledge it and provide extra resources to combat it.
Another powerful tool is to regularly have new code reviewed by peers during a sprint. Another set of eyes could catch solutions that may cause problems down the line. It does take some time out of sprints, but it will certainly be worth it in the context of an entire project.
However, code review has its downside too. Developers are often too busy to dig really deeply into others’ codes, so they only really spot glaring mistakes, and nitpicking is likely to lead to tension within the team. So, it can be a powerful tool in minimizing technical debt, but it should be applied carefully. [Gene Mal, CTO at Static Jobs]
Automated testing is an extremely powerful tool too often disregarded. However, when it’s neglected, it becomes incredibly risky and hard to change anything in large bodies of code. It leads to features that can’t be implemented, or only at extreme cost, by putting in disproportional amounts of manpower and working hours.
It is worth the effort to create tests and testable code. It is worth considering to implement practices like test-driven development (TDD), it could get rid of a lot of problems with code.
Agile architecture has a lot of advantages, being more open to changes when in the process of building the software, which is basically guaranteed to happen on any project. However, it does require the code to be flexible and maintainable, so the Agile approach will naturally make developers keep the code in good shape, which helps prevent the accumulation of massive technical debt.
If something goes wrong, it’s a good idea to do some post-mortem research on the issue. In order to be productive, the research should never be about finding a person to blame. Instead, the focus should be on understanding the problem and its cause, so the team can take the necessary steps to prevent the same problem from happening again. [Jason Fennell @ SFELC speech]
Even if you do all the above, and avoid stacking up tech debt as much as possible, you will still have to deal with some. It can’t be avoided, so you should implement practices and processes to keep tech debt at bay.
Not all tech debt is created equal, so you should prioritize what to address, and what not to address, at any given time. Cruft at a part of code that is used and changed often is far more important than cruft at a part that is barely ever used or changed.
It’s high-interest debt because there is a lot of work done around it and based on it. So long as it stays in place, debt hampers all the work, and likely forces more tech debt to be added to other parts of the code. So whenever possible, these issues should be prioritized first, making everything smoother in the long run.
The rule for Boy Scouts is to always leave the campsite in better shape than you found it. It translates to software development in a way to always fix technical debt when you find it.
Naturally, it can’t be done without boundaries, or else it could consume all the time in the world. However, if you set aside a percentage of time in each sprint specifically to fix any tech debt developers may find, it can go a long way towards keeping the product as debt-free as possible. Depending on the situation, anywhere from 5-33% could be a good choice. [Kenneth S. Rubin: Essential Scrum]
Dedicating entire sprints towards fixing technical debt is a bad idea. On the one hand, clients tend to not like it. To them, it can seem like you are spending their time and money to fix something you did wrong in the first place. On the other hand, it also suggests you have done a lot of work encumbered with tech debt already, so you have likely paid higher interest on the debt already than was necessary.
You’re better off designating a maximum amount of time spent repaying technical debt in each sprint, and use it to fix high-priority or happened-upon issues. It keeps the client happy, and it keeps tech debt at a manageable level.
It’s also important to note that technical debt should not always be repaid. When a product is near the end of its life, if it’s built for a short cycle or if it’s a throwaway prototype, technical debt is not a major concern. These instances are few and far between, but you can save some time and effort when they do come up.
Partner and Managing Director
One interesting case study is almost a worst-case scenario where things were in melt-down. (Believe it or not, this is not the worst scenario I’ve seen, but it’s close.)
This was at a large SaaS company that had been trying to solve their own technical debt problem for a little over a year before we were engaged to help them out.
They had a fairly monolithic application based on older versions of their chosen platforms (Python, PHP, and Java). To make matters worse, almost everything was hard-coded, including specific customer customizations that were built into the code. There was literally source code that evaluated “if” statements for hard-coded customer names of the logged-in user, and it did different things based on the result.
The client had taken the approach to try and “boil the ocean” with an all-or-nothing rewrite of the core on the most recent versions of each of the languages. In certain cases, they planned to port some of the legacy PHP code to Python or Java.
What they found was that at every turn, the amount of work and overall complexity of the project kept expanding when they uncovered some new dependency. Meanwhile, product management needed new features and fixes to stay competitive and to keep customers happy.
A year into the project, they had essentially no meaningful progress and no end in sight. In the mean-time, they were at the point where releasing even a bug fix was taking months and even then, it often had to be rolled back as a result of unanticipated effects on other parts of the application.
New features? Forget about it.
Estimates from the development team for even simple things like a new field on the user interface were coming in at 3-4 months (and then missed). This was (as we understood things) at least the second attempt at this.
Needless to say, everyone was unhappy, the culture had turned toxic with blame, and nobody trusted anyone. C-level leadership was looking at the product and development organizations and asking some pretty hard questions. Heads were rolling.
We were hired by a relatively new CTO who had signed on.
Our team came in and did a thorough assessment of the business objectives, situation, architecture, existing technology capabilities, etc. What we came up with was a plan to start with one team in one product area and to utilize a combination of new development processes consisting of a fully cross-functional team. We also implemented the use of micro-services to start isolating and making independent core capabilities of the system.
There was a double bottom line we targeted:
1) Get the people who had been fighting with each other aligned to a common purpose to work together in an environment where they had to support each other instead of back-stab each other.
2) Demonstrate that the technical debt could successfully be dealt with incrementally.
We chose the team with the worst track record to start our work. We applied, among other techniques and tools, what is known as the “Strangler” pattern [Martin Fowler] to identify where to carve stuff out of the monolith, and we used a microservices-based architecture to build the new services. This allowed us to carve out important and isolatable pieces of the application.
Once a new micro-service was completed, legacy code could be modified to use it when that area was touched or ready to be attacked. This broke the problem down into small, independent pieces and allowed an organic evolution to the modernized system.
Another aspect of this project was that all software builds were done manually using tribal knowledge on developer laptops and then copied to testing and production environments (which had their own issues). So, we also had to address the build process and automate this area, as well as part of the overall project.
It took about 3 months to get the first team turned around and productively writing code again. Once this pattern was established and demonstrated success, we continued to roll it out across the other development teams and product areas.
Over time, the product and development organizations learned how to work together instead of against each other, and the level of technical debt incrementally and continuously dropped with each iteration and release. After about 6 months, there were enough independent services and legacy code using the new services that it started to become possible for the company to add new features to the product while continuing to remediate the legacy code, thus further reducing technical debt.
Today, this company regularly releases new and enhanced features in existing applications and has successfully launched two new applications using the micro services built during this exercise into the market. The overwhelming technical debt problem has been resolved and the product development culture transformed. The discipline and capabilities we brought to the teams have been integrated and are now part of the core capabilities of the organization.
They will not find themselves in such an adverse position of overwhelming technical debt again, and now they have the tools and skills to manage technical debt correctly. The CTO who hired us was the only executive who kept his job during a top-down reorganization and was offered a higher-level position with the parent company.
Technical debt is an issue for everyone in the software development industry, but regrettably, there is no silver bullet against it. Keeping it at bay and avoiding it’s potentially crippling effects takes a lot of thorough work, strong coding principles, and well-set-up work processes.
I hope this information helps you to be more prepared the next time you take on a project.