Unleash the power of your data to reduce your technical debt

Since technical debt has become a central part of my recent professional challenges, I started writing a series of installments regarding this topic.

Do not expect any extensive and detailed work in here, if you need it you can get it googling it or from your favourite online content source.

Just expect more a light work crafted from working experiences and practical tips.

So, with no more further ado, let's start.

Do you want me to unleash what?

Data lakes are usually focused to solve problems to business and recently are even focused on using to train machine learning models.

Unfortunately, few attention has been given to the use of data lakes to protect the value of the system either by reducing the technical debt or optimizing current systems.

It might be good to shift your mindset towards the use of your data to build a more robust and resilient system by reducing your technical debt, on how to help your development teams track their advance by using metrics and how to optimize your current systems.

We can use some elements of the paradigm of evolutionary architectures, such as fitness functions, to proactively apply techniques to reduce your technical debt in those architectural dimensions that fits best at your goals in that precise moment.

So, what is technical debt?

Technical Debt was a concept firstly used by Ward Cunningham using similarities with financial debt.

Martin Fowler identified what is considered as technical debt and what is not.

According to that classification, we consider that all what we know as bugs, issues, defects are not technical debt. They are included in the category External Quality.

External Quality includes those features, behaviours etc., the code should be done, but it does not, impacting on customers and business.

Besides defects, External Quality also includes forms of usability and UX.

Then we have the Internal Quality, referring to architectural topics, such as:

Code understandability refers to the fact that the code is not tight to a specific developer and it can easily be modified by anyone.
Code easiness to adapt, build new features. From this point stems an important metric for technical debt: Cost of (future) change..

Cost of Change impacts on the effort, both mental and code, on development time and on the developer expertise or skillset.

As you can all probably figure out at this point, it is easier to manage external quality rather than internal quality.

Taxonomy of technical debt

Also from Martin Fowler, he proposes this definition of the root causes for technical debt.

Deliberate: we are creating technical debt on purpose. This is not a bad thing, per se, the good / bad thing comes from the way we create that technical debt.

Reckless: "we don’t have time for design". See? This is not a good choice. It sounds more like an excuse rather than a justification.
Prudent: "We must ship now and deal with consequences". See again? Everyone is conscious of the not-so-great thing they are doing, but it is ok. They have identified what and when they are doing not-so-perfect. Ideally, this should be tracked to see future impacts in the system and removed whenever possible.

Inadvertent: we are creating technical debt and we do not know that we are creating it. This is the most dangerous form of technical debt, because we do not even know that we are creating it. Anyway, we can approach it in two different ways:

Reckless: Basically, I do not know what I am doing, and I do not care about it. So, there will probably something not-so-great there and I will not know anything about it, and what is worst, I will not care about it.
Prudent: "Now we know how we should have done it". Ok, at least you have your lessons learned.

Best technical debt is that technical debt which is Prudent.

Tip: Tag when a deliberate tech debt is made and carefully monitor it.

Common misunderstandings

Defects are not technical debt. Defects are just symptoms of tech debt.
Data lakes might not only contain business information, but IT info as well.

Get to know your technical debt

First thing to keep in mind, do not try to remove your technical debt, just keep it under control and live with it.

Technical debt always happens, event in the best teams, so, we should be ready to deal with it.

Some tips on how to spot where are your flaws:

If there are lots of bugs localized in just one module, class, etc, this is mot likely a sign of technical debt.
Browsing through your development version control (e.g. Git, Mercurial, etc...) you can identify instabilities, underperformances, due to release number and commits, development made.

How to live with your technical debt

As we mentioned before, technical debt is going to be there, always, in one form or another, so the best way to live with it is to develop strategies and procedures to keep it identified and under control.

One way to control it is by leveraging your data architecture to provide data coming from your systems, from development (i.e VCS) and operations (i.e. monitoring, tracing, logging) enriching your business metrics.

This way you might measure the impact your systems have on your business.

Using fitness functions from evolutionary architectures, we should be able to define priorities (i.e. metrics, goals) for the architecture, thus controlling the technical debt.

"An architectural fitness function provides an objective integrity assessment of some architectural characteristic(s)"