Since technical debt has become a central part of my recent professional challenges, I started writing a series of installments regarding this topic.
Do not expect any extensive and detailed work in here, if you need it you can get it googling it or from your favourite online content source.
Just expect more a light work crafted from working experiences and practical tips.
So, with no more further ado, let's start.
Data lakes are usually focused to solve problems to business and recently are even focused on using to train machine learning models.
Unfortunately, few attention has been given to the use of data lakes to protect the value of the system either by reducing the technical debt or optimizing current systems.
It might be good to shift your mindset towards the use of your data to build a more robust and resilient system by reducing your technical debt, on how to help your development teams track their advance by using metrics and how to optimize your current systems.
We can use some elements of the paradigm of evolutionary architectures, such as fitness functions, to proactively apply techniques to reduce your technical debt in those architectural dimensions that fits best at your goals in that precise moment.
Technical Debt was a concept firstly used by Ward Cunningham using similarities with financial debt.
Martin Fowler identified what is considered as technical debt and what is not.
According to that classification, we consider that all what we know as bugs, issues, defects are not technical debt. They are included in the category External Quality.
External Quality includes those features, behaviours etc., the code should be done, but it does not, impacting on customers and business.
Besides defects, External Quality also includes forms of usability and UX.
Then we have the Internal Quality, referring to architectural topics, such as:
Cost of Change impacts on the effort, both mental and code, on development time and on the developer expertise or skillset.
As you can all probably figure out at this point, it is easier to manage external quality rather than internal quality.
Also from Martin Fowler, he proposes this definition of the root causes for technical debt.
Deliberate: we are creating technical debt on purpose. This is not a bad thing, per se, the good / bad thing comes from the way we create that technical debt.
Inadvertent: we are creating technical debt and we do not know that we are creating it. This is the most dangerous form of technical debt, because we do not even know that we are creating it. Anyway, we can approach it in two different ways:
Best technical debt is that technical debt which is Prudent.
Tip: Tag when a deliberate tech debt is made and carefully monitor it.
First thing to keep in mind, do not try to remove your technical debt, just keep it under control and live with it.
Technical debt always happens, event in the best teams, so, we should be ready to deal with it.
Some tips on how to spot where are your flaws:
As we mentioned before, technical debt is going to be there, always, in one form or another, so the best way to live with it is to develop strategies and procedures to keep it identified and under control.
One way to control it is by leveraging your data architecture to provide data coming from your systems, from development (i.e VCS) and operations (i.e. monitoring, tracing, logging) enriching your business metrics.
This way you might measure the impact your systems have on your business.
Using fitness functions from evolutionary architectures, we should be able to define priorities (i.e. metrics, goals) for the architecture, thus controlling the technical debt.
"An architectural fitness function provides an objective integrity assessment of some architectural characteristic(s)"
This has been just a brief overview on technical debt and a different way of using your data to serve more IT goals.
Since this is a broad topic, next installments will go deeper in more specific topics, such as:
I would really appreciate your feedback on this (or in any other specific topics you might be interested) and thanks in advance.