DSC Weekly Digest 15 March 2021

Twenty years ago, a group of programmers, project managers, and consultants met at a ski resort just outside of Ogden, Utah. What they came in with was a host of issues and problems that they had with the waterfall methodology and how code was created. What they left with was a manifesto for a new way to build software: Agile.

Since then, the Agile Movement has reframed how code gets written pervasively, and it can be argued that the methodology has had a transformative effect on the entire information technology industry.

At the same time, however, momentum in that industry has been quietly but steadily shifting away from the nexus of programming and towards the transformation, processing, analysis, and ultimately reuse of data. These are not the same things.

Programming ultimately is about creating tools. For a while, those tools were “the product”, but the reality for most organizations is that once they have a usable hammer, the need for a hammer with newer features or a better user interface is just not a strong imperative. Agile may help you create a better hammer, but if at the end of the day your goal is to put together a house, the benefits of an iteratively improved, high impact, high durability device is limited – especially when a dozen other people in the area are also working to create better hammers.

Data agility plays by a different set of rules. Data, if you think about it, is the record of processes once they are done. It is the manifold of signals from the environment, interpreted and collated to be useful. While that data accumulates over time, the value of that data comes in its ability to be combined with other data, whether the data in question are the numeric tracings of sensors, the contents of a book, or the intricacies of a model.

Data agility can then be thought of as making data accessible to those people who need it. This isn’t just about databases – it’s about ensuring that the data is structured for the best utility at all stages in its lifecycle. Increasingly, data also drives processes, which is ultimately why DevOps and Process Automation are one of the hottest areas in IT right now. It is also why discussions about what comes next in Agile are so pertinent. It is not the tools that we use that are important, but what we construct using data as the material transfigured through these tools that ultimately becomes the major test.

Join us for articles this week on Agile 2, MLDeveOps and more. This is why we run Data Science Central, and why we are expanding its focus to consider the width and breadth of digital transformation in our society. Data Science Central is your community. It is a chance to learn from other practitioners, and a chance to communicate what you know to the data science community overall. I encourage you to submit original articles and to make your name known to the people that are going to be hiring in the coming year. As always let us know what you think.

In media res,
Kurt Cagle
Community Editor,
Data Science Central