So what were the answers popping in your head ? Random forest, SVM, K means, Knn or even Deep Learning and its variants ?
Now some of you might laugh and say how on earth can you predict so far ahead, predicting things 100 yrs into future is crazy.
Well the answer is Lindy effect. Yes, the heuristic I am using to predict this is Lindy Effect.
Alright next logical question would be what is Lindy Effect?
Lindy Effect
Wikipedia defines the Lindy effect as follows
The Lindy effect is a concept that the future life expectancy of some non-perishable things like a technology or an idea is proportional to their current age, so that every additional period of survival implies a longer remaining life expectancy.
One of my favorite author Nassim Taleb in his famous book Antifragile: Things that gain from disorder defines Lindy Effect as follows.
If a book has been in print for forty years, I can expect it to be in print for another forty years. But, and that is the main difference, if it survives another decade, then it will be expected to be in print another fifty years. This, simply, as a rule, tells you why things that have been around for a long time are not “aging” like persons, but “aging” in reverse. Every year that passes without extinction doubles the additional life expectancy. This is an indicator of some robustness. The robustness of an item is proportional to its life!
His article on Lindy Effect ‘An Expert called Lindy’ is a highly recommended read.
Well, because it has survived this long. Regression (method of least squares) as a concept was first invented in the 1800’s by Carl Friedrich Gaussand Adrien-Marie Legendre. They used it to determine the orbital paths of planets and other bodies around the sun.
The word ‘Regression’ was coined by Francis Galton to describe the observation that the taller fathers tend to have relatively shorter sons while shorter fathers tend to have relatively taller sons!!
Okay so quite clearly Regression has been around for more than 200 years already !! so going by the Lindy Effect heuristic it will last another 200 years. So in fact I might be little conservative in saying that Regression will continue to be in use in year 2118.
The case in point would be the below report from a 2016 Kdnuggets survey
In fact in another survey conducted by kdnuggets in 2011, Regression came a close second. So going by the Lindy effect it has become ‘more immortal’ in 5 years by topping the charts !!
Regression is still the widely used ML algorithm. People are using Regression or continue to use Regression because
People across various domains continue to use Regression because it has worked for them . There is a clear ROI that people have gained by using Regression. For example, In marketing, The driving force behind Market Mix Modeling is Regression. It is still a popular technique and many FMCG companies believe the outputs from MMM. The same holds true for other domains too.
Well so far the Lindy Effect has not been Kind to Neural Nets or lets call it AI. It has already faced AI Winter. The longevity of Neural Nets and its variants have been hampered by ‘AI Winter’ in the 20th century. Such disruption is not a good sign for the longevity of technology or in this case algorithms.
But on the brighter side AI related advancements has grown from strength to strength in last decade. And I as an eternal student, continue to be fascinated about the latest AI breakthroughs. So a safe bet could be that we could see Neural Nets and its variants to survive another 10–20 years, with the hope that ‘Singularity’ fear expressed by Elon Musk does not cause another AI winter.
What can Mitigate Lindy Effect of a Machine Learning Algo?
Machine Learning Overkill : Yes, Lindy effect will get mitigated because of wrong application of machine learning algorithms and overkill of it. I have come across situations where people have used a Machine Learning algorithm where a simple common sense baseline approach would have worked. Mr. Rama Ramkrishnan captures this essence excellently in his article .
The recent fad of Data Science being the sexiest job is not helping the cause either. The machine learning algorithms have become like a hammer at the hands of data scientists. Everything looks like a nail to be hit upon. In due process the wrong application or overkill of machine learning will cause disenchantment among people when it does not deliver value. It will be a self inflicted ‘AI Winter’.
But for time being Regression will have the last laugh now and probably even in the year 2118.
Hope you liked my article. You can also comment below your opinions about the article.
You can reach out to me on
Comment
Thanks for your comments Omar. Yes, I do agree with you . The use cases may be simpler but time and again Regression 'Works' when compared to other ML Algorithms. The interpret-ability factor also plays a crucial role in the popularity of Regression. Most of the modern use cases are utilizing other ML algorithms / Neural networks but the fact that the cost (both in terms of data and money) far out weighs the result, does not make it popular. It basically all boils down to the ROI, especially in Industry.
I think the popularity of regression is also a function of the complexity level of problems its designed to solve. Since most of the current use cases are not highly complex and/or attempting to mimic human intelligence, there numbers will remain smaller relative to the regression. So regression is not going anywhere soon.
Thank You William. Glad to know a Data Science Stalwart like you liked my article and to an extent agrees to the point of view expressed in the article.
Enjoyed your article and agree that regression is likely to continue to be relevant for a long time. And while I don't see it being challenged directly, the Top 10 Algo chart deserves some comment. Regulated industries, specifically lending and insurance were the earliest adopters of predictive analytics and still represent the largest number of models and probably data scientists. They are constrained to 'explainable' algos(regression and simple decision trees) and I think that is a significant driver behind their statistical dominance.
Looking forward the open question is how common will embedded AI become that can compete with this current market dynamic. It's possible that this could upset the popularity scale moving deep nets up the chart, but not replace the utility of regression. It's also possible that all of data science applications will end up as a middle layer described as 'systems of intelligence' and become more or less invisible to end users.
© 2019 Data Science Central ® Powered by
Badges | Report an Issue | Privacy Policy | Terms of Service
Most Popular Content on DSC
To not miss this type of content in the future, subscribe to our newsletter.
Other popular resources
Archives: 2008-2014 | 2015-2016 | 2017-2019 | Book 1 | Book 2 | More
Most popular articles
You need to be a member of Data Science Central to add comments!
Join Data Science Central