Subscribe to DSC Newsletter

Data Science: What goes around comes around

Many, many moons ago I began my first data science project. It involved a search algorithm for selecting the most probable mode of inheritance for a congenital anomaly given 10,000 two-generation pedigrees. Given the complexity of the algorithm I had to do my own coding: in FORTRAN ! It was tedious to say the least but I felt that I thoroughly knew the ins and outs of the methodology as a consequence writing code.

After that project I discovered a new analytical tool that promised to alleviate most of the required coding. It was called SAS (at the time the User Guide was a single book), and it took care of my needing to write much code. As SAS proliferated so did my use of it. Soon I was exclusively using SAS for all of my analytics.

Then a paradigm shift occurred. Machine learning offered a new way to approach previously intractable problems. Off the shelf niche programs were initially used but eventually got supplanted by offerings from SAS and other analytic titans. I was so hooked on writing minimal code that I continued using canned programs even though they left a lot to be desired.

Eventually I felt so stymied by canned code that I began to look at MATLAB. True, that product could also be used through minimal coding, but just learning a bit of MATLAB programming dramatically increased its capabilities.So out went the Stanley toolbox software, replaced by MATLAB. Unfortunately, MATLAB's neural network and optimization toolboxes were prohibitively expensive. I wrote a genetic algorithm in MATLAB but it felt "coarse" and not very extensible.

I had an epiphany: what if I went back to doing most of the coding myself. I could construct algorithms exactly the way I wanted them along with a nice GUI. There was a buzz about a language called Python that seemed to be gaining traction in the data science community. Fast forwarding, I am where I was many, many moons ago; writing the majority of the code (in Python, not FORTRAN) for my analytic methods. Yes computers are incredibly faster than when I started out and more data can be stored on a flash drive than on a hard drive of yesteryear. But at the bottom line is the fact that if you want to do serious data mining, then rolling up one's sleeves and writing code is unavoidable.

 

 

Views: 402

Comment

You need to be a member of Data Science Central to add comments!

Join Data Science Central

Videos

  • Add Videos
  • View All

© 2019   Data Science Central ®   Powered by

Badges  |  Report an Issue  |  Privacy Policy  |  Terms of Service