For many scientists and data analysts, outliers are like a ‘black box’ in conventional statistics. Many believe that these outlier observations arise due to errors or due to improper procedures in the experiment. Majority of them eliminate the outliers unscientifically by brute force. Some identify them statistically but discard them as if they are junk. Some understand importance of the outliers but they do not know how to deal with it. If you are one among them or interested in scope of…

When the performance of an employee is evaluated, ideally there are no externalities to complicate the analysis. If the employee has a computer that is constantly freezing up - or the servers in the company frequently operate slowly - the employee's performance data will reflect the functionality and effectiveness of these systems. If the company occupies a highly competitive market, declining sales data is attributable at least in part to competition rather than the behaviours of employees.…

The first computer program that I encountered mimicking or emulating human interaction through language was called "Eliza." The version that I knew ran on the Commodore PET. It communicated in English. Eliza made comments that made some sense but which indicated lack of understanding of the conversation. If a person mentions "mother," Eliza might…

When I read a blog, I often find myself in deep thought as I approach the end, trying to determine if the author has said anything that I might be able to use. A blog doesn't have to say anything. Nor does it have to be useful to me specifically. It might simply offer a personal reflection on life. As a person who also writes blogs, I…

Skills, skills, skills!!! Which ones should I learn? Which ones do I need to land the job, to impress the client, to prepare for the future, to stay relevant? What programming languages should I learn? What technologies should I master? What business books should I read? Is there a course I can take, or a certification I can enroll in? Should I focus on being a…

Given the nature of the community, presumably many visitors already have a strong understanding of the nature of quantitative data. Perhaps more mysterious is the idea of qualitative data especially since it can sometimes be expressed in quantitative terms. For instance, "stress" as an internal response to an externality differs from person to person; yet it would be possible to canvas a large number of people and express stress levels as an aggregate based on a perceptual gradient: minimal,…

Being the son of a mechanic, I have spent many years handling power tools. I'm especially fond of a couple of hammer-drills in my possession. They can effortlessly drill holes through concrete. At least, this is what my father once claimed. He handed down his most treasured tools to me. I'm big on pliers and screwdrivers. This might be due to my vocational training as a technician. Even today - long after I completed my diploma and continued to further my education - I still carry a licence…

We've given Hadoop almost 10 years to mature, invested billions, and very few companies are  seeing the return on investment.  Several companies have tried to make Hadoop a real-time analytical platform, incorporating SQL-like facades on top, but the latency is still not where it needs to be for interactive applications.  Even Google, a true big data user, has moved on and is using more dataflow / flow-based programming approaches.    Why?  It just makes sense...

• Why should I…

The idea of environmental determinism once made a lot of sense. Hostile climates and habitats prevented the expansion of human populations. The conceptual opposite of determinism is called possibilism. These days, human populations can found living in many inhospitable habitats. This isn't because humans have physically evolved. But rather, we normally occupy built-environments. We exist through our technologies and advanced forms of social interaction: a person might not be able to build a…

This blog is about the peculiar nature in which software sometimes gets developed. I hope that many readers will recognize the relevance of data science in the examples taken from my own projects. I propose that development is the product of creativity more than accreditation. Creativity is something complicated that interacts with a person over his or her life circumstances. Many people know how to write . . . sentences and paragraphs. However, the ability to write well does not necessarily…

In this post, I discuss the basic characteristics of code that I have personally used to extract online data - in a process these days often called data-mining.  I intend to cover some general features.  Those that wish to do so can also compile the coding samples.

Over the years, I have programmed in a number of computer programming languages including Visual Basic, Perl, Python, and LISP (AutoLISP).  The coding samples on this blog are written in Java, my language of…