Summary:  Bias in modeling has long been a public concern that is now amplified and focused on the disparate treatment models may cause for African Americans.  Defining and correcting the bias presents difficult issues for data scientists that need to be carefully thought through before reaching conclusions.

 …

After being confined for 2 months and hearing the word pandemic almost daily, seeing graphs, opinions, time lines and values that rise and fall, I became curious. Firstly, what is a pandemic? Is it the same as an epidemic? A plague? Was the plague an epidemic? What do epidemiologists study? Yes, they study a discipline called epidemiology.

What is epidemiology?

Epidemiology studies the distribution, frequency, relationships, predictions, and…

How to maintain Model Effectiveness after deployed

When we ready to deploy a good predictive model which given a good accuracy score on training and testing dataset, there is one more problem to solve.  How long will this model effectively solve the problem with the same high accuracy and what is the strategy to maintain the model accuracy. We also need to…

This is a longer form followup to my post describing the open source pandemic package on PyPI (with Python code also available on Github). You can use the code to simulate millions of people moving about in two dimensions, crossing paths and, unfortunately, getting sick.

This video illustrates the dynamic with a toy sized town of fifty people. Watch how transmission takes place. You might even discern commuting and households.…

Machine Learning (ML) development is an iterative process in which the accuracy of predictions made by the models is continuously improved by repeating the training and evaluation phases. In each of these iterations, certain parameters are tweaked continuously by developers. Any parameter manually selected based on learning from previous experiments qualify to be called a model hyper-parameter. These parameters represent intuitive decisions whose value cannot be estimated from data or from…

Summary:  Data Scientists from Booking.com share many lessons learned in the process of constantly improving their sophisticated ML models.  Not the least of which is that improving your models doesn’t always lead to improving business outcomes.

The adoption of AI/ML in business is at an inflection point. …

Summary:  Finally there are tools that let us transcend ‘correlation is not causation’ and identify true causal factors and their relative strengths in our models.  This is what prescriptive analytics was meant to be.

Just when I thought we’d figured it all out, something comes along to make…

Summary:  Whether you’re a data scientist building an implementation case to present to executives or a non-data scientist leader trying to figure this out there’s a need for a much broader framework of strategic thinking around how to capture the value of AI/ML.

There are many articles written from a tools…

Research fields usually follow the practice of categorizing continuous predictor variables, and they are the same who mostly use ANOVA. They often do it through median splits, the high value above the median and the low values below the median. However; this it seems is not that good an idea, and enlisted are some of the reasons to it:

• Median tends to vary from sample to sample. This makes the categories in various samples have various…

I have been writing about the Crosswave Differential Algorithm for a number of years. I described in previous blogs how the algorithm emerged almost by accident while I was attempting to write an application intended to support quality control. In this blog I will be discussing the event model that powers the algorithm. Events are the details and circumstances…

Introduction

There’s a lot of buzzword around the term “Sentiment Analysis” and the various ways of doing it. Great! So you report with reasonable accuracies what the sentiment about a particular brand or product is.

Opinion Mining and Sentiment Analysis

After publishing this report, your client comes back to you and…

Relation, Relationship and Association

While most players in the IT sector adopted Graph or Document databases and Hadoop based solutions, Hadoop is an enabler of HBase column store, it went almost unnoticed that several new DBMS, AtomicDB previous database engine of …

Currently, many of us are overwhelmed with mighty power of Deep Learning. We start to forget about humble graphical models. CRF is not so trendy as LSTM, but it is robust, reliable and worth noting.

In this post, you will find a short summary about CRF (aka Conditional Random Fields) – what is this thing, what is it for and some interesting facts. Enjoy!…

Regression is the first technique you’ll learn in most analytics books. It is a very useful and simple form of supervised learning used to predict a quantitative response.

Originally published on Ideatory…

"Look, they are one people, and they have all one language; and this is only the beginning of what they will do; nothing that they propose to do will now be impossible for them.  Come, let us go down, and confuse their language there, so that they will not understand one another's speech."  (Genesis 11-6,7)  On the distance between expression, meaning, and action resulting from growth of populations and…

This article was first posted in 2014 but the message bears repeating.  There is a lot being written about tools simple enough for the citizen data scientist to operate.  The unstated constraint is that if you don't have significant experience in data science then these will always be "good enough" models.  The problem is that 'good enough' models under achieve both revenue and profit.  Very small increases in model fitness can translate into much larger increases in campaign ROI.  Business…

I was often the lone wolf among my peers in university because I supported a prominent place in society for corporations and an important social role for capital. I questioned whether the directors and executives of companies entered into boardrooms really intending to “oppress” people such as minorities and people with disabilities. Did they deliberately make bathrooms inaccessible to people in wheelchairs perhaps to advance their preconceptions of who gets to go to the bathroom, I pondered…

I often find myself following missing-persons cases. I am interested in the reasoning behind the use of resources. There can be major deployments of capital during investigations. I tend to wonder which scenarios trigger more spending than others. I also recognize the visceral side of missing-persons cases. There are publicly accessible databases of missing persons in Canada, the United States, and I am certain many other countries. I imagine that it can be difficult to regard such…