A.I. based automated Anomaly detection system is gaining popularity nowadays due to the increase in data generated from various devices and the increase in ever evolving sophisticated threats from hackers etc. Anomaly detection systems can be applied across various business scenarios like monitoring financial transactions of a fintech company, highlighting fraudulent activities in a network, e-commerce price glitches among millions of products, and so on. Anomaly detection system can work…

Cost driver: One of the key drivers of cost for ecommerce businesses is the last mile delivery charges. Consumers have the option to switch between e-retailers depending on their willingness to pay delivery charges, which is generally significantly low. This notion puts the power into the hands of the customer. Delivery charges, when added just before payment, makes the customer rethink and is one of the reasons for…

1. Background

New home construction plays a significant role in housing economy, while simultaneously impacting other sectors such as timber, furniture and home appliances. New house sales is also an important indicator of country’s overall economic health and direction. In the last 50 years there has been few significant bumps and turning points in this sector that shaped the trajectory of the overall economy.  Here I review the…

In a previous blog I wrote about 6 potential applications of time series data. To recap, they are the following:

1. Trend analysis
2. Outlier/anomaly detection
3. Examining shocks/unexpected variation
4. Association analysis
5. Forecasting
6. Predictive analytics

Here I am focusing on outlier…

A time series is a sequence of data points recorded at specific time points - most often in regular time intervals (seconds, hours, days, months etc.). Every organization generates a high volume of data every single day – be it sales figure, revenue, traffic, or operating cost. Time series data mining can generate valuable information for long-term business decisions, yet they are underutilized in most organizations. Below is a list of few possible ways to…

Missing data present significant challenges to trend analysis of time series. Straightforward approaches consisting of supplementing missing data with constant or zero values or with linear trends can severely degrade the quality of the trend analysis, which significantly reduces the reliability of the trend analysis. …

Any time series classification or regression forecasting involves the Y prediction at 't+n' given the X and Y information available till time T. Obviously no data scientist or statistician can deploy the system without back testing and validating the performance of model in history. Using the future actual information in training data which could be termed as "Look Ahead Bias" is probably the gravest mistake a data scientist can make. Even the sentence “we cannot make use future…

Digging through messy data and doing numerous calculations just so you can submit a report or arrive at the result of your quarterly business development can sometimes be nigh impossible. After all, we are only human, and by the time we get to the other side of our spreadsheet equation, we have lost all sense of what we were trying to accomplish.

Luckily, there…

Visual Analytics and Data Discovery allow analysis of big data sets to find insights and valuable information. This is much more than just classical Business Intelligence (BI). See this article for more details and motivation: "Using Visual Analytics to Make Better Decisions: the Death Pill Example". Let's take a look at important characteristics to choose the right tool for…

Complex stuff! However, we can start from the basics , gain confidence in understanding our dataset and utilise whatever has already been done by other stalwarts! 

We have a 11 years monthly data on milk procurement by more than 100 milk co-ops spread across India. This contains their geographic location, quality of milk procured,  price received , products made etc etc. This is a seasonal industry  in India affected by weather, prices and in general crafty management acumen…

Batch data processing is an efficient way of processing high volumes of data is where a group of transactions is collected over a period of time. Data is collected, entered, processed and then the batch…