Recommender systems are lifesavers in the infinite seething sea of e-commerce, improving customer experience. Recommender engines are eliminating the tyranny of choice, smoothing the way for decision-making, and boosting online sales. Moreover, ubiquitous AI technologies are sneaking into e-commerce, too, not…

Meet two text mining experts in today’s interview, which explores some of the common issues faced by data scientists in text analytics. Prof. Dursun Delen and…

The new era of BigData and advances in technology have made significant transitions towards the high functionality of IoT devices. The popularity of IoT devices has led to more easier methods for BigData collection, analysis, and distribution at a rapid rate. According to a report by…

I propose a new word for data science for sparking new thinking. Signuology is defined as the study of sets of characteristic predictive signals contained within data in the form of combined features of the data that are characteristic of an observation of interest within the data.

The terms data mining and data structure imply rigid and discrete characteristics. A signal has more flexibility, borrowing from ideas contained in the superposition principle in physics. One can take the…

The key to perform any text mining operation, such as topic detection or sentiment analysis, is to transform words into numbers, sequences of words into sequences of numbers. Once we have numbers, we are back in the well-known game of data analytics, where machine learning algorithms can help us with classifying and clustering.

We will focus here exactly on that part of the analysis that transforms words…

Trends to Watch Out For and Prepare Yourself

Ok. So there’s been a lot of coverage by various websites, data science gurus, and AI experts about what 2019 holds in store for us. Everywhere you look, we have new fads and concepts for the new year. This article is going to be rather different. We are going to highlight the dark horses – the trends that no one has thought about but will completely disrupt the working IT environment (for both good and bad – depends upon which side of…

The short answer is: "No."

Source: LEGO

I started teaching myself programming in my 40s, and I am a strong advocate that everyone should learn to code. Even if you have no intent to become a developer or a full-stack data scientist, coding teaches you a couple valuable lessons:

• It disciplines you to…

Introduction

Artificial Intelligence is growing at a rapid pace in the last decade. You have seen it all unfold before your eyes. From self-driving cars to Google Brain, artificial intelligence has been at the centre of these amazing huge-impact projects.

Artificial Intelligence (AI) made headlines recently when people started reporting that Alexa was laughing unexpectedly. Those news reports led to the usual…

Summary:  Advanced analytics and AI are the fourth great lever available to create organic improvement in corporations.  We’ll describe why this one is different from the first three and why the CEO needs the direct help of data scientists to make this happen.

If you’re a CEO or any other flavor of top executive leading a…

Python and R are the two most commonly used languages for data science today. They are both fully open source products and completely free to use and modify as required under the GNU public license.

But which one is better? And, more importantly, which one should you learn?

Both are widely used and are standard tools in the hands of every data scientist.

The answer may surprise you – because as a professional data scientist, you should be ready to deal with…

One interesting metric to check the  usefulness of Everipedia as a desk reference for data mining is to compare the number of relevant articles. Go to Everipedia (https://everipedia.org/) and search for "data mining". You will get 7 articles.Then go to Wikipedia and search "data mining" You will see 4 articles (overlapped with similar Everipedia  articles).

Another example. Try the word "smoothing" which is a popular topic in data analysis.…

R has spread deep into the private sector and can be found in the production pipelines at some of the most advanced and successful enterprises. 

Learn the fundamentals of data analysis in the second edition of Data Analysis with R, authored by data scientist…

Data mining is the process of looking at large banks of information to generate new information. Intuitively, you might think that data “mining” refers to the extraction of new data, but this isn’t the case; instead, data mining is about extrapolating patterns and new knowledge from the data you’ve already collected.

Relying on techniques and technologies from the intersection of database management, statistics, and machine learning, specialists in data mining have dedicated their…

A long, long time ago (maybe 10 years) the data analytics industry was fairly easy to define and track. Back in that pre-historic era SAS was considered the gold standard of analytics companies with a comprehensive range of solutions addressing the demands of many industries.  Given the relative paucity of data, analytics tended to focus on those industries that generated usable data.  Companies that were part of the analytics universe back then would have included:

• risk…

Let’s start with the bottom line - there is no excuse for virtually any company today, regardless of size or manpower (and within reason), not to be making data analyics a part of their normal business routines. Traditional objections such as cost, resources and expertise no longer cut the mustard.  As many observers have noted, a company’s internally generated data is a key asset that needs to be leveraged in the same way as any other corporate asset if the…

One of the best ways to learn about any topic is start with very fundamental questions like What, Why etc? Good old Socratic method. In this series of articles on data mining, I plan to approach this topic in a similar fashion. 

What is Data Mining?

Simply put, Data mining is the process of sifting through large data sets to identify…

Variable reduction is a crucial step for accelerating model building without losing the potential predictive power of the data. With the advent of Big Data and sophisticated data mining techniques, the number of variables encountered is often tremendous making variable selection or dimension reduction techniques imperative to produce models with acceptable accuracy and generalization. The temptation to build an ecological model using all available information (i.e., all variables) is hard to…

Data is almost everywhere. The amount of digital data that currently exists is now growing at a rapid pace. The number is doubling every two years and it is completely transforming our basic mode of existence. According to a paper from IBM, about 2.5 billion gigabytes of data had been generated on a daily basis in the year 2012. Another article from Forbes informs us…

This post covers the following tasks using R programming:

• cleans the texts,
• sorts and aggregates by publisher names
• creates word clouds and word…

Introduction

There’s a lot of buzzword around the term “Sentiment Analysis” and the various ways of doing it. Great! So you report with reasonable accuracies what the sentiment about a particular brand or product is.

Opinion Mining and Sentiment Analysis

After publishing this report, your client comes back to you and…