Summary: Simpson’s Paradox. A source of risk for real time analytics and for the citizen data scientist.
Most of us practicing the predictive arts know to look for sources of bias in our data. There are seven that are common, the first six of which are:
Guest blog post by Vijay Rajan.
As catchy as the title of this post is (which I often do to get audience attention, approvals or argue), the real title should be "What moved the median?". This is a long read compared to my earlier post because I drill down really deep into…Continue
Added by Vincent Granville on August 31, 2015 at 7:30am — No Comments
NASA is using big data to make complex knowledge more readily available. Learn how graph visualization can help turn large corpus of documents into concrete insights.
Even in a mature and knowledge-driven organization like NASA, finding an answer to a common business issue can be frustrating. Past surveys at NASA have shown that most people have trouble finding the…Continue
Open data repositories are valuable for many reasons, including:
(1) they provide a source of insight and transparency into the domains and organizations that are represented by the data sets;
(2) they enable value creation across a variety of domains, using the data as the “fuel” for innovation, government transformation, new ideas, and new businesses;
(3) they offer a rich variety of data sets for data scientists to sharpen their data mining, knowledge discovery, and…Continue
Added by Kirk Borne on August 30, 2015 at 2:09pm — No Comments
Last year, I wrote a blog on "mass data assignments." For readers that lack a prototype or application to handle data using mass data assignments, the topic probably seems a bit evasive. In this blog, I will be reinforcing and developing…Continue
Added by Don Philip Faithful on August 29, 2015 at 5:46am — No Comments
On September 2nd, 2015, President Peña of Mexico will give his 4th State of the Country address to 120 million Mexicans (who will actually watch is another matter entirely). Given these troublesome times in terms of economy, and our endemic problems as a country, like drug traficking, corruption and violence, it will be a message worth observing and analyzing.
To this end, a few independent news outlets have taken upon themselves to embark on a live 'fact-checking' event.…Continue
Check out this infographic from XO Communications about 2016 being the year of the Zettabyte.
Added by Sheldon Smith on August 28, 2015 at 10:00am — No Comments
Ultraviolet light, when dispersed through a black light, allows us to see beyond the spectrum of the light we’re used to seeing. Ultraviolet data* works the same way – it is the data that your company is probably capturing but might not be apparent at first glance. This data is not available for analysis or insight until you find a way to “see” it.
In many cases, ultraviolet data relates to customer or user behavior, whether online or more physical, and might be as simple…Continue
Previously, we discussed the role of Amazon Redshift's sort keys and compared how both compound and interleaved keys work in theory. Throughout that post we used some dummy data and a set of Postgres queries in order to explore the…Continue
Added by sasha blumenfeld on August 28, 2015 at 7:20am — No Comments
Guest Blog by Jose Dianes at R-Data Science
The purpose of many data science projects is to end up with a model that can be used within an organisation to solve a particular problem. If this is our case, we need to determine the right representation of that model so it can be shared in the easiest, cheapest, and most effective way. Web data products are an ideal…
Statistical conclusions that do not make sense are regrettably a majority of the outcomes on large unconstrained data problems. Yet these large divergent problems sometimes settle over time into something that may make sense eventually. This can leave the data tool box full of scrubbing tools, regressions, culminations, and learning algorithms obsolete. Replaced with common normal growth estimates and a demand for clear hindsight into why the data exists and what…Continue
Added by Sigmond Axel on August 28, 2015 at 4:47am — No Comments
I’m a big fan of statistics. Other than being fun to play with and fun to illustrate, they serve a lot of important tasks for researchers. They can quickly identify which of 500 comparisons is statistically significant. They can offer data to show whether your brand users comprise 2 distinct groups of people or 7 distinct groups of people. They can offer data to show which price your consumers would refuse to pay.
But there are two ways to use statistics. The right way and the wrong…Continue
The analytical scene has recently been dominated by the prediction that we would soon experience an important shortage of analytical talent. As a response, academic programs and massive open online courses (MOOCs) have sprung up like mushrooms after the rain, all with the purpose of developing skills for the analyst or its more modern counterpart, the data scientist. However, in the …Continue
Added by Geert Verstraeten on August 27, 2015 at 11:57pm — No Comments
Originally posted by Avinash Kaushik - the Google data science evangelist. The list below shows 8 of them.
#12: Almost all reporting is off custom reports.
#11: Close to zero…
Added by Data Geek on August 27, 2015 at 8:30pm — No Comments
These authors are leading bloggers on various niche data science and big data publisher websites. Very few of these bloggers are posting on more than one website.
The authors below are displayed in a totally random order. …Continue
This is not a comprehensive list. Their data science lab was conducting a survey about defining big data. They asked many leading practitioners to provide a definition earlier this month, below are those who accepted / found the time to respond. The order in the Berkeley list below is random, I believe. …Continue
Added by Data Geek on August 27, 2015 at 8:00pm — No Comments
Interesting article recently posted here, where the DSC top LinkedIn group is listed as #1 (it is now approaching 200,000 members).
Here are the 10 largest…Continue
Added by Data Geek on August 27, 2015 at 7:30pm — No Comments
Sometimes I don’t trust Data Science, probably because my duty of care is more pronounced on account of working mostly in Legal Analytics. You see as an Analytics Practitioner in the Legal field my Data Science methodology cannot afford to yield…Continue
Added by Mkhuseli Mthukwane on August 27, 2015 at 7:35am — No Comments
A lot has been said about the value of data viz, but the folks at R2D3 have truly taken this to a whole new level by using very sophisticated but also very intuitive data viz techniques to teach the basics of machine learning. I was really blown away by the way the step-by-step visualizations on this page lead the reader through all the intuitive steps to arrive at a pretty clear understanding of machine learning, in this case focusing on decision trees.
If you are an experienced…Continue
This is not about attacking a guy - a friend of mine - who, at first glance, seems extremely overpaid, like any top executive. Indeed, the question is about whether data scientists should be coders (spending 50% to 100% of their time writing code) or not.
I believe the answer is negative. There are…Continue