Here are some of my favorite things about big data and data science, from A to Z (actually, ZZ):
C – Characterization
Are you looking for an exciting career opportunity that is just as paying as it is desirable? Harvard Business Review calls Data Scientists are the sexiest jobs of the 21st century. Data Scientist term coined when two people, DJ Patil and Jeff Hammerbacher, were trying to name their data team working on big data and did not want to limit their…Continue
Added by Vincent Granville on March 13, 2014 at 5:30pm — No Comments
Update: The most recent article on this topic can be found here.
All the regression theory developed by statisticians over the last 200 years (related to the general linear model) is useless. Regression can be performed as accurately without statistical models, including the computation of confidence intervals (for estimates, predicted values or…Continue
I received a call from an old client who stated his analytics team had a recent string of failures alarming the firm and costing money. He asked me to review and audit the teams work and analytical processes in attempt to understand and remedy the failures. The data crunching technology was…Continue
Added by Michael Walker on March 12, 2014 at 9:00pm — No Comments
Stack exchange data dump
This is an anonymized dump of all user-contributed content on the Stack Exchange network. Each site is formatted as a separate archive consisting of XML files zipped via 7-zip using bzip2 compression. Each site archive includes Posts, Users, Votes, Comments, PostHistory and PostLinks. For complete schema information, see the included readme.txt.…Continue
Added by Vincent Granville on March 12, 2014 at 2:30pm — No Comments
Here's my list:
Added by Vincent Granville on March 12, 2014 at 9:00am — No Comments
The Big Data Yawn
Over the past couple of months we have met with a number of oil and gas executives to demonstrate our Oil and Gas Solution built on Data-Tactics’ Big Data Engine (BDE). During these conversations it has become obvious that the very mention of "Big Data" produces an involuntary physiological response among business leaders - eye rolls and yawns. It appears that big data has reached the Gartner "trough of disillusionment". These executives have heard from a bewildering…Continue
Added by Sullexis LLC on March 10, 2014 at 6:00am — No Comments
In-memory database technology is fashionable in recent years as the price of RAM drops substantially and gigabyte chips become affordable. By taking advantage of the cost-performance value of RAM, leading edge database developers are boosting the performance of next-generation databases with in-memory technology. However, many developers who intend to adopt in-memory technology only think of speed in terms of RAM, and do not exploit the true power of in-memory technology.
This is an area of data science that the public is less familiar with. This example involves small data, simulations, and 18 years old crowdsourcing.
It's an attempt to explain the cause of the TWA Flight 800 that exploded near New York on July 17, 1996. I raised the possibility that a potential cause for the Malaysia Airlines flight that went missing last week, was being hit by a missile (accidental or not). Likewise, many people still believe that TWA 800 was destroyed by a…Continue
Added by Vincent Granville on March 9, 2014 at 10:30am — No Comments
I have always had a great interest in how businesses organize in order to get things done. Here I raise some discussion points intended to stimulate debate.
Principle of Systemic Domains
Not that long ago, I was completing a graduate degree in “critical” disability studies. The critical part deserves to be in quotations since it is probably subject to interpretation and all sorts of misinterpretation. I am going to suggest that in critical…Continue
Added by Don Philip Faithful on March 8, 2014 at 8:15am — No Comments
Apache Sqoop is a connectivity tool to perform data transfer between Hadoop and traditional databases (RDBMS) which contains structured data. Using sqoop, one can import data to Hadoop Distributed File System from RDBMS likeOracle, Teradata, MySQL, etc… and also export the data from Hadoop to any RDBMS in form of CSV file or direct export to databases.
There is a possibility of writing mapreduce programs that would use jdbc connectors…Continue
Added by Pavan Kumar N on March 8, 2014 at 1:12am — No Comments
Operations research (including Monte Carlo…Continue
Added by Mirko Krivanek on March 7, 2014 at 10:00am — No Comments
How do you know if an outlier is the result of a data glitch, or a real data point -- indeed maybe not an outlier. Difficult question to answer, but the chart below shows that in some cases, the outlier is not an error.
In this example, you could argue that we are not…Continue
As we move to a more technologically advanced workflow, the need for bigger data storage…Continue
Added by Vincent Granville on March 5, 2014 at 3:30pm — No Comments
While companies complain about lack of analytic talent, professionals complain about lack of jobs. Everyone wants to work for Facebook, LinkedIn, Google, Intel, Apple, Twitter or some hot start-up. It creates fierce competition getting a job interview, let alone a job. But companies that do not belong to this circle see very few candidates applying for their data scientist open positions; in addition, they are only hiring what I call technical developers (defined by a narrow set of technical…Continue
Here's the introduction. Click here to view more details about the book.
This book is a type of “handbook” on data science and data scientists, and contains information not found in traditional…Continue