Has anyone performed tests to compare computation times for different data science algorithms on different platforms? Or for sorting, merging, joins, hash table management, and other database operations? For IO operations (parsing a file)?

For very large data sets or during the initial learning stage of a machine learning system, most of the time is probably spent in data transfers rather than in-memory processing, so maybe it does not matter if one uses R or Python. Sometimes generating / summarizing the entire data set with Python / Perl, and pre-loading it into an R  table (as for generating video frames) accelerates the process: it is much faster than generating one video frame at a time, on the fly, with R. So clearly, optimizing speed is not just about using a faster procedure or faster language, but breaking down the tasks in a way that optimizes in-memory usage.

Any thoughts on this?

DSC Resources

• Career: Training | Books | Cheat Sheet | Apprenticeship | Certification | Salary Surveys | Jobs
• Knowledge: Research | Competitions | Webinars | Our Book | Members Only | Search DSC
• Buzz: Business News | Announcements | Events | RSS Feeds
• Misc: Top Links | Code Snippets | External Resources | Best Blogs | Subscribe | For Bloggers

Additional Reading

• 50 Articles about Hadoop and Related Topics
• 10 Modern Statistical Concepts Discovered by Data Scientists
• Top data science keywords on DSC
• 4 easy steps to becoming a data scientist
• 13 New Trends in Big Data and Data Science
• 22 tips for better data science
• Data Science Compared to 16 Analytic Disciplines
• How to detect spurious correlations, and how to find the real ones
• 17 short tutorials all data scientists should read (and practice)
• 10 types of data scientists
• 66 job interview questions for data scientists
• High versus low-level data science

Follow us on Twitter: @DataScienceCtrl | @AnalyticBridge