At the Data Science Association our members often complain about the major data engineering problem of finding the right tools and programming models to build both robust data processing pipelines and efficient ETL processes for data transformation and integration.…

While large data sets may provide significant value in certain cases, data diversity and integrating smart data points will provide more consistent actionable insights and high value intelligence leading to better decision-making.



For example, consider NFL football data. Focusing on large football game data sets is usually not helpful and often misleading creating…