Subscribe to DSC Newsletter

In-Memory Data Grids Allow Data Science at Faster Speeds

In-Memory Data Grids (IDG) allow organizations to collect, store, analyze and distribute large, fast-changing data sets in near real-time. Organizations are increasingly using IDG's for the efficient sharing of fast-changing data across multiple sites. IDG's provide the scalability and low latency required to enable applications to handle large workloads with fast responsiveness.

.

Data scientists need to be able to access and analyze fast-changing data quickly and easily - without regard to where it originates. The difference between high-value data science and good data science is increasingly about the ability to analyze larger amounts of data at faster speeds. Speed kills in data science and the ability to provide valuable, actionable insights to the client in a timely fashion can mean the difference between competitive advantage and no or little value-added.
.
For example, Rose financial clients use IDG's to collect and store fast-changing trading data for quickly analyzing and responding to emerging market trends. The significant reduction in time (from 60 minutes to 60 seconds) to access and analyze data improves decision-making, profitability and market competitiveness through increased performance in trading. Rose retail clients use IDG's for fast analysis of buyer patterns, purchase data and call-center communications to understand trends to improve marketing competitiveness, decision-making and profitability. Detecting and acting on consumer trends and competitors marketing and pricing immediately is critical in retail, especially the online retail space.
.
Organizations are demanding faster and easy access to information to make better decisions. IDG's enables immediate access to the right information which results in more informed decisions. Traditional Business Intelligence (BI) technology loads data onto the disk in the form of tables and multi-dimensional cubes against which queries are run. In-memory data is loaded into Random Access Memory (RAM) instead of hard disks. Thus, staff spends less development time on data modeling, query analysis, cube building and table design - and more time on high-value data science and business analysis.
.
Research shows that organizations using IDG's are able to analyze larger amounts of data at faster speeds than competitors. With in-memory tools, data available for analysis can be as large as data mart or small data warehouse which is entirely in memory. This can be accessed within seconds by multiple concurrent users at a detailed level and offers the potential for excellent analytics. The improvement in data access may be 10,000 to 1,000,000 times faster than from disk. It also minimizes the need for performance tuning and provides faster service for end users.
.
IDG's provide the following benefits:
.
1. Competitive Advantage. Organizations can make better decisions faster.
2. Speed. Improves time to find and analyze data to obtain valuable, actionable insights.
3. Better Decision-making. Organizations can improve the quality of their decision-making.
4. Productivity. Improved knowledge and business process efficiency increases profitability and reduces waste.
5. Customer Experience. Provides faster, more reliable service which can mean the difference between success and failure, especially in online transactions.
.
IDG's have the following characteristics:
.
1.The data model is distributed across many servers in a single location or across multiple locations. This distributed model is known as a "shared nothing" architecture and distribution is known as a "data fabric".
2. All servers can be active in each site.
3. All data is stored in the RAM of the servers.
4. Servers can be added or removed non-disruptively, to increase the amount of RAM available.
5. The data model is non-relational and is object-based. 
6. Distributed applications written on the .NET and Java application platforms are supported.
7. The data fabric is resilient, allowing non-disruptive automated detection and recovery of a single server or multiple servers.
.

Views: 976

Comment

You need to be a member of Data Science Central to add comments!

Join Data Science Central

Comment by Jean Michel LeTennier on June 16, 2013 at 6:57am

Imagine being able to do anything your mind can imagine, well now it can..
The 6th Normal Form is not a TERM.. it is a GOAL.. the "holy Grail" if you will of data management and more importantly "Information" management.. Imagine being able to Store IDEAS as opposed to disconnected bits of data..

A brief "INCORRECT" comment on WIKI.. "A relvar R [table] is in sixth normal form (abbreviated 6NF) if and only if it satisfies no nontrivial join dependencies at all — where, as before, a join dependency is trivial if and only if at least one of the projections (possibly U_projections) involved is taken over the set of all attributes of the relvar [table] concerned.[Date et al.]["

The TRUE definition of 6th Normal form is OBJECT Database.. where each and every piece of information/data is ATOMIC in nature and can be associated with any other piece of data/information, and thus NO restrictions.. NO constraints.. NO tables, NO Rows, NO VIEWS, NO CUBES... the correct term is "ASSOCIATIVE DATABASE" or Information system as it is in 3 dimensions by default.. and technically (N) dimensions

the advantages are thus:
100x SQL/ROW/TABLE speed
1/3 disk space
1 EXABYTE capacity - Single instance storage.. (no piece of data is ever duplicated)
Security - Un-hackable - there is nothing to hack into
NO QUERIES - we use filtering
NO TABLES - thus no indexing to worry about
Automated data aggregation - as many sources as required..

and that is just the start.. ;-)

let me know if you are interested in seeing it.. ? as a scientist. I think you would find it fascinating.. basically a 10 year old can now be taught to build data warehouses. ;-)

send me your external email and I will send you more info if you like, and yes this is going to market as we speak ..
JM
917-751-3131
If you want more information, I can add you to my Dropbox.com shared folder if you would like more information and a video is available there as well 

Follow Us

@DataScienceCtrl | RSS Feeds

Top Content 

1 Have You Heard About Unsupervised Decision Trees

Have You Heard About Unsupervised Decision Trees
2 Non-traditional strategies for mid-career switch to #Datascience and #AI

Non-traditional strategies for mid-career switch to #Datascience and #AI
3 How to Become a Data Scientist - On your own

How to Become a Data Scientist - On your own
4 Difference between Machine Learning, Data Science, AI, Deep Learning, and Statistics

Difference between Machine Learning, Data Science, AI, Deep Learning, and Statistics
5 5 Free Statistics eBooks You Need to Read This Autumn

5 Free Statistics eBooks You Need to Read This Autumn
6 Dealing With Imbalanced Datasets

Dealing With Imbalanced Datasets

Announcements

Decision Optimization with Prescriptive Analytics Applications

10 Considerations When Building a Dashboard

Earn your Master's in Applied Data Science in 18 Months

The Keys to Enterprise-Ready Machine Learning

Boost your business intelligence with smart data

Data Science for Big Data with Anaconda

The Fast Path to Success with AI

The Data Scientist’s Guide to Apache Spark™

The degree that data-driven companies covet

Optimizing Machine Learning With Tensorflow

Videos

  • Add Videos
  • View All

Resources

Top Categories

Machine Learning

R Programming

Python for Data Science

Visualization, Dashboards

NoSQL and NewSQL

Big Data

Cheat Sheets

Internet of Things

Excel

© 2017   Data Science Central   Powered by

Badges  |  Report an Issue  |  Privacy Policy  |  Terms of Service