Subscribe to DSC Newsletter

11 Features any database, SQL or NoSQL, should have

Here's some important features that I think all databases should have:

  1. Offering text standardization function to help with (1) data cleaning, (2) reducing volume of text information, (3) merging data from different sources or having different character sets
  2. Ability to categorize information (text in particular, and tagged data more generally), using built-in or ad hoc taxonomies (with a customized number of categories and subcategories), together with clustering algorithms. A data record can be assigned to multiple categories.
  3. Ability to efficiently store images, books, records with high variance in length, possibly though an auxiliary file management system and data compression algorithms, accessible from the database.
  4. Ability to process data remotely on your local machine, or externally, especially computer-intensive computations performed on a small number of records. Also, optimizing use of cache systems for faster retrieval.
  5. Offer SQL to NoSQL translation and SQL code beautifier.
  6. Offer great visuals (integration with Tableau?) and useful, efficient dashboards (integration with Birt?)
  7. API and Web/browser access: database calls can be made with HTTPS requests, with parameters (argument and value) embedded in the query string attached to the URL. Also, allow recovery of extracted / processed data on a server, via HTTPS calls, automatically. This is a solution to allow for machine-to-machine communications.
  8. DBA tools available to sophisticated users, such as fine-tuning query optimization algorithms, allowing hash joins, switching from row to column database when needed (to optimize data retrieval in specific cases).
  9. Real time transaction processing and built in functions such as computations of "time elapsed since last 5 transactions" at various levels of aggregation.
  10. Ability to automatically erase old records and keep only historical summaries (aggregates) for data older than (say) 12 months.
  11. Security (TBD)

Note that some database systems are very different from traditional DB architecture. For instance, I created a web app to find keywords related to keywords specified by a user. This system (it's more like a student project than a real app, though you can test it here) has the following features:

  • It's based on a file management system (no actual database)
  • It is a table with several million entries, each entry being a keyword and a related keyword, plus metrics that measure the quality of the match (how strong the relationship between the two keywords is), as well as frequencies attached to these two keywords, and when they are jointly found. The function to measure the quality of the match can be customized by the user.
  • The table is split into 27 x 27 small text files. For instance, the file cu_keywords.txt contains all enties for keywords starting with letter cu (it's a bit more complicated than that, but this shows you how I replicated the indexing capabilities of modern databases).
  • It's running on a shared server, at it peaks hundreds of users were using it and the time to get an answer (retrive keyword related data) for each query was less than 0.5 second - most of that time spent on transferring data over the Internet, with very little time used to extract the data on the server.
  • It offers API and web access (and indeed, no other types of access)

Related articles

Views: 22805

Comment

You need to be a member of Data Science Central to add comments!

Join Data Science Central

Comment by Erko on July 11, 2014 at 12:02pm
The enhancement of SQL to be able to run your standard data mining algoritms on The database server.

This is something I believe being more and more focused on. Examples of this is eg SAP hana, IBM pure data for analytics, oracles data mining for darabases etc.

It speeds up the processing and leta you focus on the results instead.

Follow Us

Videos

  • Add Videos
  • View All

Resources

© 2016   Data Science Central   Powered by

Badges  |  Report an Issue  |  Privacy Policy  |  Terms of Service