This post first appeared on the i Think blog.
What is the semantic layer you ask?
The semantic layer is the underpinning of modern business intelligence platforms. Vendors with robust semantic capabilities command more than 50% of a $14 billion plus market:
Source: Gartner Worldwide BI Market Share *(SAS - not sure if they offer a semantic layer in the classical sense - please let me know in the comments)
This data actually understates the actual market share and wide spread usage of semantic layers within enterprise BI shops. Several BI tools in the "Others" category, like Birst also offer solid semantic capabilities. The real number might be closer to 60% or even higher.
So yeah, it's a big deal, worth at least $7 billion. It was pioneered by Business Objects in the early 90's well before the acquisition by SAP and they still remain the market leader in BI after all these years.
Back to the original question - what is the semantic layer exactly? From wikipedia:
A semantic layer is a business representation of corporate data that helps end users access data autonomously using common business term
Close, but I think thats too high level and simplistic. Here is my definition:
Semantic layers map concepts like Customer, Product, Clicks, and Revenue to a set of data transformation rules, aggregation rules, datasources, tables, and columns, allowing users to ask novel questions without knowing anything about the underlying structure of data. There are three big reasons why enterprise IT loves the semantic layer:
Okay, if you are still reading this, you are either a really good friend, a data nerd, or a very curious individual. In any case, thanks and congratulations you're about to peek into the future.
Note: the BI semantic layer is different from the popular notion of the semantic web (RDF's) in that the links between semantics in the BI tool is business rules and datasources while in the semantic web the links are other associated relevant data (ie. country -> country_table -> city_table -> city vs. country -> city in the semantic web ).
Now for some bad news for the semantic layer fans (including myself in this group). After a decade and a half, we are seeing major cracks in this strategy and sometimes downright collapse. End users are running away en masse from centrally managed BI tools and the semantic layer is partly to blame. The proof is simply in the meteoric rise of Tableau who once called enterprise BI a pencil attached to a brick. That brick for better or worse is the semantic layer. Here are three reasons end users are fleeing from enterprise BI and semantic layers:
It's time to unleash the semantic layer. The great unbundling of the BI stack is already underway with Visualizations out of the gate first. The next to go will be the Semantic Layer.
Startups like mine, Vero Analytics, are already working hard to solve the many problems I described above while preserving the benefits of a semantic layer. Over the next months and years you will see the emergence of a new category of tools under the banner of "Semantic Data Preparation." Like Visualization Tools, this new category will target business units and end users with lower learning curves, intuitive user interfaces, and a collaboration model that makes sense for the 21st century. It will at once enable transparency of what and how Revenue should be calculated, while allowing those rules to seamlessly evolve over a collaborative framework.
These new tools will marry reporting, analytics, and data preparation into one workflow. This means users will be able to self service questions like "Who are my Top 10 Customers", then follow it up by enriching, correcting, and transforming that dataset. All the while adding and discovering new semantic concepts that becomes shareable and reusable across the whole organization.
The enterprise BI stack will start to change with a bring your own visualization tool strategy at the top level. Each business unit will and should be free to choose their preferred visualizations toolkit even if that means sticking with excel. This tier of visualizations will sit on top of the Semantic Data Prep tier with direct integration from visualization tools. Here users will define data sets, build pipelines, collaborate on semantic definitions, and possibly even cache datasets.
I am looking forward to this future where truth and creativity lives together.
As BI professionals for the last 10 years and the founder and CEO of Vero Analytics my team and I are thrilled to be part of this revolution. We recently announced the wide availability of our first release of the Vero Designer in open beta (free extended trials). Our work is not complete but the Vero Designer will give you a taste of how the Semantic Engine will work in the future.