Eve recommended an apple to Adam, and they shared the first buyer’s remorse.


Water, Water Everywhere

Recommender systems select products for customers based on past experience. Sometimes the product choices are called “items” or “content”, and other nouns are substituted for the customer. The most familiar case is that of a retail businesses – many customers with many products. Amazon, Netflix and Pandora famously bet their businesses on this technology. When you see guidance to other products, a recommender is at work.

For newly minted data scientists, the subject is covered in basic training. For marketers, these systems are a holy grail, a pearl without price, the silver bullet of digital engagement and personalization. Customers are expressing themselves with choices. On some websites, the customers contribute reviews, provide ratings and share advice. The recommender system is a mediator in a very personal action.


How did a subject so important get so complicated?


Forest for the Trees

The theory and algorithms for recommenders are well established. In 2015, the grandmotherly Association for Computing Machinery (ACM) is holding it’s 9th conference on recommender systems (recsys) in Vienna, Austria. At the University of Minnesota, GroupLens continues to make news, having pioneered recommending internet news to readers.  Enter social media, networking, elections and dating services - recommenders have a rich new world of application.


But, theory advances with complexity and new terminology. The subject can seem daunting, so there is a benefit to return to the basics.

  • The first general method is called Collaborative Filtering, and is focused on a matrix of C customers scoring or buying P products. Technically, this method is blind math; it does not need supplemental traits about Customers or Products. A collaborative filter would match people who like suspense films to those films that are suspenseful, without defining “suspense”.  Collaborative Filtering is done in two ways, User Based or Item Based. The former groups similar customers, usually in real time, and is also called Memory Based as a result. The latter clusters products, usually in batch, offline and may also be called Model Based.  To be complete, this large matrix can also be solved with machine learning and linear algebra.
  • A second approach is called Content Filtering. By contrast, this method depends on supplemental traits to classify the customer or the product. Customer data can be enhanced with segmentation, demographic information and personal preferences or “likes”. Product data can be enhanced with features, source attributes and use cases.  The additional traits on either Products or Customers are called tags.  Tagging can muddy the problem with sparsity – you may not have every tag for item.
  • As a third option, consider brute force. The term Knowledge Based is sometimes used. A good product manager may be able to tell which products are mutually exclusive, purchased together or purchased in progression.  A good marketer may know which customer profiles should not see certain products. Building a model of known business rules simplifies the recommender’s effort and prevents some obvious errors.

All three of these paths can be taken together. Most recommenders become hybrids, combining several methods and enhancing over time. The tools to construct the systems return to a familiar few basics in statistics:

  • Dimension Reduction – See Principal Components Analysis (PCA), Linear Discriminant Analysis (LDA) and Independent Components Analysis (ICA).
  • Nearest Neighbor – Which customers are similar? Which products are bought together? These issues need a measure of distance.
  • Machine Learning – Recommenders are like personalized models, with a theta for each product. Libraries in R or Mahout can help.
  • Neural Networks & Classification – Particularly with Content Filtering, advanced trees are common.


With all this power, why do I get promotions for skin cream?


My Kingdom for a Horse

Recommenders are suffering from the curses of our emerging Data Sciences. The early adopters have an advantage, so solutions that are “out of the box” are popular. Like other tools, recommenders can be purchased as hosted personalization services. Their use, too, is often in the digital channel. The internet is cheap and forgiving when these systems start. But, to quote H. L. Mencken,


       “For every complex problem there is an answer that is clear, simple, and wrong.”  


The subjects below are emerging in both recommender statistics and marketing. Consider researching these for building or improving your recommender (using the movie recommender as a poster child):

  • Noise over Signal – Model, then selectively remove, dominant features to find less obvious traits. The movie recommender immediately recognizes gender and other tastes are drowned out.
  • More Data – This is the analyst’s eternal cry, but it has merit. Netflix invites customers to rate any movie, not just the ones served there.
  • Serendipity – Insert intentionally unexpected items. This term is now formalized and included in software design.
  • Validate – Like all marketing efforts, recommenders deserve a control, A/B testing, expectations and performance quantified on every enhancement.
  • Trust – We can agree on comedies and disagree on dramas. The subject of trust is emerging now in recommender methods.


Tired advice remains.  Using every method, applying every tool and addressing every issue provides a recipe for disaster.  Consider the risk and opportunity of customer interaction, start with basic methods and make a roadmap of learning forward.


Good luck and stay tuned – this subject will challenge us all.


Views: 2515


You need to be a member of Data Science Central to add comments!

Join Data Science Central

Comment by Sione Palu on January 14, 2015 at 12:46pm

A recent hot topic in recommender systems today that published work from academic researches have appeared in recent years is to incorporate user context is the "context-aware recommender system".  There are more published work in this area but I've come across the following which may be useful to readers here:

#1) G. Adomavicius, R. Sankaranarayanan, S. Sen, and
A. Tuzhilin. Incorporating contextual information in
recommender systems using a multidimensional
approach. ACM Transactions on Information System,

#2) L. Baltrunas and X. Amatriain. Towards
time-dependant recommendation based on implicit
feedback. In Workshop on Context-Aware
Recommender Systems (CARS 2009) in ACM Recsys
2009, 2009.

#3) L. Baltrunas and F. Ricci. Context-based splitting of
item ratings in collaborative filtering. In L. D.
Bergman, A. Tuzhilin, R. Burke, A. Felfernig, and
L. Schmidt-Thieme, editors, RecSys, pages 245–248.
ACM, 2009.

#4) K. Oku, S. Nakajima, J. Miyazaki, and S. Uemura.
Context-aware svm for context-dependent information
recommendation. In Proceedings of the 7th
international Conference on Mobile Data
Management, 2006.

#5) C. Ono, Y. Takishima, Y. Motomura, and H. Asoh.
Context-aware preference model based on a study of
difference between real and supposed situation data.
In UMAP ’09: Proceedings of the 17th International
Conference on User Modeling, Adaptation, and
Personalization, pages 102–113, Berlin, Heidelberg,
2009. Springer-Verlag

© 2021   TechTarget, Inc.   Powered by

Badges  |  Report an Issue  |  Privacy Policy  |  Terms of Service