Summary:  What changes in your analytics and Big Data stack would you have to make if you only had 10 milliseconds to make a decision?  There’s an entire industry that has to live by that rule.

This is a great story about the collision of ecommerce, digital advertising, predictive analytics, and AI in a new digital battleground, the automation and optimization of advertising targeting and spend.  Like so many interesting opportunities this new market starts with a pain point and an unmet need.  In this case the lag in digital advertising spend, particularly in mobile devices.

Mary Meeker in her 2016 Internet Trends report that we featured last week pointed out this trend.

On the far right side you can clearly see that while viewers spent 25% of their time on mobile devices, that category represented only 12% of spend.  Meeker figures this is a $22 Billion opportunity and needless to say tech entrepreneurs took notice as well.

Programmatic Advertising and Marketing

The first thing to understand is that buying advertising in this space, both mobile and internet, is different from traditional ad purchases.  Traditional ad purchases are oriented toward acquiring space on a given site that the buyers are familiar with and relying on the publisher to accurately describe the audience.  The new model is “Programmatic Advertising” that puts the emphasis on delivering the desired audience regardless of the site being visited.

All of this sounds like a slam dunk improvement.  But in this area that’s been developing rapidly there’s more than a little Wild West feel.  A variety of exchanges numbering at least 100 of been established where marketers or their agencies can bid for and buy digital spots in real time but they’ve come with their own challenges.

• Lack of transparency and potential conflicts of interest.  There’s been a lot of arbitrage and closed-door rebate deals between exchanges, publishers, and agencies that make true net price and value difficult to assess.
• Brand Safety.  You certainly don’t want your well recognized wholesome brand ending up being advertised on some sketchy site and there are plenty of horror stories around this.
• Fraud.  There are still plenty of marginal publishers creating click bait sites, stacked ads and still plenty of click fraud going on.

So these factors in combination with the newness of the concept of Programmatic Advertising have held things back.  Over the last several years and particularly in the last two or three years a number of what are referred to as “Demand Side Platforms” have been introduced to solve these issues.

Demand Side Platforms (DSPs)

DSPs seek to simplify the life of marketers and agencies by providing a single platform that provide at least:

1. A single view to manage digital programmatic buys across multiple ad exchanges and all digital device types.
2. Machine learning and optimization algorithms that allow marketers and agencies to optimize ad buys for each campaign, be that as short as a day or as long as several months.
3. Powerful audience management tools enabled by predictive analytics that enable planning for audience extension and cross-device targeting beyond the immediate benefits of programmatic ad buying.
4. The ability to assess and execute against at least 1 Million ad queries per second.

And you can bet that the 20+ competitors in this space are equally anxious to expand these programmatic buy capabilities to traditional print, radio, and TV ads.

If you’re not familiar with this market segment, here are the nine that Forrester says are leaders as of this spring.

Data Science and Big Data in DSPs

What attracted my attention are the really formidable challenges of volume, speed, and predictive accuracy.  Picture an ad agency or large marketer with multiple products, each with multiple campaigns of different durations and different buyer, price, channel, and geographic targets. 

On the supply side visualize the firehose of unique advertising opportunities arriving at the rate of up to 1 Million per second

• that need to be assessed for viewer desirability (and brand safety)
• selected as potentially part of an optimized campaign advertising mix
• a price bid prepared and submitted (some of which you win and some you don’t)
• an electronic hand shake reached
• and the ad served
• all in about 100 milliseconds.

Adapting Data Science to This Environment

You can see the challenges.  Here are just a few comments and observations from two of the leaders in this category.  The first come from Beth Logan, Senior Director of Optimization at DataXu as presented in a recent webinar.

• In total DataXu processes about 2 PB of data per day and makes 1.6 million individual bid decisions per day deciding whether a particular web page view advertising opportunity is likely to convert, what that page view is worth, and how it will fit in the overall campaign budget optimization.
• In the time it takes to serve up a web site view to a single viewer (well under 500 ms) an entire on-line bidding process has taken place with multiple bidders vying for the opportunity to display an ad to that viewer in real time.
• The total DataXu processing time can’t exceed 100 ms and the bid/no bid decision and calculation of the bid value is allocated only 10 ms.
• DataXu must maintain an extremely large number of predictive models based on real time click through and conversion rates that are modeled to predict expected conversion rate based on about 50 different features of the web site, the viewer, and a number of external factors that can include geography and even weather. 
• The models are refreshed daily but even at this rate the types of ML that can be used are severely constrained by memory and time.  Given the limit on total memory (a business investment decision) and time (a limit on the complexity and type of algorithms that can be used) DataXu has settled on mostly Bayesian techniques that work well with one pass Hadoop reads.  Given more time or memory they could be more accurate but are more than acceptable in this time-compacted environment.
• DataXu accomplishes all this with a home grown Hadoop system and is only now planning its migration to Spark.

DataXu has actively expanded beyond traditional mobile and internet markets to include addressable television and radio moving toward a platform that includes both traditional and digital media.

While DataXu has both ‘one click’ automatic optimization as well as expert override controls, Forrester gives Rocket Fuel the nod for its use of AI.  Forrester says:

“Consider Rocket Fuel if you value a hands-off approach that relies on machine learning.  Rocket Fuel’s focus is on providing marketers with an autonomous programmatic buying platform that is independent of human optimization. Rocket Fuel stresses that its artificial intelligence (AI) consistently out-performs marketer manual optimizations and is looking to further automate the process for marketers by eliminating the need to monitor performance and the “guesswork” involved in determining optimal investment allocation. “

It appears that there are at least two things that Rocket Fuel adds to the equation.  The first is its extensive use of AI to convert campaign goals to ad selection metrics with as little need for human intervention as possible. Whether you think of this as AI or as an ‘expert system’ (AI’s little brother) the impact is very much the same.  Add in automated optimization routines and this approaches pretty powerful data science.  This is very much the same direction in automating predictive analytics taken by the new one-click-to-model platform vendors.

The second element is not necessarily data science but a clever and useful enhancement which Rocket Fuel brands as “marketing in the moment”.  Rhonda Shantz, VP Marketing explained in a recent webinar, the goal here is to go beyond segments and profiles.  This recognizes that many family members may be sharing an IP address and even devices.  The claim is that they are able to develop and integrate a score into the automated decisioning that indicates when and on what devices one person in that household is most receptive to a particular campaign ad. 

There are a large number of potential variables.  In one example ‘Annie’ is just finishing yoga and is starting to make a dinner reservation.  She would be a good target for a restaurant add, amplified by other factors such as:  she’s on a mobile device, she just checked the weather, the local time is 6:30 pm (positive factors in scoring), that she’s just seen three ads, already browsed restaurants on line, has never clicked on an ad (negative factors in scoring).  The impact as Ms. Shantz explains is to stop marketing to devices and start marketing to people.

As Forrester points out each of these nine platforms brings a somewhat different mix of capabilities but still succeeds in this remarkable high volume, high speed world in which you and I are personally auctioned off every time we open a web page.  In total we may not click on or even much like the ads on our web pages but with these advances in the Big Data technology stack and the inclusion of increasingly fast and sophisticated predictive analytics and optimization, at least we stand a better chance of seeing ads that are relevant to us – and oh yes, more successful for the marketer.

About the author:  Bill Vorhies is Editorial Director for Data Science Central and has practiced as a data scientist and commercial predictive modeler since 2001.  He can be reached at:

[email protected]