Real time video stream trend analysis for
accurate ad targeting

About the client

A leading independent software platform that enables brands and agencies to plan, buy, measure and optimize their global advertising. By improving transparency and leveraging real-time data, the platform provides holistic planning tools, analytics and insights across television, video, display, mobile, social and other brand advertising initiatives.

Tech stack
JAVA, PHP – Hadoop : HDFS, MapReduce, HBase, Hive – Membase – Memcache – MySQL and more.

The problem
Change is constant, and the pace of change and it’s impact is increasing with multiple media channels and an active social media. The ability to rapidly perceive and respond to this change is key to more accurate ad targeting. This requires an ability to process information to identify trends and gain insights quickly.
In such a scenario, analyzing video trends for ad targeting is a challenge. The custom data analysis platform was running on 300+ node cluster, working on customer’s Hadoop data pipeline running on Amazon EC2.

The solution
We began this project when the hadoop ecosystem was still in infancy. Every visitor was identified as new or returning from the http queries during batch processing hadoop jobs. Oozie was still in its early stage and there was a homegrown workflow engine for troubleshooting that ran into distributed computing issues in lock management and scheduling.

The project involved managing 500 servers spread over 4 regions. In the performance troubleshooting of serialization and compression layers, the Ints, Longs and Collections in Java introduce significant size overheads unless handled at job design.

The benefits
As the stability and performance issues were solved, engagement reports were published at higher frequencies, micro trends could be measured and used in targeting. For example, a tweet can cause one of the topics to be more searched that makes some inventories hotter than others which is important information for ad targeting.
If the same information is available at a later period, leveraging it may not be as effective as the trends may be short-lived. This translated into customer success with some of the key adopters of the platform especially in the impact and reach reports for advertisers

We’re open for business
Let’s get started