Things I’ve Learned from 13 Years of CPG Data Experience

By Jeff Doherty, Chief Product Enrichment & Operating Officer

For brands looking to get closer to the consumer, the allure of data is understandable. Nearly every point of the buyer journey is now knowable – from credit card transactions on Sunday grocery runs to mid-week convenience store stops for beer and milk. 

But many of us have quickly figured out that an abundance of data doesn’t instantly translate into valuable and usable insights. 

One thing I’ve learned from 13 years of working with CPG data: It might be ubiquitous, but it’s also messy, complicated and in a constant state of change. 

In an article I wrote recently for Mediapost,I discussed how raw data on its own isn’t enough to provide actionable insights. Instead, like oil, raw data needs some refinement first. Data feeds first need to be evaluated, vetted and cleansed. 

Another thing I’ve learned: Good data science is a key component for delivering actionable insights. If you want to measure incremental sales, understand who your brand buyers are or grow your direct-to-consumer business, you’ll need models and data science to get you there.  A strong team of data scientists will know how to combine data from disparate sources, leverage modern technology to fill gaps and apply modeling strategies to gain unique insights. This is what teams at NCS have been doing since the inception of the company. 

Why Data Science Is More Important Now

Today’s advertising landscape is turbulent. Brands have experienced firsthand how macroeconomic factors like the pandemic, inflation and a potential economic downturn have changed consumers’ purchasing behaviors. 

At the same time, consumer expectations for privacy are reshaping how data is collected and used. Even as hunger grows for consumer insights, privacy remains a priority at NCS. We continue to take steps to ensure our data strategy is pursued with the utmost care for the consumer.

These trends have accelerated interest in bringing data and analytics in-house. Brands and agencies see cleanrooms or other data-safe environments as a way to create their own source of truth for consumer purchasing. Already, 20% of marketing leaders have a data cleanroom, and 24% say they’re planning to establish one, according to a survey conducted by the CMO Council in collaboration with NCS. 

Given the speed with which the CPG landscape is changing, brands need refined data and good data science more than ever. It’s the best way to ensure data strategies remain resilient and reliable even in dynamic markets. 

Create a representative mix

The trends discussed above also present challenges for data-driven advertising strategies. Despite data being everywhere, no brand, retailer or vendor has all the shopping and purchase data. Some major retailers have closed their data environments and aren’t sharing their data with third-parties. A small handful of these retailers account for as much as 40% of all shopping data.

So how do you get the right insights with only 60% of the data? That’s where data science makes a difference. A knowledgeable data scientist leverages modeling, which doesn’t require having every source of data. 

In my experience, the right mix of data sources combined with modeling is the key to producing an accurate view of changing consumer purchase behavior. While the optimal dataset doesn’t need every data feed available, there are three qualities it must have. It must be sufficiently large enough to scale across a population, representative of a wide range of factors and include granular details about household shopping behavior.

To create this mix, the data scientist will ensure data sources are varied and balanced, but make adjustments across outlets to ensure consistency and resiliency. Even if a data feed is removed entirely, advertisers can still be confident about the quality of the insights.

Close the gaps

Next, the data scientist will use machine learning models to fill the gaps of any missing data. This compensates for any changes in retailer output and ensures the resiliency of the dataset. 

After all, it’s inevitable that trends shift. For instance, even though retailers may or may not share their first-party data today, it’s possible they’ll change their minds in a few years. Or, another trend impacting data availability may emerge. Machine learning helps ensure the model continues to provide accurate insights, allowing it to withstand dynamic conditions and, in fact, improve over time. 

Model for insights

Once they have an optimal dataset, data scientists can use modeling to analyze the impact of advertising on consumer behavior. An experienced data scientist has spent decades fine-tuning the process and identifying purchase behavior by households. They’ll create a model that defines household buyers by type. Such clusters allow the data scientist to understand consumer purchase behaviors at the household level and apply these insights to similar households across the U.S. 

At NCS, our data scientists take this process a step further. We create clusters of households with similar behavior and characteristics, which provide an estimated spend for every household in America. These Purchase Graphs, as we call them, are key to all the NCS solutions. This approach has enabled us to create high-quality, unique insights. 

Build for the future

For me, the most exciting aspect of the NCS approach to data science is its versatility. Advertisers can use our unique insights to measure total sales, predict consumer behavior and inspire product innovation. They can access insights via NCS products or bring our data insights stream right into their own in-house cleanrooms. 

Either way, partnering with NCS means you’ll have the insights you need to develop more robust and effective advertising strategies.