Ads Creative Quality Framework
Defining and measuring what makes an ad worth looking at
A diary study with participants in the US to deep dive into attributes of creative quality and identify ways to measure and track it. The work culminated in a framework that ML and DS teams use for assessing creative quality at scale.
The Problem
Instagram's ads team had long treated creative quality as a subset of ad relevance. But a recent survey revealed that creative quality is a separate construct. The problem wasn't what ads were being shown but how they looked, sounded, and felt. The team needed a way to define creative quality that was grounded in how users actually evaluate ads, specific and structured so that they can be assessed at scale.
My Role
Sole researcher. I collaborated with a Data scientist and ML engineer to operationalize the framework.
My Approach
Before designing the study, I partnered with the PM and Data Science to understand how creative quality was currently being measured and where the gaps were. DS was already tracking technical quality signals like load speed, image clarity, and video jitter. What wasn't being measured was how users actually perceive and evaluate creative quality in the moment, the experiential and aesthetic dimensions that drive appeal or aversion. That scoping defined the focus of the research.
This was the first time the team was looking at experiential creative quality as a standalone construct, so we started in the US to deep dive on how users defined it, what mattered and what didn't. I ran a 7-day diary study using dScout with 40 to 45 participants, where participants submitted different types of content: ads whose creative quality stood out, ones where it didn't, and non-ad content whose creative aspects caught their attention. The study was structured around unprompted and prompted attributes. Recruitment was deliberate across surface preference (high Feed, Reels, and Stories users), age, ad engagement levels, and proxies for commercial relevance.
Analysis took a grounded theory approach: I coded individual diary entries across specific dimensions like text usage, brand match, video hooks, music quality, emotional response, and perceived trustworthiness, then worked back up to higher-level themes across the full dataset. I used Manus for video analysis and AI tooling to speed up the coding phase.
After the framework was established, I worked directly with ML engineers to refine LLM prompts for automated creative evaluation at scale, testing which dimensions could be reliably assessed through annotation and iterating on prompt design to improve consistency across dimensions.
The Impact
Established a foundation for LLM-based creative evaluation by partnering with an ML engineer to test and refine prompts against specific framework dimensions.
Surface-specific findings informed how teams think about creative quality standards across Feed, Reels, and Stories rather than applying a one-size-fits-all approach.
Reflections
Validating the framework across markets would be an important next step to understand whether the tier structure holds and how creative quality expectations shift across regions. On the analysis side, there was likely more headroom to systematize AI tooling earlier in the process and reduce manual work.
The images are illustrative. Data and dimensions have been modified to protect confidentiality.
