Assessing the Effectiveness of LLMs in Media Planning with R

Large Language Models (LLMs) have made waves in generating media recommendations quickly, yet a deeper inquiry shows the challenge lies in the accuracy of these suggestions. Key questions arise: Are the reach calculations accurate, are the underlying assumptions clear, and do the plans align with client briefs? As more professionals in media agencies turn to AI for support, understanding the capabilities and limitations of these models is essential.

With my role at Havas Media, I developed the Havas AI Media Quality Index (HAI-Q), which benchmarks media planning within the German market. This tool highlights the shortcomings of generic AI assessments and underscores the complexities involved in translating AI-generated text into actionable media strategies. A model may produce a coherent write-up about media planning, but errors can reside within budget allocations, audience definitions, or calculations related to Gross Rating Points (GRP).

Creating an Evaluation Tool in R

To properly evaluate LLMs in a media-planning context, I devised a straightforward evaluation structure using R. The essential components include:

1. A dataset with specific questions and expected outputs.
2. An AI solver to provide answers.
3. A scoring mechanism to assess those answers.

This framework is supported by the vitals package. It enables a mix of questions, guiding both scoring and feedback for what a strong answer entails. For instance, while open-ended questions can provide a range of acceptable answers, they can make precise evaluation challenging. Thus, more concrete metrics are essential in avoiding ambiguity in grading.

Defining Evaluation Questions

Here’s a concise set of evaluation questions targeting calculation, strategy development, and interpretation:

library(dplyr)
library(tibble)
media_questions <- tribble(
~id, ~input, ~target, ~domain, ~task,
"grp-calculation",
"A campaign achieves 60% reach with an average frequency of 3. Calculate the GRP and explain the assumptions.",
"A full-credit answer should calculate 60 * 3 = 180 GRP and clarify that the GRP formula is reach (in %) multiplied by frequency. It must not misrepresent GRP.",
"Reach and GRP", "Calculation",
"budget-allocation",
"A client allocates EUR 100,000 for reaching adults aged 25-49 in Germany. Suggest a media allocation method without historical data.",
"There is no single correct allocation. A strong answer acknowledges uncertainties, articulates channel roles, avoids fictional CPMs, and proposes a measurement plan.",
"Media strategy", "Recommendation",
"reach-frequency-tradeoff",
"Given two plans with identical budgets—Plan A with greater reach and lower frequency versus Plan B with lower reach and higher frequency—how would you choose?",
"A top-tier response would assert that the choice depends on objectives, channel selection, audience size, and desired outcomes, emphasizing the reach-frequency trade-off.",
"Media strategy", "Reasoning"
)
media_questions

These questions are intentionally specific to allow for easier grading. For example, the GRP calculation question offers a numeric benchmark, while the budget allocation query tests whether the AI can express uncertainty about its recommendations.

Implementing the Solver and Scoring

The vitals introduction employs the Task$new() function to merge the dataset with both a solver and grader. The example below utilizes Claude as the solver:

library(vitals)
media_task <- Task$new(
dataset = media_questions,
solver = generate(
chat_openrouter(model = "claude-sonnet-4.6")
),
scorer = model_graded_qa(partial_credit = TRUE),
name = "Small Media Planning Evaluation"
)
# Execute the solver and scorer.
media_task$eval()
# Bind results for analysis.
media_scores <- vitals_bind(media_task)
media_scores

Employing partial_credit = TRUE permits a richer analysis, providing not just correct/incorrect feedback but also allowing for partial credit on nuanced responses. This granularity is invaluable, as it reveals various types of reasoning errors an LLM might display.

Comparing Model Performance

Model evaluations become more insightful when comparing different LLMs against the same questions. This easy pivot allows for a clear analysis of strengths and weaknesses.

media_task_openai <- media_task$clone()
media_task_openai$eval(
solver_chat = chat_openrouter(model = "gpt-5.4")
)
comparison <- vitals_bind(
claude = media_task,
openai = media_task_openai
) |>
mutate(model = recode(
task,
claude = "Claude",
openai = "OpenAI"
))
comparison |>
count(model, score)

Maintaining consistent questions while varying model components is crucial for interpreting score variances accurately. These evaluations focus primarily on the quality of text output rather than the full scope of agent performance.

Challenges in Media Planning Evaluations

While media planning seems a fitting application for AI, this complexity often leads to evaluation difficulties. When multiple valid strategies exist, it becomes crucial to gauge a model's reasoning rather than expecting a singular "correct" answer.

Multiple Valid Strategies

Many challenges in assessing media strategies arise from the subjective nature of planning, where multiple experienced professionals may support competing allocations. Here are some useful metrics for broader evaluation:

• Making assumptions explicit;
• Working within budget constraints;
• Aligning channel choices with objectives;
• Distinguishing concrete facts from hypotheses;
• Outlining measurement and iterative improvement plans.

Evaluators must recognize sound reasoning without inadvertently declaring one planner's opinion as the definitive standard.

Time-sensitive and Local Media Facts

Media data isn’t static; reach and cost insights are often market-specific. A model trained on general media terms may falter without precise regional data or temporal context. This highlights the need for benchmarking against current local datasets to ensure accuracy.

Terminology Clarity

Media planning uses terminology that, while sounding similar, has distinct implications in calculations. For instance, understanding the differences between gross/net reach and average/effective frequency is critical. Evaluations must account for these nuances to avoid misapplication of correct formulas.

Limitations of LLM Evaluators

Using the model_graded_qa() in the vitals package allows scalable evaluations, yet this approach has its limitations. Depending on a model to grade another can introduce bias, where factors like language confidence might distort evaluations.

For rigorous assessments, it’s prudent to calibrate LLM graders against human benchmarks and categorize scores based on distinct assessment dimensions, such as:

• Factual accuracy;
• Numerical precision;
• Compliance with constraints;
• Reasoning quality;
• Practical utility of recommendations.

Aggregated scoring obscures detailed feedback; clarity on where models fail is essential for meaningful improvement.

Vitals as a Tool, Not a Comprehensive Framework

While vitals effectively invokes LLM responses, it is not fully equipped as an AI agent evaluation framework. Evaluations must also consider the interactions with tools and databases for a holistic perspective.

Insights from the Havas HAI-Q Evaluation

The Havas HAI-Q comprises practical tasks that span several media planning areas, including:

1. Strategy formulation
2. Channel allocation and budgeting
3. Target group analysis
4. Reach and GRP calculations
5. Assessment of media effectiveness

When assessing the general-purpose LLMs, the results were revealing:

The takeaways from these results are sobering; even GPT-5, the most competent among the models evaluated, answered less than half of the questions correctly. With 16 accurate responses out of 35, there's a significant error margin that isn’t acceptable for media planning accuracy.

Upon further investigation, the results highlight that these models find numeric tasks—such as audience size calculations—especially challenging. Generating persuasive narratives alone does not ensure foundational numerical accuracy, often leading to significant misjudgments that might not be immediately apparent in qualitative evaluations.

There’s also a clear performance trajectory from older models to more recent ones. The jump from GPT-4o to GPT-5 reveals a marked increase from just 11.4% to 45.7% 정확성, highlighting advancement even if the recent models aren't yet thorough enough for unsupervised deployment in media planning.

Claude Sonnet 4.5's lower performance doesn’t imply general inferiority; it simply points to the distinct needs of domain-specific evaluations that the HAI-Q encompasses comprehensively.

Despite the aggregate achievements, attention to detail is paramount. The assessment outcomes need to reflect question difficulty and acknowledge score variations across multiple runs to provide a nuanced understanding of results. What stands out is that current LLMs excel at language fluency but still grapple with numerical tasks critical for effective media planning.

Concluding Thoughts

Integrating AI in media planning begins with feedback-driven questions directly drawn from industry workflows. Utilizing tools like vitals can help structure that integration, but establishing clear criteria for “correct” responses is paramount given the subjective nature of planning.

As media planning encompasses varying strategies and frameworks, the evaluation must also consider the broader interactions between the model, the workflow, and all intermediaries involved. The results from HAI-Q offer a reminder that while AI can enhance efficiency in media operations, human oversight remains critical, particularly regarding numerical accuracy.

The transition from GPT-4o to GPT-5 signifies promising progress towards a more reliable AI workforce in this arena. Yet, until LLM outputs are validated more rigorously, treating recommendations with caution remains a priority. The next steps involve expanding model evaluations to encapsulate end-to-end interactions, ensuring comprehensive measurements of tool usage and entire planning sequences.

Additional Resources

• Getting started with vitals
• vitals documentation
• HAI-Q: Havas AI Media Quality Index
• ellmer documentation