Methodology

How we evaluate audience models

Every number on this page comes from a frozen holdout, a named build, and a baseline it had to beat.

The first public scorecard publishes after the September 2026 run. Until then this page explains exactly what it will contain.

What each number means

MAPE

Mean absolute percent error between the modeled curve and the measured one. Reported as a median too, so one outlier cannot define the claim.

Drop timing hit rate

How often a flagged drop landed within 5% of the video duration of where it happened.

Average view duration error

How far the modeled average view duration lands from the measured one, in seconds and in percentage points.

End hold error

How far the modeled retention at the final point of the video lands from the measured retention there.

R squared

How much of the measured curve shape the modeled curve explains, point by point, across the whole video.

Spearman rho

Whether the model ranks the low points and high points in the same order the measured curve does.

95% confidence interval

The range from resampling whole creators, or whole channels when creator groups are unavailable. We withhold it when identity is incomplete or fewer than 20 independent groups exist.

The baselines the model has to beat

The last three are reported by the training harness. They will appear here with numbers once a trained model is scored on this holdout.

This channel's usual shape

The channel median retention curve for videos of the same duration family. The only baseline in the production scorecard today.

Global median

The median retention curve across every channel in the holdout. Reported by the training harness for comparison.

Position only

A curve that only knows how far into the video a point is, with no knowledge of the content.

Loudness only

A curve built from the audio loudness track alone, with no knowledge of the visuals or the script.

The temporal position control

The temporal position control permutes the mapping between modeled and measured curves across videos, keeping each point in its original position. A model that only learned attention decays over time scores the same either way.

A published attention probe reached a correlation of 0.47 and collapsed under this exact control. The number looked real until position alone explained it.

A trained model advances only after at least 300 frozen holdout videos, positive paired lift intervals over the channel and loudness baselines, curve error at or below 7%, at least 85% drop-timing hits, a positive position-controlled result with permutation p below 0.05, and matching hashes for the manifest, model outputs, and split.

Measured

History bands pulled directly from the platform
Published audience retention for the exact video
Fix outcomes: what happened after a creator re-screened a cut
Viewer panel response, when a panel has run

Modeled

The modeled retention curve before publish
Modeled drops and where they land
Modeled attention moments inside the video
Script lines the model flagged
The niche cohort a channel is compared against

Screen a rough cut against your own audience

Screen a rough cut See pricing