Measurement Plan Rubric — Measuring One Feature After Release
Your three-part rubric for planning, and grading, how one feature is measured after release: the parts and points, your testing-language forms, the signal table, how it grew over six versions, your practice scores, and its job inside AI Agent for PM.
A rubric for planning how to measure one feature after release, and for grading practice answers. It's the detailed version of step 5 ("Measure + Exit") in your PM decision framework, and the spec for the Measurement Plan Agent in AI Agent for PM. Built over two days of practice (2026-11-16/17) and rewritten into your testing language.
Here's why it should work, here's what I'll watch and what happens if it misses, and here's how I'd know I wasn't fooled.
One plan per bet with one hypothesis. Small fixes shipped together to one flow under one hypothesis count as one feature. Themes, OKRs and KPIs are measured directly at their own level, never by adding up feature results. Each plan names its link upward: "Contributes to: [key result or KPI]."
Three parts, 100 points
A · Hypothesis 20
Why should it work? A Gherkin line from pain and workaround to new behavior and signal, plus "Contributes to."
B · Signal table 50
What do I watch, from where, to what bar, by when, and what if it fails?
C · How could I be fooled? 30
C1 by something else: with/without, rule out a cause, fingerprint. C2 by the feature: negative scenario, coverage gap, AI default.
Your forms, in testing language
Each original area has a form you already know from QA. Ask "how could this fool me?" twice: once about the feature (guardrails) and once about something else (attribution).
When [feature]
Then [new behavior]
… which means we should see [signal]
| # | Area | Weight | Your form |
|---|---|---|---|
| 1 | "Because" and chain | 20% | Given / When / Then … which means we should see [signal] |
| 2 | Baseline and data sources | 20% | Background: today [value] from [source] over [window]; if missing, logging, a task test or a hand sample |
| 3 | Prediction and exit | 15% | Then within [time], [metric] from [baseline] to [target]. Pass → expand or next stage; Fail → rework, pause, roll back or stay |
| 4 | Leading vs. lagging, timing | 15% | Signal table: Early / Outcome / Long-term rows, each with baseline, expected change and check date |
| 5 | Attribution | 15% | Scenario with vs. without (same weeks), rule out one cause, fingerprint |
| 6 | Guardrails and blind spots | 15% | Negative scenario counted with a threshold and action; coverage gap; AI default |
Part B: the signal table
Who / what vs. From
- Who / what = whose number: a person, a group, the product or system, or money.
- From = where you look it up. Artifacts (tickets, transcripts, logs) are sources, not subjects.
Baseline and Pass if
- Baseline: the value before the feature, measured the same way and over the same window as afterward. "New" if the signal only exists with the feature, judged against Pass if.
- Pass if: an acceptance threshold set by ceiling, break-even or worth keeping, locked before outcome data. A wrong bar is fine.
Counting rules
- Count per unit of work.
- User-facing features get the adoption funnel.
- Small numbers give direction, not precision; track per account in B2B.
- Doubt the feature? Add the doubt as a diagnostic row.
Match the effort to the stakes
| Level | What you do |
|---|---|
| Light | Part A plus one Outcome row |
| Medium | A, a full B, and C1 with a staged rollout |
| Heavy | All parts, a holdout, a 30–90 day review |
The level depends on cost, reversibility, uncertainty and whether the result will be repeated as a claim.
Filling it in
Skeleton first
Write a rough line for every part before going deep. An empty area scores 0.
Defaults, not invention
- Exit rule: "if [metric] < [X] by [date], we [rework / pause / roll back / stay]."
- Comparison: holdout → staged rollout → own before plus steady total volume.
- "From ___" after every metric.
AI default, always
A pass doesn't mean correct. Grade a sample for correctness, and track reverts and cost per successful output.
In AI Agent for PM
Light check: every feature
The Gherkin line, the feature's tags and "Contributes to"; one Outcome row, preferably a tag-level signal the pipeline already tracks; one negative scenario. Mostly template plus code; the LLM drafts only part A.
Full plan: exceptions only
Strategic bets, AI features, or results Sales or Marketing will repeat as a claim. The Measurement Plan Checker Agent scores it against the rubric and confirms, flags or rejects, never rewrites.
Data is a boundary
Missing or unvalidated data means "not measured." The agent can only suggest how to measure it, filed into intake as a new candidate feature. No invented numbers.
How it got here
| Version | Date | What changed |
|---|---|---|
| v5.1 | Nov 17 | Scope rule: one feature, not a group; "Contributes to" line |
| v5 | Nov 17 | Part B reworded (Who / what, Baseline, From, Pass if); restored effort levels, funnel, per-unit and small-number rules |
| v4 | Nov 17 | Six areas regrouped into three parts with sub-item scoring |
| v3 | Nov 17 | Rewritten in your testing language |
| v2 | Nov 17 | "Filling it in": skeleton, six questions, defaults, practice ladder |
| v1 | Nov 16 | Six areas and weights, templates and full example lines |
Dropped by v5's value rating: the six-question list and the practice ladder.
Key principles
- Match effort to stakes; always do at least the light version.
- Hearsay is a lead, not data: decide in advance what counts, then count. Observed behavior beats opinion.
- Sample randomly, use proxies with different blind spots, and state the limits.
- Every metric needs a data source; specify logging when the feature is new.
- AI features: correctness is separate from pass or acceptance.
- Long-term goals (retention, win rate) are a hypothesis backed by leading indicators, not a measured claim.