Explainer Projects

AI Agent for PM — Pipeline Architecture v1.5

Your multi-agent pipeline from raw product signal to a human-reviewed, sequenced roadmap: every stage, the shadow track, the Measurement Plan Agent, design principles, evals and how it got here.

Generated
November 18, 2026, 2:57 PM MT
Created
November 18, 2026, 2:45 PM MT
Built from
  • wiki/entities/ai-agent-for-pm.md

A multi-agent pipeline that takes raw product signal to a human-reviewed, sequenced roadmap. It started after the Keysight layoff as the first step in applying Gas Town logic to business work.

Current design: architecture v1.5 (2026-11-18). That's v1.2, plus the fixes from the PM Lenses review (v1.3), plus the Measurement Plan Agent (v1.4), plus a shadow track for your own untested frameworks, STARTIFY and RADAR (v1.5).

The wiki page is the master

The wiki entity page is the master for the architecture. The older "AI the PM" project file and pipeline-layers.md were retired on 2026-11-18 and now point to it. A separate private HTML view, "AI Agent for PM: pipeline v1.4", is behind v1.5.

v1.5current architecture
14pipeline stages
3human steps: Align, Review, Pass if lock
3 + MoSCoWestablished frameworks rank
2own frameworks in shadow
8design principles

The pipeline (v1.5)

Stages in order, coloured by what runs them. A ✓ marks a stage with an LLM checker (the two-agent rule). Dashed boxes are human steps.
StageWhat runsType
Ingest5 source-specific classifiers in parallel on a small model (provisional until it passes the gold set, v1.3). Usage and behavioral data come in as lookups plus anomaly events, not LLM-classified (v1.3). Stakeholder sources get their own labeling family (v1.3, design open). Friction Stage Tagging for customer sources. Account ID joined from CRM.LLM, per item
Theme registryEmbed, assign to existing themes, cluster leftovers with HDBSCAN. Noise goes to a doesn't-fit pile.code
LabelJourney Stage and Signal Dimension labeling agents, only on new or changed themes. Label-vs-item-tag check.LLM + check
LinkCross-Source Merge Agent → Merge Checker AgentLLM + LLM checker
DefineProblem Definition Agent (template, MoSCoW subfeatures) → Problem Definition Checker Agent. Early review on deterministic triggers.LLM + LLM checker
Align v1.3Stakeholder Alignment Review: sales, CS and engineering review the Problem Definition and scope and sign off explicitly. Silence isn't agreement, and disagreements are logged. Nothing is scored until it's aligned.human gate
StrategizeStrategic Narrative Agent, fed by the Competitive / Market Scan AgentLLM
EstimateEngineering Handoff: engineering's agents return Effort / Job Size.interface
ScoreRanking: RICE, Kano, WSJF. The LLM proposes subjective inputs and code computes, with a stability check. Shadow (v1.5): STARTIFY and RADAR score too, hidden from ranking and Review.LLM + code
SynthesizeSynthesis Agent surfaces score-vs-narrative and RICE-vs-WSJF disagreement (established frameworks only).LLM
SequenceMoSCoW Agent + WSJF Rank (a sort)LLM + code
ReviewThe PM reorders and overrides. Every override is tagged contractual / strategic-not-modeled / gut-call / other. Shadow scores aren't shown here (v1.5).human
Measure v1.4Measurement Plan Agent: a light check for every feature (template + code, with the LLM drafting only the hypothesis line). A full rubric plan plus the Measurement Plan Checker Agent only for exceptions the PM picks. The PM locks Pass if. Replaces the per-feature OKR / KPI Agent.code + LLM, checker for exceptions, human
DeliverUser Story → Acceptance Criteria / Edge Case (now with the plan's logging requirements) → Roadmap Narrative (now with the per-cycle OKR rollup) → Confluence/Jira → Release NotesLLM + code

Agent count

26 LLM agents in v1.2. v1.3 adds one human gate (Align). v1.4 adds two LLM agents (Measurement Plan, Measurement Plan Checker) and removes one (the per-feature OKR / KPI Agent), a net of +1. v1.5 adds one shadow agent (RADAR Score) beside the STARTIFY Agent, which is now also shadow, plus a per-cycle shadow evaluation job in code.

Also added: one human step (Pass if lock), a per-cycle tag-level measurement job (code) and a per-cycle OKR rollup. The usage classifier and the stakeholder labeling family change the count, and the final roster is still to be set.

Which document is current

SubjectMasterStatus
ArchitectureThe wiki page (v1.5)Older copies retired, superseded or merged
Layer viewRetired 2026-11-18Private HTML view "pipeline v1.4" is behind v1.5
Friction Stage Taggingfriction-stage-tagging.md in the projectSpec and publishing notes in the wiki
Article "Gas Town for PMs"The Claude doc (live)v1.4 and v1.5 changes laid as 32 suggestions on 2026-11-18, awaiting your review. Its diagram still shows "7 layers, 3 patrols".
Codegithub.com/io-world/PMingestionBuilds layers 1 and 2 of the pre-registry design

Not yet located anywhere: the separate ingestion-layer build spec and the Backlog Tagging Agent draft.

Shadow track: your own frameworks (v1.5)

Only established frameworks (RICE, Kano, WSJF, MoSCoW) rank features. Your own frameworks, STARTIFY and RADAR, are untested theories with no data behind them yet. They run in shadow: they score every feature, but their scores never reach the ranking, the Synthesis Agent or the Review gate (decided 2026-11-18). The shadow exists to produce the data you don't have: whether either framework adds anything.

STARTIFY Agent

Proposes the eight dimension scores after the Strategic Narrative, and code computes the /44 total. The narrative and market scan still run, because the narrative also feeds the Synthesis Agent's score-vs-narrative check.

RADAR Score Agent

The LLM proposes a −3 to +3 score per stakeholder group with evidence, and code computes confidence, net impact and priority. Team leads aren't asked to confirm rows, so an untested framework costs nobody's time. The team impact view, gates, capacity split and calibration loop don't run.

Shadow evaluation (code, each cycle, reported to you)

  • How well each shadow ranking agrees with your final order at the Review gate.
  • How it compares with the established frameworks' ranking: where they disagree, and who was right later.
  • Whether features a shadow framework ranked high got OK 60-day verdicts more often than those it ranked low.
  • Small numbers give direction, not proof, so read it over several cycles.
Promotion is a feature request (decided 2026-11-18)

There's no fixed number of cycles. When the shadow report suggests a framework adds something, promoting it goes through the pipeline like any other feature, with its own measurement plan stating what "adds something" means. The report itself changes nothing, so it stays within the no-loop-back rule.

Measurement Plan Agent (v1.4)

Is this feature OK, or did we build the wrong thing?

It isn't ongoing monitoring of every feature, and its results don't feed into building or scoring other features. Spec: the Measurement Plan Rubric v5.1, one plan per feature. It runs after Review, once a feature is committed, and before build, because the baseline has to exist before launch.

Default: the light check

  • Part A: the Gherkin line (pain + workaround → feature → new behavior → signal), the feature's tags (journey stage, theme_id, product area) and "Contributes to: [key result or KPI]". The only part the LLM drafts.
  • One Outcome row: Who / what · Signal · From · Baseline · By · Pass if · If it fails, preferably a tag-level signal already tracked.
  • One negative scenario: how it could look like a pass but be a fail.
  • One check at about 60 days. Verdict: OK / wrong feature / not measured.
  • Mostly template plus code. No checker agent.

Exceptions: the full rubric plan

Only when the PM chooses it: strategic bets, AI features, or results someone (Sales, Marketing) will repeat as a claim. The agent drafts all three parts, and the Measurement Plan Checker Agent scores it against the rubric's sub-items. It confirms, flags or rejects, never rewrites. Logging rows engineering confirms feasible become instrumentation acceptance criteria.

Human step: Pass if lock. The PM approves each Pass if bar before launch. It can't change once outcome data arrives.

Where results go: no loop back

The 60-day verdict is reported to you. It's never fed into scoring, prioritization or the Problem Definition of other features (decided 2026-11-17).

Tag-level measurement is the main ongoing view: a per-cycle code job (no LLM) reports metrics by tag across shipped features, like friction by journey stage per active account and tickets per tag. Tag-level signals may loop back, as the Friction Stage Score does (option a).

OKR link

  • Down: OKRs inform Problem Definition and Align, and each feature names the key result or KPI it contributes to.
  • Up: each cycle, the Roadmap Narrative reports each key result measured directly, with tag-level metrics and 60-day verdicts attached as evidence. Feature results are never added up into key-result progress.

Why the OKR / KPI Agent was removed

In v1.2 it ran per feature and drafted "the outcome metric a feature is held to". That's a feature success metric, not an OKR or KPI, and the light check now covers it. Keeping both would mean two agents drafting the same metric.

Data sources: a boundary, not a job

Building or repairing data sources is outside the pipeline (decided 2026-11-17). When an Outcome row depends on a source that isn't connected and validated, the verdict is "not measured". The agent only suggests how it could be measured, and that suggestion enters intake as a new candidate feature. No invented numbers and no hand-sample workarounds.

Still human by nature: task tests and AI correctness grading for exception plans. The agent suggests the method, and the PM assigns who does it. Evals: light checks are validated by code with spot checks, and full plans by a gold set of plans you wrote and scored.

Feedback loops

LoopWhat it doesFeeds back?
Friction Stage ScorePrimary metric is account breadth: accounts with friction in a stage ÷ active customer accounts. Secondary views are value-weighted breadth and conversation share. It lags, and must be read against what shipped.Yes, informs the next cycle
Prioritization Pattern AgentDraft vs. final order across cycles, by theme_id and rationale tag.Reported to you, never auto-fed into scoring
Feature check (v1.4)Each feature's 60-day verdict: OK / wrong feature / not measured.No, reported to you only
Tag-level measurement (v1.4)Per-cycle code job reporting metrics by tag across shipped features.Yes, may inform the next cycle
Backlog Tagging Agent (supply side)Customer-Journey-Driven vs. Engineering-Driven (Dependency, Infrastructure, Tech Debt, Compliance, Reliability/Ops, Data Integrity), with Security as a yes/no modifier, diffed against the Friction Stage Score.Diff against friction

Design principles

  1. Every agent states its uncertainty. Correlation is described, not asserted.
  2. Default to what a regular PM already does, e.g. Reach from analytics, not tickets.
  3. Two-agent rule from Gas City: an LLM step whose error would propagate unseen gets a checker that reads the evidence and confirms, flags or rejects, never rewrites. A Proposer/Critic shape.
  4. Engineering facts come from engineering's agents. PM agents never estimate effort.
  1. LLM calls only on judgment. Volume work is code.
  2. Align before scoring (v1.3). Stakeholders agree on the problem and scope before anything is ranked. The roadmap narrative reports a decision they already agreed to.
  3. Prove it before trusting it (v1.3). Every LLM checker and model choice is validated against the gold set, and the pipeline is measured against a manual baseline.
  4. Established frameworks rank; your own run in shadow (v1.5). A framework without outside credibility or data can score in shadow but can't rank until you promote it on shadow evidence.

Evals and pipeline metrics (v1.3)

  • Gold set (decided): a frozen set of records and features you labeled and ranked yourself. It measures classifier accuracy (which settles the small-model choice), Merge Checker and Problem Definition Checker agreement with your labels, and scorer drift. Blind spot-checks against anchoring at the gate stay in.
  • Baseline (decided): time and document one prioritization cycle done by hand before the pipeline replaces it.
  • Shadow run (added 2026-11-10): for the first real cycles, run the pipeline alongside the manual process. Its draft order is compared with yours but doesn't drive decisions, and it's promoted only once checker accuracy and override rate meet the bar. Source: shadow deployment in the AI Product Manager's Handbook (Ch. 1).

Framed publicly as one custom Gas City pack, with each PM job a formula.

Four-layer metrics

LayerMetric
BusinessPM hours per cycle vs. baseline; decision quality (share of features with an "OK" 60-day verdict vs. "wrong feature")
UserDraft order accepted vs. overridden; stakeholder alignment reached on the first pass
AI qualityClassifier accuracy, checker–human agreement, scorer stability, all on the gold set
SystemCost per cycle, run time per cycle
GuardrailOverrides tagged gut-call trending up; checker rejection rate spiking

Version history

2026-08-13

v1.0 — feedback loops

Feedback loops and override rationale. Earlier architecture notes are superseded.

2026-10-20

v1.2 — critical review

Two-agent checker rule, Engineering Handoff, Theme Registry, account-breadth friction scoring, code-computed scores. Reconciled the v1.0, v1.1 and wiki "v2" forks.

2026-11-10

v1.3 — PM Lenses review

A review against five PM Lenses (evals, metrics, business and cost, where to implement AI, owning failures) found five problems: no stakeholder alignment step (your own failure pattern), unvalidated checkers, a small model chosen without evals, no baseline or metrics, and "default to PM practice" skipping the eliminate rung. You chose to fix the agents. Added Align, the gold set, a provisional small model, the manual baseline, four-layer metrics, usage lookups, the stakeholder labeling family and principles 6–7. The architecture master moved to the wiki. A shadow run was added the same day.

2026-11-17

v1.4 — Measurement Plan Agent

Came from two days of measurement practice that produced the rubric through v5. You said the rubric had outgrown what you can hold in your head and should be its own agent. A same-day scope correction made it a light check per feature, a full plan only for exceptions, and ongoing measurement per tag, with no per-feature loop back. The per-feature OKR / KPI Agent was removed.

2026-11-18

v1.5 — shadow track

The deep lint noted RADAR wasn't in the pipeline and its calibration loop conflicted with no-loop-back. You first added RADAR as a fifth ranking framework, then pointed out that RADAR and STARTIFY are untested theories. Claude suggested a shadow track instead of removal, and you chose it. Ranking is now RICE, Kano and WSJF, with MoSCoW in Sequence. Principle 8 was added.

Carry-over status

Not yet updated

The "Gas Town for PMs" article has the v1.4 and v1.5 changes as suggestions. The private HTML view is still at v1.4.

Status and open questions

Status

  • Layers 1 and 2 run as PMingestion (k-means, re-clustered each run).
  • The theme registry, checkers, handoff and everything from Define onward are design only.
  • Not connected: CRM/billing, product analytics, Confluence/Jira, engineering agents.

Decided since v1.3

  • Usage data: lookups plus anomaly events, joined by a product-area catalog (three implementation decisions pending).
  • Gold set widened into the eval backbone.
  • Measurement Plan Agent replaces the per-feature OKR / KPI Agent.
  • Light check for every feature, full plan only for exceptions.
  • Building trusted sources is outside the pipeline.
  • The agent suggests measurement methods, and the PM assigns people.
  • Shadow promotion has no fixed cycle count.
? Open
  • Gold set size, and how it's refreshed.
  • Stakeholder Alignment Review: who's in the group per cycle, and what happens when they stay deadlocked?
  • Is each of the five scoring methods there for a decision or for trust? Framed as an eliminate-vs-adoption call (where to implement AI). Partly moot since STARTIFY moved to shadow.
  • Mayor agent, registry thresholds, labeling family for stakeholder and artifact sources, Gaps #2 to #4.
  • Shadow track: is the shadow LLM cost worth it on every feature, or only on medium and heavy ones?