Skip to content

PATTERN Cited by 1 source

Report agent self-validates schema

Intent

The terminal stage of an AI pipeline is a reporting agent that takes the upstream findings and writes them against a predefined schema — but instead of relying on the model to emit perfectly-conforming output the first time, the agent runs the schema validator on its own output, reads the validation errors, fixes them itself, and only submits to the ingest API when the validator passes.

Canonical articulation

Cloudflare's Report stage of the vulnerability discovery harness (Source: sources/2026-05-18-cloudflare-project-glasswing-what-mythos-showed-us):

"An agent writes a structured report against a predefined schema, fixes any validation errors against that schema itself, and submits the report to an ingest API. Output is queryable data, not free-form prose."

The shape

upstream findings (Hunt/Validate/Trace output)
    │
    ▼
report agent generates initial structured output
    │
    ▼
schema validator
    │
    ▼
validates? ── yes → submit to ingest API
    │
    no
    │
    ▼
agent reads validation errors, fixes them
    │
    └──── (loop until valid or budget exhausted)

Why "free-form prose" is the wrong terminal output

The Cloudflare verbatim is direct: "Output is queryable data, not free-form prose." The pipeline is feeding a downstream system (Cloudflare's vulnerability management process); that system needs records that can be:

  • Indexed and queried — by severity, attack class, consumer repo, etc.
  • Joined with other data — CVE records, ticketing, fleet inventory.
  • Aggregated — for dashboards, executive reporting.
  • Diffed — across pipeline runs, to track new vs resolved findings.

A free-form prose report serves exactly none of these use cases at fleet scale. The Report stage exists because the value of the harness output is downstream consumption, not human-readable narrative.

Why the agent self-validates instead of relying on

constrained decoding

Constrained decoding (forcing the model to emit only schema-valid tokens) is the alternative; Cloudflare's pattern explicitly does the validate-and-fix loop instead. Reasons in tension:

  • Constrained decoding can degrade content quality. Forcing tokens to be schema-valid can cause the model to emit technically valid but semantically wrong content — fields filled with placeholders, summaries that don't summarise, etc.
  • Validate-and-fix lets the model think first, format second. The model can produce its best-quality content first and then reshape it to fit the schema, rather than formatting and content-generating simultaneously.
  • Validation feedback is a useful learning signal. When a schema fails (e.g., "severity must be one of CRITICAL/ HIGH/MEDIUM/LOW"), the validator's error message itself helps the model understand the constraint.

The pattern is a sibling of decouple-reasoning-from-structured-output — generate content first, then format — applied with an explicit validation loop.

Why the agent fixes the errors itself

Pushing validation errors back to the user (or to a manual fix-up step) negates the harness's queryable-data property — the pipeline would no longer produce ready-to-consume output. The model is already in scope for content generation; asking it to also handle its own format violations is a small marginal cost.

This is a specialisation of the agent self-correction loop applied to schema conformance, exactly analogous to proof by compile-and-run applied to runtime-behaviour conformance — the "compiler" in this case is the schema validator, the "run" is the serialisation, and the "fix" is regenerating the output section that violated the schema.

Sibling patterns on the wiki

Pattern What gets validated What runs the validation
Report agent self-validates schema Output against schema Schema validator inside the agent loop
patterns/proof-by-compile-and-run Hypothesised exploit behaviour Compile-and-run in scratch env
patterns/schema-validation-before-deploy Config CI / deploy pipeline (out of agent loop)
patterns/client-side-schema-validation User input Client-side runtime

The agent-internal validation loop is what distinguishes this pattern from the deploy-time / build-time validation patterns: the agent fixes the issue rather than just reporting it.

What the pattern requires

  • Predefined schema. A specific, declared shape the output conforms to (JSON Schema, protobuf, OpenAPI, etc.).
  • Programmatic validator. A library or service the agent can call to check its own output.
  • Tool integration. The validator must be exposed as a tool the agent can call.
  • Fix-up budget cap — like the patterns/proof-by-compile-and-run loop, this loop must terminate; budget exhaustion implies the report can't be produced and should be flagged for human review.

What the pattern doesn't fix

  • Content quality. A schema-valid report can still contain wrong information. The pattern is a format guarantee, not a truth guarantee. Quality is upstream (Validate stage's job).
  • Schema evolution. When the downstream system's schema changes, this pattern doesn't catch it automatically — the agent has whatever schema it was given.
  • Semantic conflicts between fields (e.g., severity CRITICAL but no PoC attached) — typically not expressible as schema validations and must be handled separately.

Generalises across the wiki

The pattern is a specialisation of self-validation loops that show up across AI pipelines:

  • Test-then-fix loops in coding agents — agent runs the test suite, reads failures, fixes failures.
  • Lint-then-fix loops in code-review pipelines.
  • Type-check-then-fix loops in TypeScript / Rust / similar agents.

Open / not disclosed

  • Cloudflare's specific schema shape — the post mentions "a predefined schema" but doesn't publish it.
  • Ingest API destination — not named.
  • Failure-rate of the loop — how often it converges vs exhausts budget.

Seen in

Last updated · 542 distilled / 1,571 read