Skip to content
Agent OS
AOS-VSN-003 Normative foundation

Success Metrics and Product Evidence

Success Metrics and Product Evidence: scope, decisions, requirements, evidence, risks, and traceability for the Agent OS programme.

Edit on GitHub →

Success Metrics and Product Evidence

This specification defines the product thesis, intended user value, measurable evidence, and boundaries that prevent a vision statement from becoming an unverifiable platform promise.

Table of Contents

Purpose and Scope

Area: Product Strategy.

This specification defines the product thesis, intended user value, measurable evidence, and boundaries that prevent a vision statement from becoming an unverifiable platform promise.

This document owns the semantics implied by Success Metrics and Product Evidence. It does not assert that every described subsystem already exists. It defines the target model, constraints, evidence needed to trust an implementation, and the boundary with adjacent documents.

Normative Position

  1. Define the boundary, owner, inputs, outputs, failure modes, observability, security authority, lifecycle, and compatibility policy.
  2. Keep implementation facts separate from desired outcomes and unverified assumptions.
  3. Require a reproducible evidence bundle before downstream components depend on the result.

Operating Model

The operating model is contract-first and evidence-driven. A component declares its authority, resources, lifecycle, error model, cancellation and timeout behavior, observability, version, and compatibility promise. Backends are replaceable only when the same conformance suite passes and no forbidden platform type leaks into portable layers.

Implementation proceeds through a reference model or mock, deterministic QEMU evidence where relevant, documentation-first physical hardware, and quality-hardware evidence. Pixel 9 adapters remain quarantined according to ADR-0004.

Requirements

  • R01. Define the boundary, owner, inputs, outputs, failure modes, observability, security authority, lifecycle, and compatibility policy.
  • R02. Keep implementation facts separate from desired outcomes and unverified assumptions.
  • R03. Require a reproducible evidence bundle before downstream components depend on the result.
  • R04. Specify normal, partial, denied, timeout, cancellation, restart, upgrade, and permanent-failure behavior.
  • R05. Expose structured diagnostics without leaking secrets or vendor-specific implementation details.
  • R06. Link material unknowns to a claim and, when testable, an experiment with an owner and gate.
  • R07. Update affected documentation and task data when evidence changes the model.

Failure and Degradation

Degradation must be explicit rather than accidental. The system reports capability absence, reduced quality, unavailable provider, stale data, or unsafe condition through typed states. It must not silently fall back to broader authority, unrestricted legacy execution, unverified firmware, lossy data migration, or irreversible agent action.

Recovery defines what state is retained, reconstructed, re-enrolled, compensated, or intentionally discarded. Unsupported hardware or providers are rejected at binding time where possible.

Evidence and Acceptance

  • Reviewed specification.
  • Linked tasks, claims, sources and experiments.
  • Conformance or acceptance evidence.
  • Evidence records target identity, hardware revision, firmware, source commit, toolchain, configuration, seed, timestamps, artifacts, expected result, actual result, and reviewer.
  • Acceptance requires the referenced tasks to meet their own criteria; prose completion is not implementation completion.

Implementation Obligations

Task Obligation Priority Gate/Milestone Verification
AOS-PROD-080 Benchmark entity-first shell against task baselines P1 M8 EXP-070 with pilot then appropriately sized study and accessibility representation

Risks and Open Questions

  • Undefined ownership and failure semantics create hidden coupling.
  • Unverified source claims can become architecture accidentally.
  • Open-question rule: an unanswered high-impact question becomes a claim/experiment record and cannot be hidden in meeting notes.
  • Stop rule: work stops or changes track when legal rights, recovery, debug access, safety, or the required evidence path is unavailable.

Related Documents

Planning Reference Anchors

Product Evidence

AOS-PROD-080 — Benchmark entity-first shell against task baselines