AGENTS AND EVALUATION

AI Systems

Agents for engineering workflows where the output must be useful, reviewable, and safe to operate.

Agent design

The design goal is a bounded engineering system, not an unconstrained chatbot. Tools, context, permissions, and completion criteria are all explicit. Typical capabilities:

Test authoring and scenario generation
Python-to-TypeScript test migration
Test validation and stabilization
Runtime investigation through scoped tools
Skill and SOP loading based on the task
Structured summaries of work performed and remaining gaps

Engineering architecture

The systems are organized as a pipeline rather than a single model call:

Layer	Responsibility
Task contract	Define the scenario, inputs, expected artifacts, and failure conditions
Agent runtime	Load the relevant skills and SOPs, then use only scoped tools
Execution boundary	Separate authoring, validation, and optional runtime execution
Evaluation gates	Apply deterministic checks before semantic judging
Evidence flow	Persist artifacts, verdicts, diagnostics, and upload metadata

This separation makes it possible to distinguish an agent defect from a tool, infrastructure, or environment failure.

Deterministic benchmarks

Agent quality needs a repeatable measurement layer:

flowchart TD A["Curated ground truth"] --> B["Repeatable commit or scenario data"] B --> C["Agent tool calls"] C --> D["Deterministic gates (exact checks first)"] D --> E["Semantic judging (only where exact checks fall short)"] E --> F["Structured verdict + evidence report"] classDef gate fill:#0e7490,color:#fbfaf6,stroke:#0e7490 classDef verdict fill:#12212b,color:#b9eee8,stroke:#12212b class D,E gate class F verdict

Verdict	Meaning
Match	The agent identifies the key expected result
Partial match	The agent identifies part of the expected result
Miss	The agent fails to identify the important result
Error	The evaluation could not complete reliably

Making LLM evaluation trustworthy

Keep scenario inputs and expected outcomes versioned.
Use deterministic string or rule checks for safety-critical conditions.
Use semantic judging only where exact matching is insufficient.
Record evidence, not just a score.
Separate agent quality from infrastructure or tool failures.
Re-run misses and partial matches through a diagnosis workflow.

Why this matters

The goal is not to make an agent appear intelligent. The goal is to know when it is correct, when it is uncertain, and when it should not be trusted.