Skip to content
ContractSpec
Start here

Package manual · platform

·Observability, testing and qualification

lib.agent-evals

Agent evaluation runners, adapters, graders, and recommendation evidence.

platformlibraryv2.2.4experimentalservernode
@lssm-tech/lib.agent-evals
bun add @lssm-tech/lib.agent-evals
Package source ↗Manifest ↗

Manifest declarations describe the current workspace; they do not verify registry publication or runtime execution.

Package guide

Deterministic agent evaluation runners, model adapters, graders, and recommendation evidence for ContractSpec.

What it provides

  • AgentEvaluationRunner for candidate-vs-baseline trial aggregation.
  • AgentEvalsFeature package metadata for docs and discovery.
  • Scripted and live model runner adapters.
  • Exact-match, schema, policy, artifact, and environment-outcome graders.
  • An LLM-judge adapter that turns an injected judge client into a grader.
  • Replay bundle refs, redaction leak detection, and cost/latency summaries.
  • Fail-closed output, blocking-grader, deterministic-safety, and trial-integrity gates for production-shaped candidate comparisons.
  • Route- and release-bound simulated qualification decisions with mandatory baseline evidence for model upgrades.
  • A checkpointed rollout state machine covering offline, shadow, internal, 1%, 10%, 50%, production, and evidence-backed rollback stages.
  • V2 immutable dataset and exact evaluation-subject bindings covering route, model profile, invocation settings, application purpose, evaluator set, environment, and release versions.
  • Reproducible verified-success, cost-per-verified-success, p50/p95 latency, structured-output, tool, fallback, provider, and human-review metrics by dataset segment.
  • Dataset digests include an application-owned confidence-interval or power analysis method, minimum sample count, and rationale reference. Missing samples, invalid segment weights, uncovered segments, or rebound cases fail before recommendation.
  • V2 comparison artifacts project trials to sanitized case/segment refs, numeric metrics, grader verdicts, replay digests, and boolean failure flags; prompts, responses, artifact bodies, metadata payloads, and replay URIs are never included.
  • CAS-ready qualification promotion/revocation and verified canary histories; expired, revoked, rebound, discontinuous, or re-digested rollout evidence fails closed.

Model evaluation never grants provider access or effect authority. A candidate becomes live_verified only after every rollout checkpoint has complete health evidence; a newer model is not promoted merely because it exists or scores well on average.

Local commands

  • bun run build
  • bun run typecheck
  • bun run test
  • bun run lint:check

Relationships

THIS PACKAGElib.agent-evals27 import paths

Arrows point from a package to its direct dependency. Counts are derived from captured workspace manifests.

3 declarations

Declared capabilities

Lib Agent Evals

Agent evaluation runners, adapters, graders, and recommendation evidence.

Scripts

  • publish:pkgbun publish --tolerate-republish --ignore-scripts --verbose
  • publish:pkg:canarybun publish:pkg --tag canary
  • buildcontractspec-bun-build build
  • build:bundlecontractspec-bun-build transpile
  • build:typescontractspec-bun-build types
  • devcontractspec-bun-build dev
  • cleanrimraf dist .turbo
  • lintbun run lint:fix
  • lint:fixnode ../../../../scripts/biome.cjs check --write --unsafe --only=nursery/useSortedClasses . && node ../../../../scripts/biome.cjs check --write .
  • lint:checknode ../../../../scripts/biome.cjs check .
  • testbun test
  • prebuildcontractspec-bun-build prebuild
  • typechecktsgo --noEmit