lib.agent-evals
Agent evaluation runners, adapters, graders, and recommendation evidence.
@lssm-tech/lib.agent-evalsbun add @lssm-tech/lib.agent-evalsManifest declarations describe the current workspace; they do not verify registry publication or runtime execution.
Package guide
Deterministic agent evaluation runners, model adapters, graders, and recommendation evidence for ContractSpec.
What it provides
AgentEvaluationRunnerfor candidate-vs-baseline trial aggregation.AgentEvalsFeaturepackage metadata for docs and discovery.- Scripted and live model runner adapters.
- Exact-match, schema, policy, artifact, and environment-outcome graders.
- An LLM-judge adapter that turns an injected judge client into a grader.
- Replay bundle refs, redaction leak detection, and cost/latency summaries.
- Fail-closed output, blocking-grader, deterministic-safety, and trial-integrity gates for production-shaped candidate comparisons.
- Route- and release-bound simulated qualification decisions with mandatory baseline evidence for model upgrades.
- A checkpointed rollout state machine covering offline, shadow, internal, 1%, 10%, 50%, production, and evidence-backed rollback stages.
- V2 immutable dataset and exact evaluation-subject bindings covering route, model profile, invocation settings, application purpose, evaluator set, environment, and release versions.
- Reproducible verified-success, cost-per-verified-success, p50/p95 latency, structured-output, tool, fallback, provider, and human-review metrics by dataset segment.
- Dataset digests include an application-owned confidence-interval or power analysis method, minimum sample count, and rationale reference. Missing samples, invalid segment weights, uncovered segments, or rebound cases fail before recommendation.
- V2 comparison artifacts project trials to sanitized case/segment refs, numeric metrics, grader verdicts, replay digests, and boolean failure flags; prompts, responses, artifact bodies, metadata payloads, and replay URIs are never included.
- CAS-ready qualification promotion/revocation and verified canary histories; expired, revoked, rebound, discontinuous, or re-digested rollout evidence fails closed.
Model evaluation never grants provider access or effect authority. A candidate becomes live_verified only after every rollout checkpoint has complete health evidence; a newer model is not promoted merely because it exists or scores well on average.
Local commands
bun run buildbun run typecheckbun run testbun run lint:check
Relationships
Arrows point from a package to its direct dependency. Counts are derived from captured workspace manifests.
3 declarations
@lssm-tech/lib.agent-protocol-specworkspace:^
Read the package manual ↗@lssm-tech/lib.assurance-specworkspace:^
Read the package manual ↗@lssm-tech/lib.contracts-spec-coreworkspace:^
Read the package manual ↗
Declared capabilities
Agent evaluation runners, adapters, graders, and recommendation evidence.
Scripts
publish:pkgbun publish --tolerate-republish --ignore-scripts --verbosepublish:pkg:canarybun publish:pkg --tag canarybuildcontractspec-bun-build buildbuild:bundlecontractspec-bun-build transpilebuild:typescontractspec-bun-build typesdevcontractspec-bun-build devcleanrimraf dist .turbolintbun run lint:fixlint:fixnode ../../../../scripts/biome.cjs check --write --unsafe --only=nursery/useSortedClasses . && node ../../../../scripts/biome.cjs check --write .lint:checknode ../../../../scripts/biome.cjs check .testbun testprebuildcontractspec-bun-build prebuildtypechecktsgo --noEmit