Skip to content
ContractSpec
Start here

Package manual · platform

·Integrations and provider selection

lib.provider-ranking

AI provider ranking: benchmark ingestion, scoring, and model comparison

platformlibraryv2.0.4experimentalservernode
@lssm-tech/lib.provider-ranking
bun add @lssm-tech/lib.provider-ranking
Package source ↗Manifest ↗

Manifest declarations describe the current workspace; they do not verify registry publication or runtime execution.

Package guide

Website: https://contractspec.io

AI provider ranking: benchmark ingestion, scoring, and model comparison.

What It Provides

  • Layer: lib.
  • Consumers: module.provider-ranking.
  • Related ContractSpec packages include @lssm-tech/tool.bun, @lssm-tech/tool.typescript.
  • Related ContractSpec packages include @lssm-tech/tool.bun, @lssm-tech/tool.typescript.

Installation

npm install @lssm-tech/lib.provider-ranking

or

bun add @lssm-tech/lib.provider-ranking

Usage

Import the root entrypoint from @lssm-tech/lib.provider-ranking, or choose a documented subpath when you only need one part of the package surface.

Architecture

  • src/eval is part of the package's public or composition surface.
  • src/in-memory-store.ts is part of the package's public or composition surface.
  • src/index.ts is the root public barrel and package entrypoint.
  • src/ingesters is part of the package's public or composition surface.
  • src/scoring is part of the package's public or composition surface.
  • src/store.ts is part of the package's public or composition surface.
  • src/types.ts is shared public type definitions.

Public Entry Points

  • Export . resolves through ./src/index.ts.
  • Export ./eval resolves through ./src/eval/index.ts.
  • Export ./eval/runner resolves through ./src/eval/runner.ts.
  • Export ./eval/types resolves through ./src/eval/types.ts.
  • Export ./in-memory-store resolves through ./src/in-memory-store.ts.
  • Export ./ingesters resolves through ./src/ingesters/index.ts.
  • Export ./ingesters/artificial-analysis resolves through ./src/ingesters/artificial-analysis.ts.
  • Export ./ingesters/chatbot-arena resolves through ./src/ingesters/chatbot-arena.ts.
  • Export ./ingesters/fetch-utils resolves through ./src/ingesters/fetch-utils.ts.
  • Export ./ingesters/open-llm-leaderboard resolves through ./src/ingesters/open-llm-leaderboard.ts.
  • Export ./ingesters/registry resolves through ./src/ingesters/registry.ts.
  • Export ./ingesters/swe-bench resolves through ./src/ingesters/swe-bench.ts.
  • Export ./ingesters/types resolves through ./src/ingesters/types.ts.
  • Export ./provider-ranking.feature resolves through ./src/provider-ranking.feature.ts.
  • Export ./routing resolves through ./src/routing/index.ts.
  • Export ./routing/classify-prompt resolves through ./src/routing/classify-prompt.ts.
  • Export ./routing/config resolves through ./src/routing/config.ts.
  • Export ./routing/escalation resolves through ./src/routing/escalation.ts.
  • Export ./routing/model-capabilities resolves through ./src/routing/model-capabilities.ts.
  • Export ./routing/profiles resolves through ./src/routing/profiles.ts.
  • Export ./routing/resolve-difficulty resolves through ./src/routing/resolve-difficulty.ts.
  • Export ./routing/role-matrix resolves through ./src/routing/role-matrix.ts.
  • Export ./routing/scoring resolves through ./src/routing/scoring.ts.
  • Export ./routing/select-model resolves through ./src/routing/select-model.ts.
  • Export ./routing/telemetry resolves through ./src/routing/telemetry.ts.
  • Export ./routing/types resolves through ./src/routing/types.ts.
  • Export ./routing/weights resolves through ./src/routing/weights.ts.
  • Export ./scoring resolves through ./src/scoring/index.ts.
  • Export ./scoring/composite-scorer resolves through ./src/scoring/composite-scorer.ts.
  • Export ./scoring/dimension-weights resolves through ./src/scoring/dimension-weights.ts.
  • Export ./scoring/normalizer resolves through ./src/scoring/normalizer.ts.
  • Export ./store resolves through ./src/store.ts.
  • Export ./types resolves through ./src/types.ts.
  • The package publishes 32 total export subpaths; keep docs aligned with package.json.

Local Commands

  • bun run dev — contractspec-bun-build dev
  • bun run build — contractspec-bun-build build
  • bun run test — bun test --pass-with-no-tests
  • bun run lint — bun lint:fix
  • bun run lint:check — biome check .
  • bun run lint:fix — biome check --write --unsafe --only=nursery/useSortedClasses . && biome check --write .
  • bun run typecheck — tsgo --noEmit
  • bun run publish:pkg — bun publish --tolerate-republish --ignore-scripts --verbose
  • bun run publish:pkg:canary — bun publish:pkg --tag canary
  • bun run clean — rimraf dist .turbo
  • bun run build:bundle — contractspec-bun-build transpile
  • bun run build:types — contractspec-bun-build types
  • bun run prebuild — contractspec-bun-build prebuild

Recent Updates

  • Replace eslint+prettier by biomejs to optimize speed.
  • Resolve lint, build, and type errors across nine packages.
  • Add first-class transport, auth, versioning, and BYOK support across all integrations.
  • Add AI provider ranking system with ranking-driven model selection.

Notes

  • Store interface is the adapter boundary — do not leak implementation details.
  • Scoring algorithms must stay deterministic (no randomness, no side effects).
  • Benchmark dimension enum is shared across ingesters and scoring — keep in sync.

Agent-aware routing

@lssm-tech/lib.provider-ranking/routing adds deterministic Builder model routing without executing providers. Use selectModelForAgent(input) to classify difficulty, apply manual thinking and risk floors, enforce required capabilities, select cheap/optimized/best/critical policy, and return an inspectable routing decision.

This surface is research/discovery and offline candidate-generation guidance. It is not Protocol V1 qualification, Connect authorization, persisted selection, fallback, or production routing truth. Every suggested candidate must still pass hard policy gates, operation-specific evaluation, durable Agent Host admission, and exact Connect verification before execution.

Critical risk (payments, contract-liability, compliance, destructive data loss, legal) cannot be downgraded by budget mode or manual thinking settings.

Canonical provider ranking contracts are available through the explicit ./contracts family. Existing contracts-spec subpaths preserve the same contract objects and nominal registry types. These exports do not execute workflows, fetch benchmarks or run paid evaluations.

Relationships

THIS PACKAGElib.provider-ranking59 import paths

Arrows point from a package to its direct dependency. Counts are derived from captured workspace manifests.

5 declarations

Declared capabilities

Lib Provider Ranking

AI provider ranking: benchmark ingestion, scoring, and model comparison

Scripts

  • publish:pkgbun publish --tolerate-republish --ignore-scripts --verbose
  • publish:pkg:canarybun publish:pkg --tag canary
  • buildcontractspec-bun-build build
  • build:bundlecontractspec-bun-build transpile
  • build:typescontractspec-bun-build types
  • devcontractspec-bun-build dev
  • cleanrimraf dist .turbo
  • lintbun lint:fix
  • lint:fixnode ../../../../scripts/biome.cjs check --write --unsafe --only=nursery/useSortedClasses . && node ../../../../scripts/biome.cjs check --write .
  • lint:checknode ../../../../scripts/biome.cjs check .
  • testbun test --pass-with-no-tests
  • prebuildcontractspec-bun-build prebuild
  • typechecktsgo --noEmit