lib.knowledge
RAG and knowledge base primitives
@lssm-tech/lib.knowledgebun add @lssm-tech/lib.knowledgeManifest declarations describe the current workspace; they do not verify registry publication or runtime execution.
Package guide
@lssm-tech/lib.knowledge provides the retrieval, ingestion, query, and access-control primitives used to turn documents and external content into searchable knowledge for agents and workflows.
Website: https://contractspec.io
Canonical source-management contracts are available under ./contracts/operations, ./contracts/operation-models, ./contracts/knowledge.capability, and ./contracts/knowledge.feature. The latter retains the platform.knowledge feature identity; the existing package feature remains libs.knowledge. The five built-in spaces (email-threads, financial-docs, financial-overview, support-faq, and uploaded-docs) live under ./contracts/spaces/ alongside Product Canon. Historical ContractSpec paths forward to these exact objects and the existing nominal registry. Source policies and registration behavior remain unchanged.
Installation
bun add @lssm-tech/lib.knowledge
or
npm install @lssm-tech/lib.knowledge
What belongs here
This package currently owns these related concerns:
- Retrieval contracts and implementations:
KnowledgeRetriever,StaticRetriever, andVectorRetriever. - Ingestion and indexing pipeline pieces:
DocumentProcessor,EmbeddingService,VectorIndexer, and ingestion adapters. - Retrieval-augmented query flow:
KnowledgeQueryServiceandKnowledgeQueryOptions. - Runtime convenience for OSS consumers:
KnowledgeRuntimeandcreateKnowledgeRuntime. - First-class Gmail and Google Drive orchestration for ingestion/query flows.
- Mutation governance helpers for dry-runs, approvals, idempotency, audit evidence, and outbound-send gates.
- CompanyOS connector readiness manifests for deterministic support desk, CRM, warehouse, collaboration, project, docs, email, and meeting coverage without live provider calls or production writes.
- Access guardrails and localization:
KnowledgeAccessGuardand thei18nsurface.
Use this package when you need the knowledge-layer primitives inside ContractSpec. It is not the source of truth for knowledge-space specs, tenant bindings, provider SDKs, or background job orchestration.
Core workflows
Quickstart: one runtime for ingest + retrieve + answer
import { createKnowledgeRuntime } from "@lssm-tech/lib.knowledge";
const knowledge = createKnowledgeRuntime({
collection: "knowledge-support-faq",
namespace: "tenant-acme",
spaceKey: "support-faq",
embeddings: embeddingProvider,
vectorStore: vectorStoreProvider,
llm: llmProvider,
});
await knowledge.ingestDocument({
id: "faq-rotate-key",
mimeType: "text/plain",
data: new TextEncoder().encode("Rotate API keys from Settings > API."),
metadata: { locale: "en", category: "canonical" },
});
const snippets = await knowledge.retrieve("How do I rotate a key?", {
spaceKey: "support-faq",
tenantId: "tenant-acme",
locale: "en",
category: "canonical",
});
const answer = await knowledge.query("How do I rotate a key?", {
namespace: "tenant-acme",
filter: { locale: "en" },
});Sync Gmail and Google Drive into knowledge
import { createKnowledgeRuntime } from "@lssm-tech/lib.knowledge";
const knowledge = createKnowledgeRuntime({
collection: "knowledge-workspace",
namespace: "tenant-acme",
spaceKey: "workspace",
embeddings: embeddingProvider,
vectorStore: vectorStoreProvider,
gmail: gmailProvider,
drive: googleDriveProvider,
deltaCheckpointStore,
});
await knowledge.syncGmail(
{ label: "support" },
{
sourceId: "src_gmail_support",
evidenceRef: "audit://sync/gmail/support",
},
);
await knowledge.syncDriveFiles(
{ query: "mimeType = 'text/plain' and trashed = false" },
{
sourceId: "src_drive_support",
evidenceRef: "audit://sync/drive/support",
},
);
await knowledge.watchDriveChanges(
{
channelId: "drive-support-watch",
webhookUrl: "https://app.example.com/webhooks/google-drive",
},
{ sourceId: "src_drive_support" },
);Gmail and Drive sync use provider contracts from @lssm-tech/lib.contracts-integrations. Delta-aware callers can pass and receive lease, cursor/watermark, webhook channel, provider-event, dedupe, idempotency, replay-checkpoint, and tombstone state through ProviderDeltaSyncState. When deltaCheckpointStore is configured, syncGmail(), syncDriveFiles(), and watchDriveChanges() load the last checkpoint by source ID before calling the provider, then save the returned provider delta before the caller acknowledges sync work.
Govern knowledge mutations
import {
executeGovernedKnowledgeMutation,
} from "@lssm-tech/lib.knowledge/governance";
const result = await executeGovernedKnowledgeMutation(
{
operation: "gmail.message.send",
requiresApproval: true,
outboundSend: true,
governance: {
idempotencyKey: "tenant:gmail-send:123",
auditEvidence: { evidenceRef: "audit://gmail-send/123" },
approvalRefs: [{ id: "approval-123" }],
outboundSendGate: {
status: "approved",
evidenceRef: "gate://gmail-send/123",
},
},
},
() => gmailProvider.sendMessage(message),
{ audit: (envelope) => auditTrail.write(envelope) },
);Set governance.dryRun to inspect the plan without executing. Non-dry-run mutations require an idempotency key and audit evidence; approval-required and outbound-send operations require approval refs, and outbound sends require an approved send gate. Every result includes an auditEnvelope so app runtimes, Connect review packets, and provider replay tooling can reference the same decision evidence.
Ingest documents into a vector index
import {
DocumentProcessor,
EmbeddingService,
StorageIngestionAdapter,
VectorIndexer,
} from "@lssm-tech/lib.knowledge";
const processor = new DocumentProcessor();
const embeddings = new EmbeddingService(embeddingProvider);
const indexer = new VectorIndexer(vectorStoreProvider, {
collection: "knowledge-docs",
namespace: "tenant-acme",
});
const adapter = new StorageIngestionAdapter(processor, embeddings, indexer);
await adapter.ingestObject(objectFromStorageProvider);Run retrieval or RAG queries
Verified grounded answers can be projected through projectGroundedResultToContextEvidenceV1. The projection revalidates trusted scope, citations, redaction identity, and replay fingerprints before returning viewer-scoped excerpts. It never grants instruction or effect authority.
import {
KnowledgeQueryService,
createVectorRetriever,
} from "@lssm-tech/lib.knowledge";
const retriever = createVectorRetriever({
embeddings: embeddingProvider,
vectorStore: vectorStoreProvider,
spaceCollections: {
"support-faq": "knowledge-support-faq",
},
});
const snippets = await retriever.retrieve("How do I rotate a key?", {
spaceKey: "support-faq",
topK: 3,
locale: "en",
category: "canonical",
});
const queryService = new KnowledgeQueryService(
embeddingProvider,
vectorStoreProvider,
llmProvider,
{
collection: "knowledge-support-faq",
namespace: "tenant-acme",
topK: 5,
}
);
const answer = await queryService.query("How do I rotate a key?", {
namespace: "tenant-acme",
topK: 3,
filter: { locale: "en" },
});Use static knowledge for tests or lightweight examples
import { createStaticRetriever } from "@lssm-tech/lib.knowledge";
const retriever = createStaticRetriever({
"product-canon": [
"Rotate API keys from Settings > API.",
"Only cite reviewed canon content in answers.",
].join("\n"),
});
const snippets = await retriever.retrieve("rotate", {
spaceKey: "product-canon",
topK: 1,
});Localize prompts and access messages
import { createKnowledgeI18n } from "@lssm-tech/lib.knowledge/i18n";
const i18n = createKnowledgeI18n("fr");
const noResults = i18n.t("query.noResults");Typical flow:
- Extract fragments from raw content.
- Embed fragments and upsert them into a vector collection.
- Use
GmailIngestionAdapter,DriveIngestionAdapter, or runtime sync helpers when content originates from workspace providers. - Retrieve snippets through a retriever or generate an answer through
KnowledgeQueryService. - Gate reads and writes with
KnowledgeAccessGuard, and gate mutations with the governance helpers when operating against external systems.
API map
Retrieval
KnowledgeRetriever: shared retrieval interface for semantic and static retrieval.RetrieverConfig: shared defaults for retriever implementations.StaticRetrieverandcreateStaticRetriever: in-memory, line-oriented retrieval for simple spaces and tests.VectorRetrieverandcreateVectorRetriever: embedding + vector-store backed retrieval.RetrievalOptions: query filters such asspaceKey,topK,minScore,tenantId,locale,category, and provider filter metadata.RetrievalResult: returned content, source, score, and optional metadata.
Query
KnowledgeQueryService: embed -> search -> prompt -> LLM chat flow for knowledge-backed answers.KnowledgeQueryConfig: collection, namespace, prompt, filter,topK, and locale defaults.KnowledgeQueryOptions: per-query overrides for namespace,topK, filter, locale, and prompt.KnowledgeAnswer: answer text, references, and optional token usage.
Ingestion
RawDocument,DocumentFragment, andDocumentProcessor: extract raw document content into indexable fragments.EmbeddingService: batch fragment embeddings through anEmbeddingProvider.VectorIndexerandVectorIndexConfig: map embeddings toVectorStoreProvider.upsert()requests and persist canonicalpayload.textcontent for later retrieval/query use.GmailIngestionAdapter: convert email threads into plaintext documents and index them.DriveIngestionAdapter: convert Google Drive files into raw documents, skip tombstones, and index them.StorageIngestionAdapter: fetch object content and run the same process/index pipeline.KnowledgeRuntimeandcreateKnowledgeRuntime: convenience composition for the common ingest -> sync -> retrieve -> answer path.KnowledgeProviderDeltaCheckpointStore: optional runtime checkpoint store for loading and saving Gmail/Drive provider delta state per source.KnowledgePersistencePort: optional database persistence port for fragments, sync checkpoints, and query evidence. It records provenance without changing retriever orKnowledgeAnswershapes.
Mutation governance
KnowledgeMutationRequest: describes a proposed mutation, whether it needs approval, and whether it sends outbound content.KnowledgeMutationGovernance: carries dry-run, approval refs, idempotency key, audit evidence, and outbound-send gate state.evaluateKnowledgeMutationGovernance: returns the plan, missing evidence, and block reasons without executing.executeGovernedKnowledgeMutation: executes only when the plan is allowed; otherwise returnsdry_runorblocked.
Grounded Company Intelligence retrieval
./groundingexposes an additive, fail-closed grounded-answer service. Authorization preflight runs before retrieval, and an empty or denied evidence scope produces zero retrieval, draft, and verification calls.- Hybrid channels are bounded to three calls, postvalidated against tenant, workspace, eligible evidence, ACL, and authorization-decision scope, then deterministically fused.
- Viewer redaction happens before citation-ledger creation, fingerprints, drafting, or verification. Draft and verifier ports see only server aliases plus redacted excerpts; they cannot mint public evidence, citation, policy, or authority refs.
- Drafting is strict and atomic, independent verification is mandatory, and there is at most one repair. Only a schema-valid, fully supported result has status
answered. adaptGroundedResultToKnowledgeAnswer()adapts only a semantically validatedansweredresult to the legacyKnowledgeAnswershape. Every unsafe state remains discriminated and cannot be flattened into a successful legacy answer.- Fixture adapters are reference-test infrastructure. Production mode rejects fixture channel or provider provenance with
fixture_adapter_forbidden. - Grounded knowledge is evidence, never action authority: every result fixes
knowledgeAuthorizesActiontofalse.
Original documents and scoped evidence
./evidenceadds original-document custody and document evidence.OriginalDocumentCustodyverifies size and SHA-256 against the delivered bytes, stores originals content-addressed and insert-only through anObjectStorageProvider, appends account-bound provenance for duplicate bytes, and quarantines media-type mismatches.DocumentExtractorRegistryships only a strict UTF-8 text extractor.registerPdfTextExtractor()andregisterOcrExtractor()accept the Builder runtime extractor shapes (for exampleDefaultBuilderPdfExtractor).extractWithReadability()returnsreadable,partial,unreadableorneeds_ocrwith typed reasons and a next action; empty output is never a success and partial OCR never claims completeness.DocumentEvidencePipelineruns custody → extraction →FragmentEvidenceRecords → a partitioned vector projection. Re-ingesting identical bytes with the same extractor plan replays the run; a new version supersedes the previous evidence. Production mode refuses a non-durableDocumentEvidenceLedger. Ledgers receive each run's scope, source and fragment text;readFragmentTexts()returns text for current evidence only and withdrawal erases it. The durable implementation lives in Company Intelligence (canonical CI records plus custody/extraction tables).retrieveScopedEvidence()searches only the context/account partitions granted to the viewer and re-validates each candidate against the ledger (current state, visibility, fragment fingerprint, original custody) before ranking.createScopedEvidenceGroundingChannel()plugs it into./grounding.resolveEvidenceCitations()resolves citations to the original (ref, digest, media type, size) and the fragment locator, and rejects withdrawn, superseded, foreign or tampered citations.consumeKnowledgeSourceWithdrawal()consumes theknowledge.source.withdrawnevent (./contracts/evidence-events): evidence becomeswithdrawnfirst, then vectors are deleted.createGmailAttachmentCustodyPort()andcreateEvidenceGmailWithdrawal()connect Gmail attachment hand-offs and thread withdrawals.resolveKnowledgeProcessingRoute()binds one explicit model route per workflow capability (exact model, account, data-processing terms, budget, explicit fallbacks); missing or invalid configuration isconfiguration_required.createRoutedChat()pins the route model, enforces the budget and records usage; indeterminate outcomes are never retried.
Guardrails and i18n
KnowledgeAccessGuardandKnowledgeAccessGuardOptions: policy-aware checks for read/write/search operations.KnowledgeAccessContextandKnowledgeAccessResult: runtime input/output contracts for access checks.createKnowledgeI18nandgetDefaultI18n: localization entrypoints for prompts, guard messages, and ingestion formatting../i18n,./i18n/catalogs/*,./i18n/keys,./i18n/locale, and./i18n/messages: public i18n subpaths.
Public entrypoints
The root barrel at src/index.ts re-exports public symbols from:
./access./ingestion./knowledge.feature./query./runtime./retriever./types./vector-payload
Published subpaths from package.json are grouped around:
- access:
./access,./access/guard - retrieval:
./retriever,./retriever/interface,./retriever/static-retriever,./retriever/vector-retriever - query:
./query,./query/service - ingestion:
./ingestion,./ingestion/document-processor,./ingestion/drive-adapter,./ingestion/embedding-service,./ingestion/gmail-adapter,./ingestion/storage-adapter,./ingestion/vector-indexer - governance:
./governance - grounded retrieval:
./grounding - document evidence:
./evidence,./contracts/evidence-events - feature/runtime helpers:
./knowledge.feature, rootKnowledgeRuntime/createKnowledgeRuntime - i18n:
./i18n,./i18n/catalogs,./i18n/catalogs/en,./i18n/catalogs/es,./i18n/catalogs/fr,./i18n/keys,./i18n/locale,./i18n/messages - shared types/helpers:
./types,./vector-payload
Use package.json as the exhaustive source of truth for subpaths; the README calls out the clusters that matter most to consumers.
Operational semantics and gotchas
DocumentProcessoronly ships built-in extractors fortext/plainandapplication/json.- If no extractor matches,
DocumentProcessor.process()throws unless you registered a fallback such as*/*. - If an extractor returns no fragments, or only whitespace fragments,
DocumentProcessor.process()returns[]so nothing unreadable is embedded;VectorIndexer.upsert()skips empty batches. UseextractWithReadability()for an explicit readability state. EmbeddingServicebatches fragments; the default batch size is16.StaticRetriever.retrieve()does simple line-level substring matching. It is not semantic retrieval.StaticRetrieverhonorstopKandminScore, but its scores are always1.0for matching lines.VectorRetriever.retrieve()returns[]when the requestedspaceKeyis not mapped to a collection.VectorRetrieverusestenantIdas the vector-store namespace when provided and automatically merges typedlocale/categoryfilters into the provider filter payload.VectorIndexer.upsert()stores fragment text underpayload.text, together with merged fragment/config metadata anddocumentId. Fragment metadata wins over config metadata when keys collide.KnowledgeQueryServicealways runs embed -> vector search -> prompt build ->LLMProvider.chat(), using indexedpayload.textas the primary context source and falling back to legacypayload.contentwhen needed. Per-query overrides let you change namespace,topK, filter, locale, and prompt without rebuilding the service.KnowledgeAccessGuarddefaults to blocking writes toexternalandephemeralknowledge categories.KnowledgeAccessGuardalso blocks writes whenResolvedKnowledge.space.access.automationWritableisfalse.KnowledgeAccessGuarddefaults to requiring workflow binding and not requiring agent binding. If a scoped workflow or agent allow-list exists, missing names are denied rather than silently bypassed.GmailIngestionAdapterconverts threads into plaintext and strips HTML when text bodies are missing.DriveIngestionAdapterindexes Google Drive files through the same processor -> embedding -> indexer path and skips tombstoned metadata or file records before indexing.KnowledgeRuntime.syncGmail(),KnowledgeRuntime.syncDriveFiles(), andKnowledgeRuntime.watchDriveChanges()are convenience orchestration methods; provider scheduling, credential refresh, and long-running background sync remain outside this package.- Runtime sync helpers load and save
ProviderDeltaSyncStatethrough the optional checkpoint store when a source ID is provided. KnowledgeRuntimecan call an optional persistence port after ingestion/query to store fragments and query evidence; failures are explicit host-adapter concerns rather than hidden vector-store behavior.executeGovernedKnowledgeMutation()never executes dry-runs or blocked mutations.- Grounded retrieval never performs provider discovery, credential resolution, persistence, dispatch, or action execution. Hosts own runtime adapters and must preserve the authorization/redaction boundary.
- Non-dry-run mutations require
idempotencyKeyandauditEvidence.evidenceRef. - Mutation execution returns an audit envelope and can call an audit sink before returning.
- Outbound sends require an approved
outboundSendGateand at least one approval ref. - This package localizes access messages, query prompts, and Gmail formatting through the
i18nsurface.
When not to use this package
- Do not use it as a vector store implementation.
- Do not use it as an embedding-model implementation.
- Do not use it as a full sync scheduler or background job system.
- Do not use it as the source of truth for
KnowledgeSpaceSpec, knowledge sources, or tenant bindings.
Related packages
@lssm-tech/lib.contracts-integrations: provider interfaces used by vector, embedding, LLM, email, and storage integrations.@lssm-tech/lib.contracts-spec: source of knowledge-space specs, resolved bindings, and shared translation helpers.@lssm-tech/lib.ai-agent: major consumer of retrieval and query contracts.@lssm-tech/example.knowledge-canon: runnable package-level example of binding knowledge spaces and answering throughlib.knowledge.- Context-storage modules and bundles in the repo reuse the ingestion pipeline primitives for document indexing flows.
Local commands
bun run lint:checkbun run typecheckbun run testbun run test:company-intelligence:grounding
Grounding uses the pure Core grounding contracts and validator. Legacy adapters keep their original two-argument calls, fail-closed schema/semantic checks and wire results. Company host bindings and evidence projections remain outside Core.
Canonical ownership
The complete portable Knowledge binding/source/spec families remain canonical Core forwards (including the current complete source helper). Knowledge owns the original nominal KnowledgeSpaceRegistry and Product Canon singleton. The access guard retains its exact full ResolvedKnowledge/ResolvedAppConfig signatures through the canonical RuntimeCore host types; no generic widening or host-field clone is introduced. Old registry and Product Canon leaves forward the same implementations. Source-only qualification precedes task adoption and affected producer/consumer execution.
Relationships
Arrows point from a package to its direct dependency. Counts are derived from captured workspace manifests.
7 declarations
zodcatalog:
@lssm-tech/lib.contracts-integrationsworkspace:^
Read the package manual ↗@lssm-tech/lib.contracts-spec-coreworkspace:^
Read the package manual ↗@lssm-tech/lib.contracts-runtime-coreworkspace:^
Read the package manual ↗@lssm-tech/lib.contracts-runtime-kernelworkspace:^
Read the package manual ↗@lssm-tech/lib.translation-runtimeworkspace:^
Read the package manual ↗
Declared capabilities
RAG and knowledge base primitives
Scripts
publish:pkgbun publish --tolerate-republish --ignore-scripts --verbosepublish:pkg:canarybun publish:pkg --tag canarybuildcontractspec-bun-build buildbuild:bundlecontractspec-bun-build transpilebuild:typescontractspec-bun-build typesdevcontractspec-bun-build devcleanrimraf dist .turbolintbun lint:fixlint:fixnode ../../../../scripts/biome.cjs check --write --unsafe --only=nursery/useSortedClasses . && node ../../../../scripts/biome.cjs check --write .lint:checknode ../../../../scripts/biome.cjs check .testbun test --pass-with-no-teststest:company-intelligence:groundingbun test testdata/company-intelligence/grounding/grounding.test.tsprebuildcontractspec-bun-build prebuildtypechecktsgo --noEmit