dsh-llm

English | 中文

Provider-neutral LLM vocabulary and abstract service. This package defines the canonical language spoken by the agent loop, session logs, and every plugin.

Service: LlmRuntime (ctx key: llm)

An adapter registry plus a single streaming call API, interceptable via a waterfall event.

Public API

• ctx.llm.registerAdapter(providers: string[], adapter: LlmAdapter): AdapterRegistrationHandle Register one adapter instance for the given provider routes. Registration is all-or-nothing, and is disposed with the calling fiber. The returned disposer also carries replace(providers): the candidate route set is validated in full before anything moves, so a conflict with another adapter leaves the current routes registered and serving, and the swap itself is one synchronous section with no observable gap. replace([]) is legal — a registration holding zero routes — unlike an empty initial registration.
• ctx.llm.listProviders(): LlmProviderInfo[] Describe registered provider routes in registration order.
• ctx.llm.registerConfigurableProviders(entries: readonly LlmConfigurableProvider[]): DirectoryRegistrationHandle Declare provider routes an adapter plugin can activate through configuration — registered or dormant — each naming its owning settings namespace and the path to its profile inside that section. All-or-nothing (INVALID_DIRECTORY/DUPLICATE_DIRECTORY), disposed with the calling fiber. The handle also carries replace(entries): the candidate set is validated in full before anything moves, so an entry another registration already declares leaves the current set intact, and an empty array is legal there. A plugin whose declared set follows its configuration must use replace rather than disposing and re-registering — the latter strands the directory empty whenever the new set is refused.
• ctx.llm.listConfigurableProviders(): LlmConfigurableProvider[] List the declared directory in declaration order; configuration surfaces merge it with listProviders() to mark each entry live or dormant. An entry may carry declared — whether the owning adapter knows that route only because configuration named it. Only the adapter can answer, so absence means "this adapter draws no such distinction", never "shipped".
• ctx.llm.registerModelDiscovery(settingsNs: string, discover): () => void Offer to interrogate provider endpoints for the settings namespace this plugin owns. One offer per namespace (INVALID_DISCOVERY/DUPLICATE_DISCOVERY), disposed with the calling fiber.
• ctx.llm.listModelDiscoveryNamespaces(): string[] List the namespaces that can interrogate an endpoint, so a surface offers the action only where it works.
• ctx.llm.discoverModels(settingsNs: string, request: LlmModelDiscoveryRequest): Promise<LlmDiscoveredModel[]> Ask one endpoint which models it advertises.
• ctx.llm.providerRetryPolicy(provider: string): ResolvedRetryPolicy Return the provider-owned retry policy captured during registration, with normal defaults resolved.
• ctx.llm.listModels(provider: string): Promise<LlmModelInfo[]> Discover the models one registered provider currently advertises.
• ctx.llm.resolveModelInfo(provider: string, model: string, signal?: AbortSignal): Promise<LlmResolvedModelInfo> Resolve validated exact-model identity plus available context, output-default, and reasoning metadata from the owning adapter, with optional cancellation for asynchronous adapters.
• ctx.llm.resolveCallConfig(config: LlmCallConfig, signal?: AbortSignal): Promise<LlmCallConfig> Validate an explicit effort and materialize adapter-configured call defaults without clamping.
• ctx.llm.prepareCall(config: LlmCallConfig, signal?: AbortSignal): Promise<PreparedLlmCall> Resolve a config plus detached context metadata and markers for fields supplied by adapter defaults in one exact-model lookup, then capture its current adapter registration and immutable retry policy as one cancellable, one-shot call.
• ctx.llm.stream(options: GenerateOptions): AsyncIterable<StreamChunk> Stream one model call as raw chunks (token-level deltas). Consumers assemble the chunks into blocks/messages with BlockAssembler.

LlmRuntime normalizes failures from final adapter selection, synchronous dispatch, iterator construction, and iteration into the stream protocol's single terminal form: finish { kind: 'error' | 'aborted', failure }. A failure after partial deltas may leave content blocks open; consumers discard that incomplete output. Errors from llm/stream middleware, nested calls, adapter cleanup, and downstream consumers remain thrown because they are plugin or consumer failures rather than model-request outcomes. A prepared call exposes the immutable retry policy captured with its exact adapter registration; a route handled entirely by middleware has no serving policy.

Interrogating an endpoint is configuration-time work over a draft, keyed by settings namespace rather than by provider route — the provider a surface is adding does not exist yet, so there is no route to name. The request may still name a route it is editing, and an adapter that already describes that route answers from its own knowledge without a network call; baseURL is optional and one of the two is required. The request otherwise carries the endpoint, the protocol, and a credential the harness uses for that one interrogation and never stores — nothing here reads or writes settings or credentials, and the reply is candidate metadata a surface may offer for adoption, never a registered catalog. LlmDiscoveredModel makes every field but id optional because most provider listings disclose an id and nothing else; a surface adopting one still owes the capacities its adapter requires. Duplicate and unusable ids are dropped, an unserved namespace fails with NO_DISCOVERY, and a request naming neither a route nor an endpoint fails with INVALID_DISCOVERY.

Provider and model metadata is a discovery surface, not a routing whitelist. registerAdapter() still owns provider exclusivity and captures the adapter's retry policy for each route, while an adapter may accept model ids absent from listModels(); consumers must not reject a request because its model is unlisted. Returned selector metadata is detached and invalid or duplicate adapter entries fail with INVALID_ADAPTER or INVALID_CATALOG.

Every topology commit point — adapter routes registering or disposing, directory entries appearing or withdrawing — emits the payload-free llm/adapters-updated event after the mutation, so consumers re-read listProviders()/listModels()/listConfigurableProviders() instead of polling. Observer failures are contained (logged, non-vetoing); only INVARIANT-coded failures rethrow after the fan-out.

Exact-model metadata is a separate correctness query, not a catalog decoration or global LLM setting. resolveModelInfo() asks the adapter that owns the exact provider/model route once; an adapter can describe an unlisted dynamic model, and absent context, defaultMaxTokens, or reasoning fields preserve unknown capacity, provider-owned output defaults, or unavailable reasoning capability. Invalid identity, context, output default, or reasoning metadata fails with INVALID_MODEL_INFO, INVALID_MODEL_CONTEXT, INVALID_MODEL_MAX_TOKENS, or INVALID_MODEL_REASONING.

defaultMaxTokens is an adapter-configured per-request output cap, not a model hard limit. resolveCallConfig() materializes it only when the request omits maxTokens; an explicit cap wins. Reasoning identifiers are opaque adapter-owned strings rather than a core enum: the same resolution accepts only an exact advertised identifier, materializes defaultEffort when present, and otherwise preserves the provider default. Asynchronous model resolvers receive the caller's signal and must settle promptly after cancellation. prepareCall() additionally exposes detached context metadata from the same lookup, reports which maxTokens and reasoningEffort fields it materialized in adapterDefaults, and retains the exact adapter registration through header logging and terminal dispatch, so HMR cannot combine one adapter's capability result with another adapter's request; reusing its one-shot handle or changing its call-config fields fails with INVALID_PREPARED_CALL. An unsupported explicit or configured effort fails with UNSUPPORTED_REASONING_EFFORT before provider I/O.

Events

Extension points

• Subclass LlmAdapter and call ctx.llm.registerAdapter(providers, adapter) to add one or more provider routes. GenerateOptions.provider selects the adapter; GenerateOptions.model is adapter-owned and may be resolved dynamically. Override providerRetryPolicy() to supply provider-owned recovery configuration, providerInfo() and asynchronous listModels() to expose selector metadata, then implement resolveModel() when exact identity, capacity, an output default, or selectable reasoning efforts are available; an asynchronous resolver must honor its optional cancellation signal. The defaults use bounded normal retry policy, use the route and model ids as names, advertise no models, and return no capacity, output default, or reasoning metadata.
• Wrap llm/stream via ctx.on() waterfall listeners for caching, logging, or routing. A wrapper that retries after emitting a chunk has no durable attempt boundary; shipped agent retry policy therefore uses agent/request-error instead.

Messages (message.ts) and content blocks (types.ts)

Message is the shared immutable value used by delivery, durable history, and model requests. Every message has a required MessageId, role, content, and typed source from creation onward. createMessage(input) mints the identity and returns a detached deep-frozen value; createUserMessage({ content, source }) fixes the user role; createAssistantMessage({ content, source }) fixes the assistant role and model source kind; createToolResultMessage({ callId, content, isError }) fixes the user role and couples the tool source to its result block; freezeMessage(message) imports an identity that already exists and never replaces it. Message rewrites preserve the identity and produce another frozen value. Browser code imports these value constructors from the dependency-minimal @deepseek-ai/dsh-llm/message entry instead of the service-bearing package root.

Message content is an array of typed blocks: text, reasoning, tool-call, tool-result. The union is derived from the merge-extensible ContentBlockMap, so plugins can add block types via declaration merging. Assistant messages use a model source carrying the provider and model that produced them plus optional adapter-private replay state. Before dispatch, LlmRuntime retains that state only when the historical provider route and target provider route are currently owned by the exact same adapter instance; the adapter then decides whether it can restore or convert the state across models/providers. The core block set is limited to blocks every shipping path honors — multimodal content (images, audio, …) has no core block type; a feature that needs one adds it via the map together with the adapter/UI/compaction support that honors it.

Streaming is a raw chunk protocol (block-start, text-delta, reasoning-delta, tool-call-delta, block-end, usage, finish). Every adapter outcome reaches consumers as one terminal finish; operational failure uses its error or aborted reason rather than throwing across the stream API. BlockAssembler is the single shared implementation that assembles chunks into blocks/messages.

Call configuration (call-config.ts)

LlmCallConfig is the provider, model, optional adapter-owned reasoning effort, and sampling scalars of one conversation's requests (provider, model, reasoningEffort, temperature, maxTokens, stop — each mapping 1:1 onto the same-named GenerateOptions field). It is per-conversation state recorded in the session log as part of the request header (see the dsh-session request/header events), never a silently-adjustable per-call knob: the agent/request waterfall proposes a replacement, prepareCall() validates it and materializes adapter defaults under the turn signal, and the loop logs the effective value plus markers for fields supplied by adapter defaults before using the prepared call's registration-bound stream. The next proposal omits marked defaults so a changed route resolves its own values; unmarked explicit fields persist. callConfigEquals(a, b) is the field-wise real-change detector; deepFreeze(value) is the ownership helper the loop applies to every built request before dispatch (llm/stream listeners and adapters read, never rewrite). markAgentLoopRequest() marks that exact object as created by the process-local agent loop, and isAgentLoopRequest() lets observers distinguish it from independently logged auxiliary calls that may also be frozen and session-associated. GenerateOptions.purpose classifies logged auxiliary compaction and session-title calls so adapters can apply purpose-specific transport policy without changing ordinary conversation requests.

App attribution (attribution.ts)

Every product adapter sends application identity on provider HTTP requests. attributionHeaders(identity?) builds the standard User-Agent, defaulting to public APP_IDENTITY; white-label deployments may replace but not suppress it. Adapters verify the wire header directly or through their library hook. See the attribution Agent Note.

API key validation (api-key.ts)

Every adapter that puts a credential in an HTTP header judges it the same way before use. normalizeApiKey(raw) trims surrounding whitespace, then accepts any non-empty printable-ASCII value (/^[\x21-\x7E]+$/, space excluded) or reports why not as an ApiKeyRejection ('empty' | 'illegalCharacters'), both carried in the ApiKeyCheck result. Absence is never judged: a caller decides whether a value was supplied before asking, since a profile naming no credential authenticates through the provider's own ambient discovery or OAuth.

Classes

• LlmAdapter — abstract base class for provider adapters. The only required method is stream().
• BlockAssembler — incrementally assembles raw chunks into complete content blocks and can create an identified, frozen assistant message from them. The agent loop feeds it raw chunks (logging them for replay) while reading the assembled blocks for history.
• HarnessError — base class for the harness error taxonomy: a stable code string (distinct from the human message) plus cause chaining. Lives here, in the leaf package every other imports, so a single base is shared without a new dependency edge. Per-package errors (LlmError, ToolArgsError, InvariantError, …) extend it. isHarnessError(value) narrows at process boundaries.
• LlmError — extends HarnessError; its stable code string (NO_ADAPTER, DUPLICATE_ADAPTER, and adapter codes like AUTH/RATE_LIMIT) matches its frozen serializable failure.code. The payload may also retain validated status, Retry-After, and branded provider request id facts; policy remains outside the error.
• errorChain(value) — renders a thrown value with its full cause chain and AggregateError members for diagnostic outputs (UI notices, logger lines, durable turn/end messages), so transport wrappers like undici's TypeError: fetch failed surface the underlying ECONNREFUSED/DNS/TLS detail instead of masking it. Rendering only — route on code, never by parsing the result.
• CONTEXT_WINDOW_EXCEEDED_CODE — the provider-neutral code both DeepSeek adapters use when a request exceeds the model context window, regardless of thrown-HTTP versus in-band finish delivery. isContextWindowExceededError(detail) is their shared conservative classifier for OpenAI-compatible provider detail.
• QUOTA_EXCEEDED_CODE — the non-transient provider-neutral code for exhausted account quota, balance, credits, budget, or usage limits. isQuotaExceededError(detail) keeps those failures distinct from request-rate limits.
• EMPTY_RESPONSE_CODE — the provider-neutral code both adapters use for a degenerate provider completion: a terminal stop that carried no content blocks at all. Classified as an error finish (not a successful empty message) because the attempt produced nothing durable; dsh-llm-retry retries it by default.
• INVALID_CREDENTIAL_CODE — the provider-neutral code for a credential that was supplied but cannot be used: malformed rather than absent, so the fix is to correct the stored value rather than supply one — the distinction from MISSING_CREDENTIAL. Deliberately excluded from the default retryable set, since a malformed credential fails identically on every attempt. assertUsableApiKey(raw, pkg, ref) throws LlmError with this code, the one shared diagnosis every adapter uses for an unusable stored credential.

Real adapters

Two adapters implement LlmAdapter on different internals: @deepseek-ai/dsh-llm-deepseek uses direct fetch with eventsource-parser SSE framing for the deepseek-official route, while @deepseek-ai/dsh-llm-pi-ai dynamically resolves configured provider/model pairs through @earendil-works/pi-ai. Both follow the StreamChunk conventions in types.ts: usage precedes finish and tool arguments remain raw strings. Adapter implementations may throw or emit a failure finish internally; LlmRuntime exposes both as a terminal failure finish. See the twin LLM adapters for the adapter rationale and the terminal-failure decision for the service boundary.

Model Experience

None, as the service adds no model-bound text, schema, or message; it only materializes and logs an adapter-configured reasoning effort.

KV Cache effect

Pass-through; the registry preserves the assembled request prefix, while the selected adapter and provider own cache reuse and routing boundaries.

Known Limitations and Deferred Work

• No retry execution, caching, or rate limiting ships in this service — provider registration stores retry policy, but llm/stream remains a single-attempt call wrapper. The agent loop separately offers proven model-request failures to agent/request-error, whose default preserves the original failure; @deepseek-ai/dsh-llm-retry is the optional executor loaded by the shared example spine.
• GenerateOptions sampling is temperature/maxTokens/stop only — no tool_choice, top_p, or penalty fields; the vocabulary grows when a producer lands (dropped inert knobs).
• Producer-gated variants stay out until produced — prefill, per-tool strict, block cache hints, and the agent message-source variant were pruned as producerless (Agent Note).
• BlockAssembler handles core block kinds only — a plugin-added block type whose stream is never closed by block-end makes blocks() throw.
• APP_IDENTITY.url names a repository that does not exist yet — the public home must be reachable before release.
• GenerateOptions.sessionId is a locally-declared brand — importing dsh-session's SessionId would cycle; a future ids-owning package would dissolve the workaround.