📦 @goodandready/dsh-vision-bridge

⚡ Overview & The Problem

When interacting with text-only LLM models (e.g. deepseek-v3, Qwen text-only variants) in DeepSeek Harness, users cannot natively attach and send images:

1. In DSH 0.1.2-alpha.2+, the core session controller performs a strict server-side modality check (ctx.llm.resolveModelInfo). If the active conversation model lacks image in inputModalities, the prompt is immediately rejected with a session/attachment-invalid error ("Model does not support image input").
2. Standard text-only adapters throw errors when encountering raw multimodal image blocks in their message payload.

How dsh-vision-bridge Solves This

dsh-vision-bridge acts as an intelligent intermediary inside the Cordis runtime:

• Server-Side Modality Bridge (v0.5.3+): Decorates ctx.llm.resolveModelInfo and ctx.llm.listModels so the session controller accepts image attachments on all models when bridging is active.
• Automatic Image Rewrite (agent/pre-step & llm/stream): Automatically intercepts image blocks, routes them to a configured vision model (e.g. Gemini, Claude, Qwen-VL, or local Ollama), receives a descriptive synthesis, and rewrites the image block into text context [The user attached an image. Description: ...] before handing it to the text-only chat model.
• Native Passthrough: Automatically detects models that natively support vision and allows images to pass directly without unnecessary rewriting.
• Rich Visual Tool Suite: Exposes 26 specialized tools for on-demand OCR, visual question answering, bounding box grounding, UI layout breakdown, and pixel differencing.

🏗️ Architecture

graph LR
    User["User attaches Image in Web UI"] --> Gateway["DSH Session Controller"]
    Gateway --> BridgeCheck{"Modality Bridge (v0.5.3)"}
    BridgeCheck -->|"Augments inputModalities"| SessionAllowed["Prompt Accepted"]
    SessionAllowed --> Hook["agent/pre-step Hook"]
    
    Hook --> CheckNative{"Does chat model support vision natively?"}
    CheckNative -->|"Yes (Native Passthrough)"| NativeLLM["Send Raw Image to Chat LLM"]
    CheckNative -->|"No (Text-Only)"| VisionRouter["Vision Bridge Channels"]
    
    VisionRouter --> VisionModel["Dedicated Vision Model\n(DSH / OpenAI / Ollama / Webhook)"]
    VisionModel --> Description["Generated Text Description + OCR"]
    Description --> Rewrite["Substitute Image with Text Marker"]
    Rewrite --> ChatModel["Send Enriched Text to Chat LLM"]
    ChatModel --> Answer["Assistant Response in Chat"]

✨ Key Features

1. Processing Modes

• hybrid (default): Automatically describes attached images in chat turns while keeping all 26 explicit vision tools available for follow-up reasoning.
• llm: Pure auto-rewrite mode — images are transparently converted to text context; tools remain callable.
• tools: Auto-rewrite disabled — the chat model is expected to explicitly invoke describe_image or OCR tools when required.

2. Multi-Channel Endpoint Routing & Fallback

Chain multiple vision backends with automatic failover, parallel racing, and circuit breaker:

• dsh-catalog: Auto-detect or select any vision-capable model already registered in DSH.
• openai-compatible: Standard OpenAI-compatible vision endpoints (vLLM, SGLang, OpenRouter, etc.).
• ollama: Auto-discovery and local inference through Ollama vision models (e.g. minicpm-v, llama3.2-vision).
• webhook / custom: External HTTP or JSON-RPC vision endpoints.

3. High-Performance LRU Description Cache

Caches vision responses by hash(bytes + prompt + model + mode) to eliminate redundant vision API calls and save token quota on repeated questions about the same image.

4. Comprehensive Visual Tool Inventory (26 Tools)

📦 Installation

dsh plugin --profile web add @goodandready/dsh-vision-bridge

After installation, restart the DSH Web UI. The configuration card is available under Settings → Plugins → vision-bridge.

⚙️ Configuration (settings.yaml)

dsh-vision-bridge:
  # Operation mode: 'hybrid' | 'llm' | 'tools'
  mode: hybrid
  
  # Auto-detect vision model or specify provider/model explicitly
  visionProvider: ""
  visionModel: ""
  
  # Allow vision models to receive images natively without rewriting ('prefer' | 'never' | 'always')
  nativePassthrough: prefer
  
  # Enable LRU description cache
  cacheEnabled: true
  cacheMaxEntries: 200
  
  # Request timeout in milliseconds
  timeoutMs: 120000
  
  # Multi-channel routing configuration
  channels: []
  channelStrategy: fallback # 'fallback' | 'race'

Parameter Reference

📄 License

MIT © GooDAnDReaDY

15. 🛡️ Enterprise Security & Settings Governance (v0.5.27)

• Masked Secrets: GET /channels automatically masks sensitive provider API keys (sk-p...7890 or ********) and reports hasApiKey: true to prevent secret leakage in browser DevTools/XHR. POST /channels preserves existing keys when a mask is submitted.
• CSRF & Origin Guard: All mutating endpoints (/config, /channels, /upload-pdf, /test) validate sec-fetch-site !== 'cross-site' via isTrustedSettingsRequest, rejecting cross-origin attacks with 403 Forbidden.
• Native settingsScope Integration: Settings UI binds directly to ctx.settingsScope (namespace: 'dsh-vision-bridge'), supporting reactive snapshot listeners and kernel state synchronization.