Skip to content
Gaunt Sloth Documentation

imageBlockFor

imageBlockFor(provider, mimeType, data): { data?: undefined; image_url: string; mime_type?: undefined; source?: undefined; source_type?: undefined; type: "image_url"; } | { data?: undefined; image_url: { url: string; }; mime_type?: undefined; source?: undefined; source_type?: undefined; type: "image_url"; } | { data?: undefined; image_url?: undefined; mime_type?: undefined; source: { data: string; media_type: string; type: "base64"; }; source_type?: undefined; type: "image"; } | { data: string; image_url?: undefined; mime_type: string; source?: undefined; source_type: "base64"; type: "image"; }

Defined in: _worktrees/docs-release/gaunt-sloth/packages/agent/src/middleware/frontendImageInjectionMiddleware.ts:82

A vision content block the target provider’s @langchain converter actually decodes. Verified against the installed converters (RC-21):

  • ollama → { type:'image_url', image_url:'<data-URL string>' }. ChatOllama’s convertToOllamaMessages only handles image_url blocks (extractBase64FromDataUrl); the LangChain standard source_type block throws “Unsupported content type: image”.
  • OpenAI-compatible (openai, openrouter, deepseek, xai, groq — served by an OpenAI-compatible Chat Completions API, whichever LangChain class fronts it) → { type:'image_url', image_url:{ url:'<data-URL>' } }. This native OpenAI shape is correct on BOTH the Completions API AND the Responses API (GS2-74 flips reasoning-capable openai models to Responses). A raw source_type standard block serialises to an invalid image part on the Responses path, so we emit the provider-native shape rather than lean on @langchain/core’s (deprecated, internal) auto-conversion.
  • anthropic → the provider-native block { type:'image', source:{ type:'base64', media_type, data } }. The LangChain standard block is NOT usable here (RC-32): in @langchain/anthropic’s _formatContentBlocks, the isDataContentBlock branch yields its conversion and then FALLS THROUGH — no continue — into the chain below, where type === 'image' matches the very same block and yields a SECOND one whose media_type is read from camelCase mimeType (a key the snake_case standard block never has) and so defaults to the literal image/jpeg. Every frame is therefore sent twice, and a non-JPEG capture 400s outright on the mislabelled copy. The native block is recognised earlier by _isAnthropicImageBlockParam and passed through untouched, exactly once.
  • google-genai / vertexai (and any unknown/default) → the LangChain standard base64 data content block { type:'image', source_type:'base64', mime_type, data }, which those native converters decode directly and which is the most broadly decodable fallback.

The through-line: emit what the target provider’s converter consumes natively rather than lean on a generic auto-conversion — the same lesson as GS2-75 on the OpenAI Responses path.

Pure and exported so each provider branch can be unit-tested directly.

Parameters

Section titled “Parameters”

provider

Section titled “provider”

string

mimeType

Section titled “mimeType”

string

data

Section titled “data”

string

Returns

Section titled “Returns”

{ data?: undefined; image_url: string; mime_type?: undefined; source?: undefined; source_type?: undefined; type: "image_url"; } | { data?: undefined; image_url: { url: string; }; mime_type?: undefined; source?: undefined; source_type?: undefined; type: "image_url"; } | { data?: undefined; image_url?: undefined; mime_type?: undefined; source: { data: string; media_type: string; type: "base64"; }; source_type?: undefined; type: "image"; } | { data: string; image_url?: undefined; mime_type: string; source?: undefined; source_type: "base64"; type: "image"; }