Skip to content
Gaunt Sloth Documentation

imageBlockFor

imageBlockFor(provider, mimeType, data): { data?: undefined; image_url: string; mime_type?: undefined; source_type?: undefined; type: "image_url"; } | { data?: undefined; image_url: { url: string; }; mime_type?: undefined; source_type?: undefined; type: "image_url"; } | { data: string; image_url?: undefined; mime_type: string; source_type: "base64"; type: "image"; }

Defined in: _worktrees/docs-release/gaunt-sloth/packages/agent/src/middleware/frontendImageInjectionMiddleware.ts:69

A vision content block the target provider’s @langchain converter actually decodes. Verified against the installed converters (RC-21):

  • ollama → { type:'image_url', image_url:'<data-URL string>' }. ChatOllama’s convertToOllamaMessages only handles image_url blocks (extractBase64FromDataUrl); the LangChain standard source_type block throws “Unsupported content type: image”.
  • OpenAI-compatible (openai, openrouter, deepseek, xai, groq — all extend ChatOpenAI / report _llmType()==='openai') → { type:'image_url', image_url:{ url:'<data-URL>' } }. This native OpenAI shape is correct on BOTH the Completions API AND the Responses API (GS2-74 flips reasoning-capable openai models to Responses). A raw source_type standard block serialises to an invalid image part on the Responses path, so we emit the provider-native shape rather than lean on @langchain/core’s (deprecated, internal) auto-conversion.
  • anthropic / google-genai / vertexai (and any unknown/default) → the LangChain standard base64 data content block { type:'image', source_type:'base64', mime_type, data }, which those native converters decode directly and which is the most broadly decodable fallback.

Pure and exported so each provider branch can be unit-tested directly.

Parameters

Section titled “Parameters”

provider

Section titled “provider”

string

mimeType

Section titled “mimeType”

string

data

Section titled “data”

string

Returns

Section titled “Returns”

{ data?: undefined; image_url: string; mime_type?: undefined; source_type?: undefined; type: "image_url"; } | { data?: undefined; image_url: { url: string; }; mime_type?: undefined; source_type?: undefined; type: "image_url"; } | { data: string; image_url?: undefined; mime_type: string; source_type: "base64"; type: "image"; }