imageBlockFor

imageBlockFor(provider, mimeType, data): { data: string; mime_type: string; source_type: "base64"; type: "image"; } | { image_url: string; source?: undefined; type: "image_url"; } | { image_url: { url: string; }; source?: undefined; type: "image_url"; } | { image_url?: undefined; source: { data: string; media_type: string; type: "base64"; }; type: "image"; }

Defined in: _worktrees/docs-release/gaunt-sloth/packages/agent/src/middleware/frontendImageInjectionMiddleware.ts:121

A vision content block the target provider’s @langchain converter actually decodes. Verified against the installed converters (RC-21):

ollama → { type:'image_url', image_url:'<data-URL string>' }. ChatOllama’s convertToOllamaMessages only handles image_url blocks (extractBase64FromDataUrl); the LangChain standard source_type block throws “Unsupported content type: image”.
image_url-consuming converters (openai, openrouter, deepseek, xai, groq, huggingface, xai-responses) → { type:'image_url', image_url:{ url:'<data-URL>' } }. The grouping is by what the converter CONSUMES, not by client family: the first six are served by an OpenAI-compatible Chat Completions API whichever LangChain class fronts them, while xai-responses reaches the same block by a different route entirely (below). This native OpenAI shape is correct on BOTH the Completions API AND the Responses API (GS2-74 flips reasoning-capable openai models to Responses). A raw source_type standard block serialises to an invalid image part on the Responses path, so we emit the provider-native shape rather than lean on @langchain/core’s (deprecated, internal) auto-conversion. huggingface (CFG-45) belongs here by MEASUREMENT, not by “it constructs a ChatOpenAI”: a fetch-capture probe of the installed @langchain/openai (no network — the client’s fetch was stubbed), built exactly as providers/huggingface.ts builds it, showed the request going to https://router.huggingface.co/v1/chat/completions — the Completions path, never Responses, since nothing on the HF path sets useResponsesApi and the library’s own model-name flip matches only OpenAI-specific ids — and openai@7.6.0’s ChatCompletionContentPart union accepts exactly one image part there: image_url:{url}. There is no input_image part on that endpoint, so the OpenAI Responses shape does not apply. The same probe showed the standard block ALSO arriving as image_url:{url} on the Completions path — but only because @langchain/core’s convertToProviderContentBlock (now @deprecated: “Don’t use data content blocks”) rewrites it, which is precisely the auto-conversion the through-line below refuses to depend on, and which yields a bare, invalid image_url part the moment the same block reaches a Responses endpoint. xai-responses (CFG-45) is the same emitted block for an UNRELATED reason, and it is the case that shows why this switch exists. It is ChatXAIResponses._llmType(), reachable only via resolveVisionProvider’s _llmType() fallback (gth’s own xai provider builds ChatXAI), and that class extends BaseChatModel directly with its OWN converter — no OpenAI client and no @langchain/core data-block conversion anywhere on the path. Its human-message branch (@langchain/xai@1.4.10 dist/converters/responses.js) recognises exactly text and image_url and rewrites every other part to { type:'input_text', text:'' }. A globalThis.fetch-capture probe (no network — that class calls the global fetch directly and accepts no injectable client) measured four blocks against https://api.x.ai/v1/responses: the standard base64 block arrived as {"type":"input_text","text":""} — the image SILENTLY DESTROYED in-process, never rejected — while image_url:{url} arrived as the vendor’s declared {"type":"input_image","image_url":"data:…","detail":"auto"}. Emitting the wire shape { type:'input_image', … } from here was measured too and is destroyed identically: the translation is the converter’s to make, so pre-empting it defeats it. Not establishable offline, and stated rather than assumed: whether api.x.ai accepts a data: URL in input_image.image_url, whose vendor type documents it as a public URL. The ruling does not rest on that — the standard block is provably destroyed before any request is built, while this block provably survives into the vendor’s own declared image item.
anthropic → the provider-native block { type:'image', source:{ type:'base64', media_type, data } }. The LangChain standard block is NOT usable here (RC-32): in @langchain/anthropic’s _formatContentBlocks, the isDataContentBlock branch yields its conversion and then FALLS THROUGH — no continue — into the chain below, where type === 'image' matches the very same block and yields a SECOND one whose media_type is read from camelCase mimeType (a key the snake_case standard block never has) and so defaults to the literal image/jpeg. Every frame is therefore sent twice, and a non-JPEG capture 400s outright on the mislabelled copy. The native block is recognised earlier by _isAnthropicImageBlockParam and passed through untouched, exactly once.
Gemini (google-genai, vertexai, and the google label both of them report from _llmType()) → the LangChain standard base64 data content block { type:'image', source_type:'base64', mime_type, data }, which those native converters decode directly.
anything else → the same standard block, as a last-resort fallback rather than as a Gemini alias (see the default arm below).

The through-line: emit what the target provider’s converter consumes natively rather than lean on a generic auto-conversion — the same lesson as GS2-75 on the OpenAI Responses path.

Exported so each provider branch can be unit-tested directly; its only effect beyond the returned block is one debug-log line on the fallback arm.

Parameters

provider

string

mimeType

string

data

string

Returns

{ data: string; mime_type: string; source_type: "base64"; type: "image"; } | { image_url: string; source?: undefined; type: "image_url"; } | { image_url: { url: string; }; source?: undefined; type: "image_url"; } | { image_url?: undefined; source: { data: string; media_type: string; type: "base64"; }; type: "image"; }