Feature Request: Native Multimodal Image Inputs for Agent Sessions

[HappyLight]

Summary

I would like the Anna Agent Session API to support native image inputs, allowing the multimodal model used by the current session to inspect images directly instead of requiring every image task to go through the two-step upload_local_file and analyze_image tool workflow.

When I asked an Agent Session to analyze a local image, I observed the following execution flow:

session.run (the text prompt contains a local image path)
  → Agent calls upload_local_file
  → Agent receives a temporary image URL
  → Agent calls analyze_image
  → analyze_image returns a textual description
  → Agent generates the final response from that description

This workflow allows a session to process images indirectly, but it is not equivalent to having the primary model receive the image and reason about it together with the current session context. It has significant limitations in latency, multi-image processing, and information fidelity.

Test Results

The test used a 2560 × 1440 PNG image with a file size of approximately 2.82 MB. The session.run input contained only the local image path and an instruction to analyze it. The session then called upload_local_file and analyze_image on its own.

Based on the tool start and end timestamps in the session stream:

Stage	Duration
upload_local_file	approximately 18.86 seconds
analyze_image	approximately 10.02 seconds
Combined tool execution time	approximately 28.88 seconds
Entire session.run	approximately 77.95 seconds

This is a single test with one image, so these results should not be treated as fixed performance numbers across all networks and runtime environments. However, they demonstrate that the current image-analysis workflow adds overhead beyond model inference, including tool selection, file upload, waiting for tool results, and a second pass in which the primary Agent turns the analysis into a final response.

The impact is likely to grow when a task needs to inspect multiple images. Use cases such as PowerPoint visual review, design comparison, and analysis of consecutive pages commonly require several images in one request. Running the full “upload → analyze → return text” workflow separately for each image can introduce substantial cumulative latency.

Information Loss in the Current Workflow

After inspecting an image, analyze_image returns a textual description. The primary Agent receives this description rather than the original image, creating a lossy information bottleneck.

If analyze_image omits a detail from its textual response, the primary Agent cannot reason directly about that detail later. Examples include:

• precise element positions, spacing, dimensions, and alignment;
• small text, partial obstruction, clipping, and overflow;
• local data points in charts and their visual relationships;
• subtle differences between multiple images;
• visual issues that can only be evaluated in combination with existing session context.

In addition, if the image-analysis result contains an OCR or visual-understanding error, the primary Agent usually cannot verify it against the original image. It can only continue reasoning from the text it received. This workflow may be sufficient for high-level image descriptions, but information loss directly affects reliability in tasks such as PowerPoint slide review, UI inspection, chart interpretation, design comparison, and iterative visual refinement.

Requested Capability

The current test app calls the API as follows:

await anna.agent.session.run({
  app_session_uuid,
  content,
});

I suggest extending this existing API with an optional image URL input while preserving the current app_session_uuid and plain-text content behavior. For example:

await anna.agent.session.run({
  app_session_uuid: "aps_xxx",
  content: "Please check these PowerPoint slides for text overflow, overlapping elements, or obvious layout issues.",
  image_urls: [
    "https://example.com/slide-1.png",
    "https://example.com/slide-2.png",
  ],
});

The image_urls field is only an illustration of a possible API shape; the Anna team can determine the final field name and image-reference structure. The important requirement is to keep existing text-only calls fully compatible while allowing developers to provide one or more images as native inputs for the current session turn.

If Anna requires local images to be stored through an upload API first, the resulting image reference should still be usable as a native Agent Session input. The current session’s multimodal model should be able to inspect that image directly instead of requiring an additional analyze_image call that converts it to text.

More specifically, I would like the API to provide the following behavior:

1. Preserve the existing plain-text content field in session.run while adding optional image URL or image-reference inputs.
2. Support multiple images in a single request so the model can compare and reason across them jointly.
3. When the selected model supports vision, send the images and session context to the same model.
4. When the selected model does not support vision, return a clear capability error or optionally fall back to analyze_image for compatibility.
5. Keep upload_local_file and analyze_image available for explicit file uploads, standalone image analysis, and models without native multimodal support, rather than making them the only path for all image-understanding tasks.

Expected Benefits

Native multimodal inputs for Agent Sessions would:

• reduce the additional latency caused by intermediate tool selection and execution;
• avoid compressing complete visual information into a one-time textual description before the primary Agent can reason about it;
• improve the reliability of multi-image comparison, spatial reasoning, and detailed visual review;
• allow images, session history, user instructions, and other context to be understood together in the same inference;
• align the Agent Session API with the input patterns offered by current multimodal model APIs, reducing integration and prompt-orchestration work for application developers.

I hope the Anna team will consider adding native multimodal image inputs to Agent Sessions while retaining the existing tool-based image-analysis workflow as a complementary compatibility option.

[hunter]

Hi @HappyLight!

Thank you for this beautifully argued feature request — the latency breakdown and the “lossy information bottleneck” analysis captured exactly why tool-mediated image analysis wasn’t enough for serious visual work. We’re delighted to say: it’s shipped!

TL;DR: Native multimodal image inputs for Agent Sessions are live as of 1.1.0-beta.99.

What’s new

session.run now accepts an optional attachments array alongside content — existing text-only calls are 100% unchanged:

await session.run({
  content: "Please check these slides for text overflow, overlapping elements, or layout issues.",
  attachments: [
    { type: "image", url: "https://example.com/slide-1.png" },
    { type: "image", url: "https://example.com/slide-2.png" },
    // or inline: { type: "image", data: "<base64>", mime_type: "image/png" }
  ],
});

Checking off your wishlist:

1. Optional & backward-compatible — plain-text content behavior is untouched.
2. Multiple images per request (up to 6) — the model compares and reasons across them jointly, together with full session context.
3. Same model, same inference — when the session’s model supports vision, images go directly to it. No analyze_image hop, no textual compression.
4. Clear capability error — if the resolved model isn’t vision-capable, the run fails fast with errorCode: "APP_MODEL_NOT_VISION_CAPABLE" (no silent degradation). Pair with modelPreferences to steer to a vision model.
5. Tools stay available — upload_local_file and analyze_image remain for explicit uploads and non-vision models.

Bonus: local-file workflows got smarter too

Your original test case — “here’s a local image path, please analyze it” — is also natively multimodal now. When the agent uploads a local image via upload_local_file mid-run, a vision-capable model sees the actual image in the same turn instead of a textual description. In our retest of your exact scenario, the run went from ~78s to ~18s, with zero analyze_image calls and the model reasoning over the real pixels.

Where to look

• Host API reference → agent.session.run now documents attachments (formats, limits, and the vision-capability error).
• The anna-app-llm-demo example (v0.3.0) includes a working attachments section for both transports — great as a starting template.

Thank you for pushing Agent Sessions toward first-class visual reasoning — PPT review, UI inspection, and design-comparison workflows all just got a lot more reliable. Can’t wait to see what you build with it!