Photograph a cracked pipe under your kitchen sink, send it to a chatbot, and ask what replacement part you need. A few years ago the reply was a flat refusal: these tools couldn’t see. Today the same assistant will describe the fitting, estimate the diameter from the pipe sitting next to it, and tell you which aisle of the hardware store stocks it. That jump, from text-only systems to ones that handle images, audio and video alongside language, is what multimodality is about. It has done more to make AI feel useful in ordinary work than most benchmark improvements.
What “multimodal” actually means
Information arrives in modes: written text, photographs, speech, video, sensor readings. A model trained on one of those is unimodal. A multimodal model takes in two or more and usually replies in a mix as well.
The engineering behind that differs more than the marketing suggests. Some systems bolt an image encoder onto a language model and train a small translation layer between them, which is cheap and works well. Others are trained on interleaved data from the start, so the network sees a photo, its caption, a spoken description of the same scene and the chart generated from it, all inside one training run. Google built Gemini along those lines, and the lab behind Gemini, AlphaFold and a Nobel Prize has been unusually open about mixing modalities during pretraining.
The machinery: patches, frames and tokens
Language models read tokens. To handle an image, the system slices it into small square patches, usually 14 or 16 pixels a side, and converts each patch into a vector that lands in roughly the same mathematical neighbourhood as a word embedding. A 1024×1024 photograph becomes a few thousand patch tokens. Audio works the same way on a different axis: the waveform is cut into frames of about 25 milliseconds, and each frame becomes a token. If you want the grounding on how these networks learn in the first place, this plain-English guide to how artificial intelligence works covers it without the maths.
Why context windows became the bottleneck
A single high-resolution photo can cost more tokens than a short essay. A 4K screenshot might burn a couple of thousand. A ten-minute video sampled at one frame per second multiplies that into the hundreds of thousands. So the limits that matter in practice aren’t just “can it see” but “how much can it see at once, and how quickly”.
The output side is a separate problem
Producing an image is handled by a diffusion model, not the language model. Producing speech means a vocoder or a streaming speech model with latency measured in a few hundred milliseconds. That’s why an assistant can describe a photo almost instantly but takes noticeably longer to generate one.
What multimodal AI is genuinely good at
Strip away the conference demos and a handful of tasks stand out because the accuracy is high enough to rely on:
- Messy documents. Invoices, lab reports, delivery notes, handwritten job sheets. Pulling line items into structured JSON works well enough that plenty of finance teams now run it in production.
- Screen understanding. Reading a dashboard, finding a button, filling a form. This is the foundation of the browser agents that started shipping through 2024 and 2025.
- Accessibility at scale. Generating alt text across image libraries with tens of thousands of photos, or captioning video for archives that never had the budget.
- Meeting capture. Speaker-aware transcription plus a summary that separates decisions from small talk.
- Visual inspection. Spotting a missing component, a misaligned label or a hairline crack on a production line, where the model is tuned on a few hundred labelled examples.
Visual search is the consumer-facing version of all this. Point a phone at a plant, a shoe or a landmark and the assistant identifies it and pulls up context. Google’s version of that pipeline, and the places it still misreads, is covered in this look at what Google’s AI search actually does well.
Where it still falls apart
The failure modes are specific, and worth knowing before you build on top of them.
- Counting. Ask for the number of people in a crowd photo and you’ll get a plausible number rather than a correct one.
- Small text. A serial number in a low-resolution image gets hallucinated with total confidence.
- Spatial relations. “Is the red handle to the left of the valve?” is surprisingly unreliable, because patch embeddings don’t preserve a clean coordinate system.
- Dense charts. Six-series line graphs with unlabelled axes get misread more often than not.
- Video over time. Models handle a frame or a short clip fine; reasoning about what changed between minute two and minute nine is still shaky.
- Prompt injection through images. Text baked into a picture can carry instructions. If the model can also browse or call tools, that’s a real security problem, not a curiosity.
The cost side nobody mentions in the demo
Images are expensive in tokens, and tokens are money. A pipeline processing 50,000 pages a month will cost meaningfully more if you send page renders instead of extracting text first. Rasterising PDFs is sometimes the right call and sometimes pure waste. Working out which is which is largely an exercise in measuring, so check the real pricing and limits of the OpenAI API before committing to an architecture; image inputs are priced per tile and the numbers add up faster than people expect.
For teams that need data residency or a private network, the same models are available through managed cloud endpoints, and Azure OpenAI Service is the usual route inside Microsoft shops. Latency matters too: an image-heavy request can take three to five times longer than a text one, which is fine for batch work and painful for anything conversational.
Getting better answers out of a multimodal model
Most disappointing results come from how the question was framed rather than from the model’s ceiling.
- Crop before you upload. A tight photo of the label beats a wide shot of the whole machine every time.
- Ask one question per image. “What’s wrong with this circuit board?” gets a rambling answer. “Is there any discolouration near the capacitors on the right edge?” gets a useful one.
- Force structure. Ask for a JSON object with named fields. It reduces invention and makes the output testable.
- Do two passes. Extract first, then verify against the original. Cheap, and it catches a lot of OCR slips.
- Supply what it can’t see. Model number, expected range, what the part should look like when it’s fine.
The direction of travel
Every major lab now ships models that accept text, image and audio on one endpoint, and the 2025 releases pushed hard on speed rather than raw capability. Check the current lineup of OpenAI models and what each one is built for if you’re picking one for a project; the gap between a fast small model and a reasoning-heavy one matters far more for image work than for chat.
The next step is continuous video and direct screen control: models that watch a livestream of a factory floor, or click through software on your behalf, without a human turning every frame into a sentence. Early versions exist, and they are slow, expensive and occasionally confidently wrong. A useful mental model is that a multimodal system today is a very well-read colleague with mediocre eyesight and no memory of what happened five minutes ago. Design around that and it earns its keep.