Multimodality Changes the Architecture, Not Just the Interface

GPT-4o is easy to describe as a better interface: ChatGPT can see, hear and speak with much lower latency. That description is correct, but I think it misses the more interesting change.

OpenAI says GPT-4o accepts combinations of text, audio, images and video, and can produce text, audio and images. More importantly, the company says it trained one model end-to-end across text, vision and audio, with all of those inputs and outputs processed by the same neural network. In voice, OpenAI reports response times as low as 232 milliseconds and an average of 320 milliseconds.

The latency is impressive. The architecture behind it matters more.

The real problem was the translation layer

Before GPT-4o, ChatGPT Voice Mode was built as a three-model pipeline. One model converted speech to text. GPT-3.5 or GPT-4 reasoned over the transcription. A third model converted the text response back into speech.

That is a sensible engineering pattern. Speech recognition, language reasoning and speech synthesis are mature enough to be composed as services. Whisper itself is a good example of a dedicated speech-recognition model: audio enters an encoder-decoder Transformer and text comes out.

But the pipeline has an information bottleneck: text.

Once audio becomes a transcript, the reasoning model receives the words but not necessarily the way those words were spoken. OpenAI points to tone, multiple speakers and background noise as information the previous Voice Mode could not expose directly to GPT-4. On the output side, a text response passed into speech synthesis does not naturally represent laughter, singing or conversational timing either.

This is not just a user-interface limitation. It is a systems limitation created by the contract between models.

Every time one modality is reduced into another before reasoning, some structure can disappear. A transcript can preserve language while losing timing and prosody. An image caption can preserve a rough description while losing spatial relationships. OCR can recover labels while ignoring the visual hierarchy that makes those labels meaningful.

A multimodal model changes where that conversion happens. Instead of forcing the application to translate everything into text first, the model can reason from the original modalities directly.

We do not know enough about GPT-4o's internal architecture to make stronger claims about how those representations are organised. The useful fact is simpler: text no longer has to be the mandatory intermediate format.

We were already moving in this direction

GPT-4o did not arrive from nowhere.

GPT-4V brought image and text inputs into GPT-4 in 2023. OpenAI described the intersection between modalities as creating capabilities that do not exist when each input type is treated separately.

Google has been moving in a similar direction with Gemini. Gemini 1.5 was introduced as a multimodal model capable of processing large amounts of text, images, audio and video in one context.

What feels different this week is that multimodality is moving from "attach another type of file" toward a real-time interaction model.

OpenAI's previous voice pipeline had average latencies of 2.8 seconds with GPT-3.5 and 5.4 seconds with GPT-4. GPT-4o's reported 320-millisecond average changes the interaction budget enough that interruption, pacing and immediate feedback start to become part of the product design rather than post-processing around a chatbot.

Google is pointing in the same direction. Project Astra, shown at I/O on May 14, is a prototype that continuously processes video and speech, combines them into a timeline of events and caches information for recall.

The interesting part is not that a model can answer a question about a camera frame. We already have visual question answering.

The change is that perception can become continuous state rather than a one-off attachment.

From requests to streams

Most LLM applications today still look like web APIs. Collect an input, send a request, wait for a completion, render the result.

Real-time multimodality pushes toward a different model: streams of state.

Audio arrives over time. Video changes every frame. A person can interrupt the system while it is responding. Something visible five seconds ago may matter to the current instruction even if it is no longer visible.

The system therefore has to decide what to retain, what to discard and when to respond.

That requires more than a prompt box connected to a model endpoint. It needs session state, buffering, temporal ordering, interruption handling and multimodal evaluation.

Latency becomes part of correctness too. An answer that arrives four seconds late may be semantically right and operationally useless.

This is why I see multimodality as an architectural shift rather than a feature checklist.

The boundary between model and application glue moves. Tasks that previously required separate perception, transcription and synthesis components can move closer to the reasoning system. In exchange, developers inherit new problems around continuous state and validation.

What this means for production workflows

For creative and technical work, the benefit is not simply talking to software instead of typing.

In Blender or a 3D production pipeline, a large amount of useful state is visual. A render can be technically valid while looking wrong. A material can have the correct node names while producing the wrong response under lighting. A viewport image can expose composition, topology or shading problems that are awkward to encode as text.

Vision-capable models already make some of this accessible. A lower-latency multimodal system suggests a more direct workflow: the model observes what the operator sees, combines that with instructions and technical context, then responds while the task is still happening.

The same applies to game development. Debugging often crosses several representations at once: code, logs, UI state, animation, frame timing and what is actually visible on screen. Today I normally have to translate those pieces into a form the model can consume. Better multimodality can reduce that translation work.

I would not remove specialist models from production stacks yet.

A dedicated transcription service can still be cheaper, easier to benchmark and easier to control. There is another practical limitation: as of May 16, the full GPT-4o experience shown in OpenAI's demos is not broadly available to developers. GPT-4o is available in the API for text and vision, while audio access has not yet been generally released.

The demos show where the architecture can go. They are not yet a production stack that developers can simply adopt.

My takeaway: the interface contract is moving inward

For years, application developers have built adapters around language models.

Convert speech to text. Convert screenshots into descriptions. Extract fields. Normalise everything into the format the model understands.

Native multimodality moves part of that adapter layer into the model itself.

If that continues, useful AI systems may stop treating text as their universal internal API. Text will remain valuable because it is compact, editable and easy to inspect. But it can become one representation among several rather than the required bridge between every perception and action.

This changes how I think about product architecture.

Instead of asking how to add voice or vision to a chatbot, the better question is what state the model needs to perceive directly and what information should remain structured outside it.

A renderer, game engine or production tool already contains rich structured state. We should not throw that structure away just because a multimodal model can look at the screen. At the same time, forcing every visual observation through manually designed metadata loses information too.

The practical architecture will probably be hybrid: structured data where software already knows the truth, multimodal perception where visual or acoustic context is expensive to formalise, and deterministic tools for actions that require control.

Where this may lead

The strongest signal I see this week is not that every model will become one giant monolith. It is that modality boundaries are becoming less visible to the reasoning layer.

Over the next few years, I expect more AI interfaces to consume ongoing state rather than isolated prompts. Screens, cameras, microphones and application events can become parallel inputs to the same reasoning process.

That would make assistants feel less like chat systems and more like software that remains aware of what is happening while work progresses.

The harder engineering problem will move toward state management, temporal grounding and evaluation. We will need to test whether a model noticed the right event, associated it with the right moment and carried the right information forward.

A text answer can be evaluated after generation. A real-time multimodal system has to be evaluated as a sequence.

GPT-4o makes the user interface look more natural.

The larger shift is underneath it: perception is moving closer to reasoning, and the application architecture will have to follow.

Sources

1. OpenAI, Hello GPT-4o, 13 May 2024.
2. OpenAI, Introducing GPT-4o and more tools to ChatGPT free users, 13 May 2024.
3. OpenAI, GPT-4V(ision) System Card, 25 September 2023.
4. OpenAI, Introducing Whisper, 21 September 2022.
5. Google DeepMind, Gemini breaks new ground with a faster model, longer context, AI agents and more, 14 May 2024.