Writing
The Model Is Becoming a Runtime
Function calling, code execution and structured output turn inference into a loop. The model stops being a text generator and starts being a control layer.
Notes
For most of the last two years, the default mental model for an LLM application has been simple: send text in, get text back.
OpenAI added function calling to GPT-4 and GPT-3.5 in June 2023. By November, GPT-4 Turbo could return valid JSON, call multiple functions in one turn, and work inside the Assistants API with Code Interpreter and Retrieval. Anthropic made tool use generally available across the Claude 3 family at the end of May. Google added parallel function calling to Gemini 1.5 in May as well.
Models are moving from producing answers to deciding what operation should happen next.
Strictly speaking, the model is not a runtime. The application around it still executes code, owns state and enforces permissions. But the model is starting to occupy a role that looks increasingly like the control layer of one.
Text generation is becoming an intermediate step
A normal language-model response is terminal. The model writes something, the application displays it, and the turn is over.
Function calling changes that contract.
Instead of asking the model to describe an action in prose, a developer gives it a set of functions with names, descriptions and parameters. The model can select one and produce arguments for it. The application executes the function, returns the result, and the model continues from the new state.
That creates a loop:
request → model decision → tool call → execution → result → model decision
The useful output is no longer necessarily the text generated by the model. Sometimes the useful output is the side effect produced by the tool.
If I ask a system to "rename these exported assets according to our convention," I need validated operations against the filesystem or asset database, not a paragraph explaining them.
OpenAI's original function-calling release made this shift explicit. GPT-4 and GPT-3.5 could return JSON arguments matching developer-defined functions, allowing natural language to be converted into API calls or database queries. OpenAI did not execute those functions for the developer; the application remained responsible for that boundary.
Structured output is an interface, not formatting
There is a temptation to think of JSON output as nicer formatting for model responses. I think it is closer to a type boundary between probabilistic reasoning and deterministic software.
OpenAI's JSON mode, introduced with GPT-4 Turbo, guarantees syntactically valid JSON. That makes parsing safer, but valid JSON is not the same as a correct schema. A model can still return the wrong field, the wrong value or a semantically bad decision.
Function schemas go a step further because they describe the operations the model is allowed to request. Anthropic's tool-use implementation follows the same general pattern: developers define tools, Claude chooses one when appropriate, and structured tool-use blocks connect the model to application code. Anthropic lists use cases such as extracting structured fields, converting natural-language requests into API calls, querying databases and automating software tasks.
A useful architecture follows from this: keep language at the human boundary and use typed structures at the software boundary. Once a decision crosses into production software, validate it.
Code execution changes the model's relationship with computation
Tool calls give a model access to functions that developers have already implemented. Code execution extends the idea.
OpenAI's Assistants API includes Code Interpreter, which can write and run Python in a sandbox, inspect the result, revise the code and run it again. In this setup, code is not only an artifact generated for a person to copy. It becomes a temporary instrument the model can use while solving the task.
That is a different capability from "write me a Python script."
A text-only model has to simulate the result of computation in its tokens. A model connected to an execution environment can externalise part of the task: calculate, inspect, plot, parse a file, observe an error, then continue.
Claude 3.5 Sonnet, released three days ago, makes the direction even clearer. Anthropic reports an internal agentic coding evaluation where the model is given a codebase, a natural-language change request and relevant tools. The task is not to generate a code snippet in isolation. It must modify an existing repository and solve the requested problem. Anthropic says Claude 3.5 Sonnet solved 64% of those tasks, compared with 38% for Claude 3 Opus.
The number matters less to me than the evaluation shape: the model is being tested inside an environment with state and actions, not as an isolated chat completion.
The model is becoming a control loop
This is where the runtime analogy becomes useful.
A traditional program has explicit control flow. The developer decides which function runs next. Conditions and state transitions are encoded ahead of time.
An LLM-based system can move some of that control flow into inference. The developer exposes a bounded set of operations. The model looks at the current state, chooses the next operation, receives the result and decides again.
The program is no longer a fixed sequence of calls. Part of the execution path is generated at runtime from the task and the observed state.
Ordinary software architecture does not disappear. It becomes more important.
The model should not own permissions. It should not be trusted to validate its own output. It should not become the source of truth for project state. Those jobs belong to deterministic systems around it.
What changes is the layer between user intent and those systems.
Why this matters in production
I see a direct application in 3D, game-development and automation pipelines because these environments already contain many small tools with strict inputs.
A Blender pipeline might expose operations such as find_objects, validate_materials, set_render_settings, export_asset or run_scene_check. A game pipeline could expose build commands, localisation checks, asset validation, test execution and project metadata queries.
I would not give a model unrestricted access to Blender's Python API or a project filesystem and hope for the best. A better design is to expose narrow operations with explicit schemas, permissions and observable results.
The model handles intent:
"Prepare the selected product models for web export, keep the current materials, flag anything above the polygon budget, and do not overwrite existing files."
The pipeline handles truth:
which objects are selected, what the polygon budget is, whether the destination exists, whether overwriting is allowed, and whether the export passed validation.
Language models can resolve instructions that are awkward to encode as rigid command syntax; production software can enforce the constraints.
For founders, the practical implication is that an AI feature should not be designed only as a chat surface. The more capable pattern is a model surrounded by typed tools, state, validation and logs.
The quality of those interfaces may matter as much as the model choice.
The next few years
I expect tool interfaces to become a normal part of application architecture for LLMs.
Models will probably get better at selecting tools, chaining them and recovering when an operation fails. The bigger change may happen outside the model: developers will expose more software capabilities as machine-readable operations rather than UI-only actions.
That could create a new abstraction layer over existing applications. Instead of building a separate natural-language workflow for every feature, software can expose a controlled set of capabilities and let the model compose them according to intent.
Long chains still amplify errors. A wrong early decision changes the state seen by later steps. Production systems will need checkpoints, validation and clear completion criteria rather than assuming that a stronger model solves orchestration automatically.
For this reason, I do not think the future is "the model replaces the application."
I think the model becomes part of the application's execution architecture.
The transition from text generation to tool use is the first clear sign. Once a model can choose operations, run code, inspect results and continue from changed state, calling it a text generator starts to describe only the smallest part of what the system is doing.
Sources
- OpenAI, Function calling and other API updates, 13 June 2023. OpenAI source
- OpenAI, New models and developer products announced at DevDay, 6 November 2023. OpenAI DevDay source
- Anthropic, Claude can now use tools, 30 May 2024. Anthropic source
- Google, Gemini 1.5 Pro updates, 1.5 Flash debut and 2 new Gemma models, 14 May 2024. Google source
- Anthropic, Claude 3.5 Sonnet, 21 June 2024. Anthropic Claude 3.5 source
More
Other write-ups
Approval Is Not Publication
Thirty seven items in the queue, one approved, nothing published. The last arrow in the diagram is the only one that pays.
Neural Rendering Is Crossing From Reconstruction Into Synthesis
Reconstruction filled in what sparse sampling missed. DLSS 5 generates appearance the renderer never computed.
The Renderer Is Becoming a Training Data Engine
The renderer used to sit at the end of the pipeline. Physical AI gives the same scene a second job: teaching a model.
Local AI Is Becoming a Compute Fabric
Local AI meant one model on one machine. Routing inference across the devices already on a network changes the unit.
Agent Infrastructure Is Becoming a Product Category
Every team used to build the loop, the store, the sandbox. That layer is being sold rather than written.
Choosing a local model with a stopwatch, not a benchmark
Three models, five hard tasks, code actually executed. The official build was 2.4 times faster and more accurate than a community repack of the same model.
We measured real time lighting against baked light, and baked won
A 3D product simulation that had to look like an offline render. The real time version ran at 60 frames per second and looked like clay. Here is the measurement and the architecture that replaced it.
What actually broke in an agency run by agents
Three failures from a delivery stack that runs on AI. None of them were the model's fault, and all three reported success while producing nothing.
A Task Without A Check Command Is Not Automated
If a task has no command that can fail, the pipeline advances on the appearance of work.
A Gate The Model Writes Is A Gate The Model Loosens
Three quality gates returned green while the work behind them was wrong, each for a different reason.
The Scoring Model Was Wrong And It Put The Worst Lead First
A weighted sum let one axis substitute for the other, so a company with money and no problem ranked in the top twenty.
Building Software Got Easy. Getting Value Out Of It Did Not
Aristo took weeks to build. Everything after the build is still in progress, and that gap is the whole story.
Capability Is Becoming an Operational Risk Surface
Safety questions used to be about the text. Once a model can act, the capability itself becomes something to operate.
The Scene Graph Is Becoming an API for AI
A scene graph exists for artists and software. Agents are becoming another consumer, and they need structure rather than pixels.
Animation Is Moving From Clips to Motion Priors
Authored keyframes and blended clips are giving way to asking which constraints define acceptable motion.
Materials Are Becoming Learned Programs
A material is texture maps, parameters and shader code. It is starting to become a small learned program that answers a rendering question.
Procedural Systems Are Expanding Beyond Geometry
Geometry Nodes started with a narrow name. Blender 5.2 puts physics, sound and object data through the same graph.
The Unit of AI Work Is Becoming the Task, Not the Turn
Chat taught us to think one turn at a time. Long-running agents make the task the thing that is scheduled, resumed and reviewed.
Game Engines Are Becoming Operating Systems for Worlds
Engines have been judged on what they render and simulate. The Unreal 6 roadmap points at operating a world rather than drawing one.
The Model Is Becoming a Replaceable Backend
Choosing a provider used to mean choosing an architecture. A stable interface makes replacement possible and evaluation makes it safe.
The Harness Is Part of the Capability
The same model behaves differently depending on context policy, tool design and execution feedback. That surrounding software is not neutral.
Physics Engines Are Becoming Trainable Components
A simulator predicts what happens next. A differentiable one can answer which parameter should change to stop the failure.
The Agent Needs an Environment, Not Just Tools
A search function and a database query were enough for short loops. Longer work needs a place to stand.
Coding Agents Are Becoming General-Purpose Computer Workers
Repositories were a friendly environment: text in, terminal actions, checkable results. That was a starting point, not a boundary.
Open Standards Outlive Model Generations
A year after the MCP bet, the argument can be checked against what happened rather than what was hoped.
Colour Management Is a Pipeline Contract
Blender 5.0 reads as better display options. Giving a file an explicit working colour space is an architectural change.
The Best Model May Be a Router, Not a Model
GPT-5 moves model selection inside the system. The interesting unit stops being which model and becomes which compute policy.
World Models Are Not Game Engines Yet
Genie 3 generates a navigable 720p world at 24 fps. Production work needs state you can inspect when something goes wrong.
Memory Is Becoming a System Capability
A follow-up to the long-context argument. Storing, selecting and expiring facts is turning into a named part of the product.
Coding Agents Change the Unit of Software Work
AI coding tools have been judged where code appears on screen. The boundary moves when the agent owns a task instead of a snippet.
Agents Need Protocols Between Each Other, Not Just Tools
Tool calling solves the inside of the loop. It says nothing about one agent reaching another built by a different team on another platform.
Agents Need a Runtime, Not a Prompt Loop
An agent is no longer well described as a prompt inside a while loop. The useful abstraction owns execution around the model.
Reasoning Is Not the Only Path to Better Models
Longer thinking improves maths and code. A model that solves a logic puzzle and misreads ordinary intent is not the better production model.
Rendering Is Becoming a Reconstruction Stack
Sparse samples, motion data and lower-resolution frames become a larger final result. Debugging becomes layered when reconstruction sits in the middle.
Agent Reliability Is an Evaluation Problem, Not a Prompting Problem
When an agent misses a step, the usual fix is a stricter prompt. The failure is more often in how completion is detected.
MCP Might Matter More Than Another Model Release
A model can reason well and still be useless inside a company if it cannot reach the files, repositories and tools where work lives.
Computer Use Is the Missing Layer Between Models and Software
Most integrations assume useful software exposes the right API. Much of real software never did.
Inference-Time Compute Is a New Scaling Axis
o1 improves when it is allowed to spend longer on a problem. A benchmark score without a compute budget is an incomplete number.
The Final Pixel Won't Come From the Renderer
Geometry, camera and scene structure stay reliable ground truth. More of final appearance is moving into learned systems.
Open Models Are Becoming Research Infrastructure
Llama 3.1 gets discussed as a benchmark result. The licence terms change which experiments are possible at all.
Multimodality Changes the Architecture, Not Just the Interface
GPT-4o is easy to read as a faster interface. Training one model end to end across text, vision and audio is an architectural change.
Benchmark Scores Are Not Model Capability
Claude 3 posts 86.8% on MMLU and 50.4% on GPQA Diamond. The chart is useful and it is not the same thing as capability.
Long Context Is Not Memory
Gemini 1.5 makes a million tokens usable. A larger working set is not a system that decides what should survive the session.