Structured Outputs & Function Calling

Every LLM application that talks to other code eventually hits the same wall: the model writes prose, your program needs data. The gap between those two is where most production incidents in this space live, and the fixes have matured enough that there is now a right answer for almost every situation. This page is the map; the rest of the cluster covers each territory in detail.

This cluster goes deeper in four supporting articles: JSON mode vs schema mode on what each provider feature actually guarantees, function calling patterns that work on the tool-use shapes that show up in real code, retry loops for validation failures on what to do when the output fails your checks anyway, and tool selection prompting on writing tool names and descriptions the model can actually choose between.

The four families of approach

Everything anyone does to get structured data out of an LLM falls into one of four families, ordered here by how much they guarantee.

1. Prompt and pray, then parse. You ask for JSON in the prompt ("respond with only a JSON object, no prose"), then parse whatever comes back, often with a cleanup pass that strips markdown fences and trailing commentary. This was the default in 2023 — function calling and grammar-constrained local decoding already existed, but prompt-and-parse was what most code actually did — and it still has a place: it works with any model, including ones behind APIs that expose no structured-output features. Failure rate on frontier models is low but not zero, and it degrades exactly when you least want it to — on long, complex outputs.

2. JSON mode. The provider guarantees the response is syntactically valid JSON. OpenAI's response_format: {"type": "json_object"} is the archetype. Crucially, this says nothing about which JSON. Field names, types, and nesting are still whatever the model felt like. Valid-but-wrong-shape is the classic failure here.

3. Schema mode, also called grammar-constrained decoding. You hand the provider a JSON Schema, and decoding is constrained so the model can only emit tokens that keep the output valid against that schema. This guarantees both parseability and structure. As of 2026-07, OpenAI offers this as response_format: {"type": "json_schema", "json_schema": {..., "strict": true}}, Google's Gemini API takes a responseSchema (an OpenAPI-style subset) or, on Gemini 2.5 and later, full JSON Schema via responseJsonSchema in the generation config, and local runtimes have caught up fast: llama.cpp compiles JSON Schemas into GBNF grammars, Ollama accepts a schema in its format parameter, and vLLM does guided decoding through grammar backends. The json-mode-vs-schema-mode article walks through the exact API calls.

4. Function calling / tool use. Instead of asking for JSON in the response body, you define a tool whose parameters are your target shape, and the model "calls" it. The structured data arrives as the tool call's arguments. This is Anthropic's primary answer to the problem: the long-established pattern on the Claude API is not a response-format switch but a tool — you define a tool whose input_schema is your target structure and force it with tool_choice: {"type": "tool", "name": "..."}. (As of 2026-07 Anthropic also offers a native structured-output mode in beta; check current docs before relying on it.) OpenAI and Google support the same pattern alongside their response-format options.

Who offers what, as of mid-2026

Provider features in this area change quarterly, so treat this as a snapshot dated 2026-07 and verify against current docs before building.

• OpenAI (GPT-4o and successors): JSON mode via json_object, full schema enforcement via json_schema with strict: true, function calling with strict schemas on tool parameters. The most complete menu.
• Anthropic (Claude Sonnet 5, Opus-class models): tool use with input_schema and forced tool_choice is the established, documented pattern for structured extraction. Tool schemas are JSON Schema, so your target shape ports over directly.
• Google (Gemini 2.5 family): responseMimeType: "application/json" for JSON mode, plus responseSchema — or full JSON Schema via responseJsonSchema on Gemini 2.5+ — for enforced structure. Also supports function calling.
• Local (Llama 3.3, Mistral, Qwen via llama.cpp / Ollama / vLLM): grammar-constrained decoding is arguably strongest here, because you control the decoder. A GBNF grammar or JSON Schema constraint makes even a 7B model emit structurally perfect output every time.

One practical note on model choice: schema enforcement removes the structural advantage bigger models used to have. A small model with constrained decoding beats a large model with prompt-and-pray on parse rate. It does not beat it on whether the content in the fields is any good.

Valid JSON is not a correct answer

This is the trap that gets teams after they adopt schema mode and watch their parse errors drop to zero: schema catches structure, not truth.

Constrained decoding guarantees that {"product_id": "SKU-99231", "confidence": 0.95} is well-formed and has the right keys and types. It does not guarantee SKU-99231 exists in your catalog, that the confidence number means anything, or that the extraction is faithful to the source text. In fact, constrained decoding can make hallucinated values more fluent, because the model is forced to produce something schema-shaped even when the honest answer is "this field isn't in the document." Give the model an out — a nullable field or an explicit "not_found" enum value — or it will invent.

So the production pipeline is always two layers: structural validation (free, if you use schema mode) and semantic validation (yours: catalog lookups, range checks, cross-field consistency). When semantic checks fail, the fix is a bounded re-prompt with the specific error — the pattern covered in retry loops for validation failures.

What consuming this looks like downstream

Once the output is schema-valid and semantically checked, it stops being "AI output" and becomes ordinary data your system can route, store, and render. That is the whole point. A concrete example of the end state: NerdSip generates its five-minute lessons as structured JSON — title, hook, sections, quiz items, each field schema-validated before anything is stored — which is why a generated lesson can be rendered, indexed, and scheduled like any hand-authored one. When structured output works, the rest of the stack does not need to know an LLM was involved.

If your use case is less "extract a record" and more "let the model take actions," the same machinery applies but the design questions shift to tool granularity and loop control — that is the ground covered by function calling patterns that work and tool selection prompting.

The takeaway

Pick the strongest guarantee your provider offers: schema mode on OpenAI or Gemini, a forced tool schema on Claude, a grammar on local runtimes, and prompt-and-parse only when nothing better exists. Treat "it parses" as table stakes handled by the platform, keep semantic validation in your own code, and give the model an explicit way to say "not found" so the schema never forces a lie. As of 2026-07 that combination makes structural failures rare enough to treat as incidents rather than routine — truncated responses and refusals can still break the shape — and the failures you will actually spend time on are content failures, and those are a validation-loop problem, not a parsing problem.