"Turn on JSON mode" is the most common half-fix in structured-output work. JSON mode and schema mode sound like the same feature with different marketing, and they are not: one guarantees the output parses, the other guarantees it matches your structure. Code that reads named fields out of the result needs the second guarantee, and a surprising amount of production code is running on the first.

This is the detailed follow-up to the cluster pillar, how to get reliable JSON out of LLMs, which maps all four families of approach. Here we compare the two provider-enforced ones.

The one-sentence difference

  • JSON mode: "the output will be syntactically valid JSON." Nothing else. Keys, types, nesting, whether it is an object or an array — all still the model's choice.
  • Schema mode: "the output will validate against this specific JSON Schema." Keys, types, required fields, enums — enforced during decoding, not checked afterward.

The failure mode JSON mode leaves open is valid-but-wrong-shape. You asked for {"items": [...], "total": number} in the prompt; the model returns {"order": {"line_items": [...], "sum": "42.50"}}. It parses fine. Your code crashes anyway, or worse, silently reads undefined.

OpenAI: json_object vs json_schema

OpenAI exposes both, which makes it the cleanest place to see the distinction. JSON mode is:

"response_format": {"type": "json_object"}

One documented quirk: with json_object you must still mention JSON in your prompt, or the API rejects the request — the mode constrains syntax but relies on the prompt for intent. Schema mode is:

"response_format": {
  "type": "json_schema",
  "json_schema": {
    "name": "invoice_extraction",
    "strict": true,
    "schema": {
      "type": "object",
      "properties": {
        "items": {"type": "array", "items": {"type": "object", "properties": {"name": {"type": "string"}, "qty": {"type": "integer"}}, "required": ["name", "qty"], "additionalProperties": false}},
        "total": {"type": "number"}
      },
      "required": ["items", "total"],
      "additionalProperties": false
    }
  }
}

With strict: true, decoding is constrained so the model cannot emit a token that would violate the schema. Note the shape requirements strict mode imposes as of 2026-07: every object needs additionalProperties: false, all properties must be listed in required (use a nullable type union for genuinely optional fields), and only a subset of JSON Schema is supported — no oneOf at the root, limited string formats. Design your schema inside that subset rather than fighting it. (OpenAI's newer Responses API expresses the same thing under text.format instead of response_format; same semantics, different envelope.)

Anthropic: there is no "JSON mode" switch — use a tool

The Claude API, as of 2026-07, does not have a response_format toggle in its long-standing stable API. The documented pattern for schema-shaped output is tool use: define a tool whose input_schema is your target structure, then force the model to call it.

{
  "tools": [{
    "name": "record_invoice",
    "description": "Record the extracted invoice data.",
    "input_schema": {
      "type": "object",
      "properties": {
        "items": {"type": "array", "items": {"type": "object", "properties": {"name": {"type": "string"}, "qty": {"type": "integer"}}, "required": ["name", "qty"]}},
        "total": {"type": "number"}
      },
      "required": ["items", "total"]
    }
  }],
  "tool_choice": {"type": "tool", "name": "record_invoice"}
}

The response contains a tool_use content block whose input field is your structured object — you never parse text at all. The tool does not have to exist; it is a schema delivery vehicle. Anthropic has additionally been shipping a native structured-output mode in beta; if you are reading this after mid-2026, check the current docs, because this is exactly the kind of gap providers close. The tool-based pattern also generalizes to real actions, which is the subject of function calling patterns that work.

Google: responseMimeType vs responseSchema

Gemini 2.5 models split the two guarantees across two generation-config fields. "responseMimeType": "application/json" alone is JSON mode. Add a schema and you get schema mode — via responseSchema, or on Gemini 2.5+ via responseJsonSchema, which accepts standard JSON Schema:

"generationConfig": {
  "responseMimeType": "application/json",
  "responseSchema": {
    "type": "OBJECT",
    "properties": {
      "items": {"type": "ARRAY", "items": {"type": "OBJECT", "properties": {"name": {"type": "STRING"}, "qty": {"type": "INTEGER"}}}},
      "total": {"type": "NUMBER"}
    },
    "required": ["items", "total"]
  }
}

The responseSchema dialect is OpenAPI-flavored (uppercase type names in the REST API) with a limited keyword set; responseJsonSchema on Gemini 2.5+ takes standard JSON Schema with broader keyword support, including anyOf and $ref. Either way, schemas kept simple and flat travel best across providers.

Local models: the same split, stronger enforcement

The distinction carries over to local runtimes, where you control the decoder directly. Ollama's format: "json" is JSON mode; as of 2026-07 its official API stops there, without schema-driven constrained decoding, so schema mode on local models comes from other runtimes. llama.cpp does it with GBNF grammars, which can express constraints beyond JSON Schema entirely, and it ships a converter from JSON Schema to GBNF. vLLM exposes guided decoding through grammar backends. As of 2026-07 this means even a Llama 3.1 8B or small Mistral running locally can be held to a schema as strictly as GPT-4o — the enforcement is a property of the sampler, not of model quality. Model quality still decides whether the values in those fields are any good.

The gotchas that survive the feature matrix

JSON mode leaves field names loose. Worth repeating because it is the whole point: json_object without a schema means your prompt is the only thing steering structure, and prompts are suggestions. If two runs can disagree about a key name, eventually they will.

Schema mode has a small latency cost. The first request with a new schema pays a compilation step (OpenAI documented added latency on first use of a schema when the feature launched; cached thereafter), and constrained sampling adds minor per-token overhead. As of 2026-07 this is rarely a reason to avoid it.

Fenced JSON still exists in the wild. True constrained decoding cannot emit a markdown code fence. But OpenAI-compatible proxies, some local runtimes, and prompt-hint "json modes" can and do return ```json ... ```. If your stack touches anything besides a first-party API, keep a fence-stripping fallback in the parser.

Schema mode does not validate meaning. A perfectly shaped object can still contain a product ID that does not exist. Structural guarantees end where your business logic begins — that handoff is covered in retry loops for validation failures.

The takeaway

Use schema mode whenever your code reads specific fields, which is nearly always: json_schema with strict: true on OpenAI, responseSchema or responseJsonSchema on Gemini, a forced tool on Claude (with a native structured-output mode in beta), a grammar on llama.cpp or guided decoding on vLLM. Reserve bare JSON mode for the narrow case where you genuinely want free-form JSON — exploratory extraction, model-chosen keys — and treat it as what it is: a parser guarantee, not a contract.