Models & Providers

Model profile

model Primary {
  provider: "openai-compatible"
  base_url: "https://gateway.example/v1"
  model: "text-model-l"
  api_key: getEnv("api_key")
  timeout: 30000
  max_tokens: 4000
  capabilities: ["json_schema", "tool_calling"]
  cache: true
  cache_ttl: 60000
  cache_max_entries: 256
  stream: false
  reasoning: false
  multimodal: false
  cost_per_1k_tokens: 0.002
}

agent Analyst {
  model: Primary
}

Per-model pricing matters when one agent runs several models: without it a cheap and an expensive model are billed at the same rate, and the escalate policy's max_cost_per_call ceiling would measure tokens rather than cost. For the agent-wide rate and the budgets → Budgets and limits.

model Primary {
  provider: "openai-compatible"
  base_url: "https://gateway.example/v1"
  model: "text-model-l"
  api_key: getEnv("api_key")
  max_token: 100        // dropped — one letter short of the real key
  cache_ttll: 5000      // dropped
  multimodel: true      // dropped — the flag stays false
}

Provider adapters

If base_url already contains the endpoint path (e.g. …/chat/completions) it is not appended a second time; the query string is preserved. An unknown provider name is rejected at parse time, before any request is made: Error: Model 'M' has unknown provider '…'. Valid values: …

Tool call mapping

Structured output support

How the request is shaped

The body sent for every ask is built with these rules:

Correlation headers are sent on every request:

  x-luma-run-id      → logical_run_id   (the logical run)
  x-luma-attempt-id  → attempt_id       (retry / policy candidate)
  x-luma-request-id  → local_request_id (a single HTTP request)

The provider ID read back from the response:
  x-request-id · request-id · anthropic-request-id → provider_request_id

These values are written into telemetry events as well, which lets you line a trace up with your gateway logs. → Correlation

Response handling

raw response (JSON or SSE)
   ↓ adapter normalization → { content, refusal, toolCalls, usage }
   ↓ refusal present → MODEL_REFUSAL (retryable: false)
   ↓ toolCalls present → run the tools, append the result, back to the loop
   ↓ content empty → MODEL_EMPTY_CONTENT (retryable)
   ↓ ```json fences are stripped
   ↓ JSON.parse fails → MODEL_INVALID_RESPONSE (retryable)
   ↓ local schema validation fails → MODEL_INVALID_RESPONSE (retryable)
   ↓ result

If the model wraps its response inside a ```json … ``` block the fences are stripped automatically. Use LUMA_DEBUG_MODEL_RESPONSE=1 to see the raw response.

Capability probe

Before the program starts, Luma sends a request to the gateway's /models endpoint for every model that will be used and verifies that the model supports the capabilities it needs.

{
  "model_runtime": {
    "capability_policy": "strict",
    "capability_probe_timeout_ms": 5000
  }
}

export LUMA_CAPABILITY_POLICY=strict
export LUMA_CAPABILITY_PROBE_TIMEOUT_MS=3000

Environment variables override the config. The probe timeout defaults to 5000 ms and can be cancelled with Ctrl-C.

Warning: 'text-model-l' has no json_schema capability information.

Streaming, reasoning, multimodal

model VisionReasoner {
  provider: "openai-compatible"
  base_url: "https://gateway.example/v1"
  model: "vision-model"
  api_key: getEnv("api_key")
  stream: true
  reasoning: true
  multimodal: true
}

For multimodal details and the file pipeline → IO & Multimodal.

Model response cache

model Cached {
  provider: "openai-compatible"
  base_url: "https://gateway.example/v1"
  model: "text-model-l"
  api_key: getEnv("api_key")
  cache: true
  cache_ttl: 60000
  cache_max_entries: 256
}

Run policy: multi-model strategies

run_policy determines how an ask call is executed across more than one model. It is attached to an agent and covers all of that agent's LLM calls.

run_policy Resilient {
  strategy: "fallback"
  models: [Primary, Backup]
  on: ["timeout", "rate_limit", "provider_error", "empty_content"]
  max_attempts: 2
}

agent Analyst {
  model: Primary
  run_policy: Resilient
}

Strategies

Configuration keys

Using a key with the wrong strategy is a parse error — tie, for example, is valid only for vote.

on: which errors bring it into play

The valid trigger values:

retryable        timeout          rate_limit       provider_error
empty_content    invalid_response refusal          invalid_tool_call
cancelled        unsupported      budget_exceeded

raw error codes can also be written:
MODEL_TIMEOUT · MODEL_RATE_LIMIT · MODEL_PROVIDER_ERROR · …

"retryable" is a special value: it looks at the error's retryable flag, not at its code. An unrecognized trigger is a parse error.

Strategy details

fallback

run_policy Resilient {
  strategy: "fallback"
  models: [Primary, Backup, Reserve]
  on: ["timeout", "rate_limit", "provider_error"]
  max_attempts: 2
}

The models are tried in order. If the error does not match the on list it is thrown right away — so a validation failure does not needlessly push the call onto a second model. max_attempts limits how many models are tried.

escalate

run_policy Adaptive {
  strategy: "escalate"
  models: [Cheap, Strong]
  on: ["invalid_response", "empty_content"]
  max_cost_per_call: 0.02
}

agent Extractor {
  model: Cheap
  run_policy: Adaptive
  retry: 1
  cost_budget: 0.5
}

The models are a ladder from the cheapest to the strongest and one rung runs at a time. A model is reached only when the model before it failed with a trigger from the on list; an error outside that list goes to the caller unchanged. When the cheap model answers, the strong model is not called at all — that is what separates escalate from combo, which runs every stage whether or not the previous one succeeded. At least two models are required: Escalate run policy 'Adaptive' must specify at least two models, from the cheapest to the strongest.

• retry is the inner loop, escalation the outer one. A stage spends its retries on the model that is running — the cheap model corrects itself first — and only then is a stronger model paid for. The policy above makes two requests to Cheap before it ever reaches Strong. Every other multi-model strategy keeps running its candidates with retry: 0.
• max_cost_per_call is a ceiling on a single policy call. It is checked after every stage against one ledger shared by the whole ladder, and it is inclusive: a call may spend exactly the ceiling, but is then left with nothing to climb with.
• Reaching the ceiling stops the climb, and the model that was not bought is named: MODEL_BUDGET_EXCEEDED · RunPolicy 'Adaptive' spent 0.03 of its max_cost_per_call of 0.02, so it did not escalate to model 'Strong'.
• Exceeding the ceiling means the result is not returned either, valid or not — the overrun is reported rather than swallowed: MODEL_BUDGET_EXCEEDED · RunPolicy 'Adaptive' spent 0.03 on this call, above its max_cost_per_call of 0.02.
• Every stage is priced with the cost_per_1k_tokens of the model profile that served it, falling back to the agent's rate. Under escalate that ledger spans the whole ladder instead of restarting at each rung, so the agent's cost_budget / token_budget can end the climb as well: Agent cost budget exceeded (0.5). With no price anywhere the cost is computed as 0 and no ceiling ever fires.

parallel

run_policy Fanout {
  strategy: "parallel"
  models: [Primary, Backup]
  select: "all"
}

agent Analyst {
  run_policy: Fanout

  fn generate() -> Candidate[] {
    ask "Generate candidate results" -> Candidate[]
  }
}

race

run_policy Fastest {
  strategy: "race"
  models: [Primary, Backup]
}

The first successful result wins; the remaining requests are cancelled with AbortSignal. If all of them fail, the first error is thrown.

vote

run_policy Consensus {
  strategy: "vote"
  models: [Primary, Backup, Reviewer, Judge]
  quorum: 2
  tie: "highest_confidence"
}

• Only schema-valid results are voted on; they are compared through their canonical JSON (keys sorted).
• If the quorum is not reached: MODEL_VOTE_NO_QUORUM (retryable: true).
• Ties: fail → MODEL_VOTE_TIE; first → takes the first one; highest_confidence → looks at the numeric confidence field in the results.
• If highest_confidence is chosen and the results carry no confidence: MODEL_VOTE_CONFIDENCE_UNAVAILABLE.

fusion

run_policy SafeFusion {
  strategy: "fusion"
  models: [Primary, Backup, Reviewer]
  synthesizer: Primary
  max_fusion_inputs: 3
  fusion_input_tokens: 4000
  synthesizer_policy: "deny_tools"
}

The candidates run concurrently; the successful ones are handed to the synthesizer model as "candidate outputs" and the final schema-valid result is produced.

• If there is no successful candidate: MODEL_FUSION_NO_INPUTS.
• If the estimated input tokens exceed fusion_input_tokens: MODEL_BUDGET_EXCEEDED.
• synthesizer_policy: "deny_tools" default gives the synthesizer no tools and rejects write tools; "inherit" passes the agent's tools through.
• The synthesizer prompt explicitly tells the model to treat the candidates as data and not to carry out the instructions inside them.

combo

run_policy Pipeline {
  strategy: "combo"
  models: [Draft, Reviewer, Finalizer]
  max_stages: 3
}

The models run in sequence. Each stage's schema-valid output is added to the next stage's prompt as "the previous stage's output, improve it". The result of the last stage is returned.

Which strategy when?