Operations

Telemetry & Tracing

Luma records every LLM request, every tool call, every agent method, every query, every HTTP call and every validation rule as a separate event. That lets you read why an extraction failed instead of guessing.

The event model

Every event has the same base shape:

json

{
  "type":       "tool",
  "name":       "calculate",
  "start":      1785242769011,
  "durationMs": 1,
  "ok":         true,
  "error":      null,
  "metadata":   { "cached": false }
}

`type`	What `name` holds	When it is emitted
`llm`	The model identifier (`text-model-l`)	For every provider request — cache hits included.
`tool`	The tool name	For every tool execution — direct and native alike.
`agent`	`Agent.method`	When an agent method call completes.
`validation`	The path of the validated field	Separately for every validation rule — including the ones that passed.
`http`	`http.<fn>` (`http.get`, `http.postJson` …)	For every `std/http` call — the failed ones included.
`query`	The query name	When a `query` returns. The `type` is the declaration kind.
`command`	The command name	When a `command` returns — same rule, its own type value.
`parallel_over`	The result variable name	Only when `on_error continue` skipped at least one element.
`error`	The operation that died	v0.9.0. One synthetic final event on a run that failed, written after the events collected so far. It carries `ok: false` like every other event.

Nine type values, not four. http, query, command and parallel_over arrived with the v0.4.0 data surface, so any filter written against llm|tool|agent|validation — including every recipe published before v0.4.0 — silently drops that whole surface out of the report. The declarations themselves are documented under Data & Integration; this page owns their telemetry view.

There are a lot of validation events

An event is emitted for every field, every rule and every array element. On a complex schema a single ask can produce hundreds of validation events. This is deliberate: you see exactly which rule ran and which one blew up. Use the jq recipes below to filter them.

Event metadata

llm

json

{
  "agent": "Analyst", "method": "analyze", "instance": "analyst",
  "model": "text-model-l", "provider": "openai-compatible",
  "logical_run_id":      "run-8f3c…",
  "attempt_id":          "run-8f3c…:attempt-1",
  "local_request_id":    "request-1a2b…",
  "provider_request_id": "req_abc123",
  "cached": false,
  "usage": {
    "prompt_tokens": 812, "completion_tokens": 143,
    "total_tokens": 955,  "reasoning_tokens": 0
  },
  "cost": 0.00955,
  "cost_per_1k_tokens": 0.01
}

tool

Field	Meaning
`logical_run_id` · `attempt_id`	The identifiers of the LLM flow that triggered the tool.
`local_request_id` · `provider_request_id`	The HTTP request that produced the tool call.
`provider_tool_call_id`	The tool call identifier the provider assigned.
`logical_tool_call_id`	`attempt_id + provider_tool_call_id` — stays unique across retries.
`timeout`	The effective timeout that was applied (ms).
`cached`	Did the result come from the cache?
`single_flight`	Was it shared with a concurrent identical call?
`idempotent` · `deduplicated`	Did it come back from the idempotency store?

agent

{ "instance": "h", "deterministic": true }

deterministic: true indicates that the method ended with return (that it made no LLM call).

validation

The rule field tells you which rule ran:

`rule` value	Meaning
`type`	Type validation (extra field: `value_type` — it was `type` until v0.9.0, which collided with the event's own `type` and produced a duplicate `luma.type` attribute in the OTel export).
`schema`	Object-level schema validation (extra field: `schema`).
`tagged_union`	Tagged union member validation (extra fields: `schema`, `member`). It is missing from most published filters — a list of rules that omits it drops every discriminated-union check.
`field`	Field-level failure (missing field, alias conflict).
`optional`	The optional field was not present — skipped.
`unknown_field`	A field that is not defined in the schema.
`format:email` · `match:^…$`	Format and pattern rules.
`minLength:2` · `maxLength:40`	Length rules.
`min:0` · `max:100`	Numeric range rules.
`minItems:1` · `maxItems:5` · `unique` · `contains` · `uniqueBy:id`	Collection rules.
`valid:x => …`	The `@valid` expression (the expression itself appears in the code).
`where`	Invariant (extra field: `expression`).
`conditional_required` · `conditional_forbidden`	`when` rules (extra field: `condition`).

severity tells a soft violation from a hard one

Since v0.8.0 a validation event also carries metadata.severity. It is the only way to tell, from the trace alone, whether a failed rule stopped the run or merely flagged the value: "warn" means the value was accepted with a finding, anything else means the failure was fatal to that attempt.

jq -r '.events[] | select(.type=="validation" and .ok==false and .metadata.severity=="warn") | .metadata.rule' trace.json

Without it a report counts a shaded run and a failed run as the same thing.

http

{ "method": "GET", "status": 200, "headers": { "x-api-key": "[REDACTED]", "cookie": "session=…" } }

The event name is http.<fn>. A failed call emits an event too, with ok: false, the reason in error and no status — a request that never reached the host has no status to report. The URL and the request body are never written to the trace; the headers are, after redaction. Details in std/http.

query · command

{ "datasource": "TestDb", "rows": 3 }

The event type is the declaration kind itself and name is the declaration name, so a query Orders produces { "type": "query", "name": "Orders" }. rows is the length of the returned list — but only where a list is returned. A single-row query (-> Schema rather than Schema[]) and every command report rows: 1, because the runtime counts arrays and treats anything else as one row. A command's affected row count is its return value, not its telemetry. Failed calls emit the same event with ok: false and datasource only.

parallel_over — a drop counter, not a span

{ "type": "parallel_over", "name": "kept", "ok": true, "metadata": { "total": 3, "skipped": 1 } }

This event is emitted only when skipped > 0, and only under on_error continue. It does not mark that a parallel over ran: a fan-out that skipped nothing leaves no event at all, and a fail_fast fan-out never emits one. ok is always true — the fan-out itself succeeded; the event reports how much of the input it dropped along the way.

That makes it the only machine-readable signal that on_error continue shortened a result list. A failed element leaves no hole — the list simply comes back shorter, and nothing else in the trace or on stdout says so. The recipe below reads it; the semantics live in parallel over.

Metrics

json

{
  "llmCalls":         3,
  "toolCalls":        5,
  "promptTokens":     2436,
  "completionTokens": 419,
  "reasoningTokens":  0,
  "totalTokens":      2855,
  "estimatedCost":    0.02855
}

Metric	How it is computed
`llmCalls`	The number of real requests that went to the provider. Cache hits and shared single-flight requests are not counted.
`toolCalls`	All tool calls (cache hits included).
`promptTokens` · `completionTokens` · `totalTokens`	Summed from the provider's `usage` field.
`reasoningTokens`	`reasoning_tokens` or `completion_tokens_details.reasoning_tokens`.
`estimatedCost`	`totalTokens / 1000 × cost_per_1k_tokens`. If the agent does not set that option, it stays `0`.

Gateways that do not return usage

Some gateways never send the usage field at all. In that case the token metrics stay 0, cost cannot be computed, and token_budget/cost_budget never fire. If you rely on budget enforcement, verify that your gateway returns usage.

Export formats

Flag	Output
`--trace`	The full report to stderr — it does not pollute the program result on stdout.
`--trace-file trace.json`	A single JSON of `{ metrics, traces, events }`.
`--trace-jsonl trace.jsonl`	One line per event; the last line is `{"type":"metrics", …}`.
`--otel-file otel.json`	An OTLP-compatible `resourceSpans` structure.

luma program.luma \
  --trace-file  /tmp/trace.json \
  --trace-jsonl /tmp/trace.jsonl \
  --otel-file   /tmp/otel.json

All four are written when the run fails, too

A failing run is when a trace is worth the most, so trace writing happens in a finally position: whatever accumulated up to the failure reaches disk, with the final error recorded as an event. The terminal message names the rules that failed and how many attempts were spent, rather than only output schema failed validation.

This was not always true — until v0.5.0 a run that ended in an error produced no trace files at all, so the one case that most needed diagnosis was the one case with nothing to read. Reported from the field as #100.

Aggregate traces

The traces array groups events by logical_run_id:

json

{
  "traceId":       "e19fbd6da18af2dc3b57561c073adf7a",
  "logicalRunId":  "run-8f3c…",
  "start":         1785242769010,
  "durationMs":    1843,
  "ok":            true,
  "attempts": [
    { "attemptId": "run-8f3c…:attempt-1", "start": …, "end": …, "ok": false, "events": 12 },
    { "attemptId": "run-8f3c…:attempt-2", "start": …, "end": …, "ok": true,  "events": 14 }
  ]
}

Events that carry no logical_run_id (deterministic agent methods, direct tool calls) are placed in traces of their own.

OpenTelemetry

json

{
  "resourceSpans": [{
    "resource":   { "attributes": [{ "key": "service.name", "value": { "stringValue": "luma" } }] },
    "scopeSpans": [{
      "scope": { "name": "luma-runtime" },
      "spans": [{
        "traceId": "e19fbd6d…", "spanId": "a1b2c3d4…",
        "name": "calculate",
        "startTimeUnixNano": "1785242769011000000",
        "endTimeUnixNano":   "1785242769012000000",
        "attributes": [
          { "key": "luma.cached", "value": { "boolValue": false } },
          { "key": "luma.type",   "value": { "stringValue": "tool" } }
        ],
        "status": { "code": 1 }
      }]
    }]
  }]
}

All event metadata is converted into span attributes under the luma.<key> prefix. status.code: 1 is success, 2 is failure. The file can be POSTed straight to an OTLP collector.

The correlation chain

logical_run_id            one logical ask flow — stays the SAME across retries
   │
   ├── attempt_id            each retry attempt / each run policy candidate
   │      │                  run-8f3c…:attempt-2
   │      │                  run-8f3c…:fallback-2-Backup
   │      │
   │      ├── local_request_id       a single HTTP request (Luma generates it)
   │      ├── provider_request_id    the identifier the gateway returned
   │      │
   │      └── provider_tool_call_id  the provider's tool call identifier
   │             └── logical_tool_call_id  attempt + tool call
   │
   └── second attempt, third attempt …

These identifiers are also sent as HTTP headers on every request:

x-luma-run-id:      run-8f3c…
x-luma-attempt-id:  run-8f3c…:attempt-2
x-luma-request-id:  request-1a2b…

That way you can match a record in your gateway logs one-to-one with an event in the Luma trace.

attempt_id pattern	When
`<run>:attempt-N`	Normal `retry` attempts.
`<run>:fallback-N-<Model>`	`fallback` policy candidates.
`<run>:parallel-N-<Model>` · `race-…` · `vote-…`	The candidates of the corresponding policy.
`<run>:fusion-<Model>`	The fusion synthesizer call.
`<run>:combo-N-<Model>`	A combo pipeline stage.

Secret redaction

Trace files have to be shareable; the runtime ensures that in two layers:

Layer	What it does
Value redaction	Every known API key value (`api_key`, `OPENAI_API_KEY`, `ANTHROPIC_API_KEY` and every model profile key) is replaced with `[REDACTED]` in error messages. `Authorization: Bearer …` headers are masked as well.
Key-name redaction	Inside metadata, every field whose name matches the `api_key`, `password`, `secret`, `credential` or `authorization` pattern — even when nested — has its whole value replaced with `[REDACTED]`.

The key-name layer is what governs HTTP headers: a header name arrives in the metadata as a key, so authorization and x-api-key lose their entire value — not just the token after Bearer, the way the value layer trims it inside an error message. Every header name the pattern does not match goes into the trace verbatim: cookie, x-auth-token, x-session and signature headers are not redacted. Only header values are additionally run through the value layer first, so a header carrying a known model API key is caught by its value regardless of its name. Either keep a credential in a header name the pattern catches, or treat the trace file as being as sensitive as the credential.

Prompt content is not redacted

Redaction covers error messages and metadata. Prompt texts are not written into trace events, but a piece of the model output can show up inside a validation error message. On top of that, the LUMA_DEBUG_MODEL_RESPONSE=1 setting appends the first 4000 characters of the raw model response to the error detail — do not leave that on in production.

jq recipes

Failed validations

jq -r '.events[] | select(.type=="validation" and .ok==false)
        | "\(.name) [\(.metadata.rule // "-")] \(.error // "-")"' trace.json

LLM requests: model, duration, tokens, cost

jq -r '.events[] | select(.type=="llm")
        | "\(.metadata.model)  \(.durationMs)ms  \(.metadata.usage.total_tokens // 0)tok  $\(.metadata.cost // 0)"' trace.json

The 10 slowest steps

jq -r '.events | sort_by(-.durationMs) | .[:10][]
        | "\(.durationMs)ms  \(.type)  \(.name)"' trace.json

Tool call summary (with cache hits)

jq -r '.events[] | select(.type=="tool")
        | "\(.name)  ok=\(.ok)  cached=\(.metadata.cached // false)  \(.durationMs)ms"' trace.json

Which retry attempt succeeded?

jq -r '.traces[] | select(.attempts | length > 1)
        | "\(.logicalRunId): " + ([.attempts[] | "\(.attemptId)=\(.ok)"] | join("  "))' trace.json

The validation rules that fail most often

jq -r '.events[] | select(.type=="validation" and .ok==false) | .metadata.rule' trace.json \
  | sort | uniq -c | sort -rn

Silently dropped elements (`parallel over`)

jq -r '.events[] | select(.type=="parallel_over")
        | "\(.name): \(.metadata.skipped) of \(.metadata.total) elements dropped"' trace.json

No output means nothing was skipped: the event does not exist when skipped is 0. This is the only place a shortened on_error continue result list shows up — run it in CI on every trace that has a parallel over in it.

The data surface: queries, commands, HTTP

jq -r '.events[] | select(.type=="query" or .type=="command" or .type=="http")
        | "\(.type)  \(.name)  ok=\(.ok)  \(.durationMs)ms  \(.metadata.rows // .metadata.status // "-")"' trace.json

All errors (of every type)

jq -r '.events[] | select(.ok==false) | "\(.type)  \(.name)  \(.error)"' trace.json

Watching the JSONL stream live

tail -f trace.jsonl | jq -r 'select(.type=="llm" or .ok==false)
                             | "\(.type) \(.name) ok=\(.ok)"'

Cost and budget tracking

luma

agent Controlled {
  model: Primary
  cost_per_1k_tokens: 0.01
  token_budget: 20000
  cost_budget: 0.50
}

The budget check runs after the usage information of the LLM response has been read: the request that exceeds the limit still completes, but the next request is blocked with MODEL_BUDGET_EXCEEDED. Because that error is retryable: false, the retry loop stops too.

# Total cost of a single run
jq '.metrics | { totalTokens, estimatedCost, llmCalls }' trace.json

# Token distribution per agent
jq -r '.events[] | select(.type=="llm")
      | "\(.metadata.agent // "top-level")\t\(.metadata.usage.total_tokens // 0)"' trace.json \
  | awk -F'\t' '{s[$1]+=$2} END {for (a in s) printf "%-24s %d\n", a, s[a]}' | sort -k2 -rn

Setting that lowers cost	Effect
`cache: true` (model profile)	If the same request repeats, the provider is not contacted.
`memory_limit`	Limits how far past messages can grow the prompt tokens.
`max_tokens`	Limits the output tokens.
`combo` run policy	Draft with the cheap model, touch up with the expensive one.
Intermediate files (`pre.md`)	Let you re-run the extraction without repeating the expensive vision step.
Narrow schemas	The JSON Schema is written into the system message, so it costs tokens on every request.

You pay for schema size on every request

The generated JSON Schema is written into the system message on every request. $defs only carries the definitions that are reachable from the root schema — unused schemas do not add cost. Even so, a deep schema graph is paid for on every request; splitting rarely used branches out lowers that fixed cost.

Telemetry & Tracing

The event model

Event metadata

llm

tool

agent

validation

http

query · command

parallel_over — a drop counter, not a span

Metrics

Export formats

Aggregate traces

OpenTelemetry

The correlation chain

Secret redaction

jq recipes

Failed validations

LLM requests: model, duration, tokens, cost

The 10 slowest steps

Tool call summary (with cache hits)

Which retry attempt succeeded?

The validation rules that fail most often

Silently dropped elements (parallel over)

The data surface: queries, commands, HTTP

All errors (of every type)

Watching the JSONL stream live

Cost and budget tracking

Silently dropped elements (`parallel over`)