Distributed Traces

Traces help you follow a specific execution path when logs are too broad and metrics are too aggregated. LyftData exposes two distinct trace surfaces: OpenTelemetry-style spans in Observability, and bounded job trace-capture records addressed by job and worker. Keep them separate when collecting, querying, or deleting evidence.

Choose the right trace

LyftData has several similarly named surfaces:

Surface	Best for	Boundary
Run & Trace in the job editor	Inspecting a transient authoring or validation run	It does not prove sustained performance, distributed placement, or production recovery
Observability traces	Following OpenTelemetry-style spans and timing through the Observability UI	Span completeness depends on instrumentation, collection, sampling, and retention
Bounded job trace captures	Inspecting a selected job/worker sample record through `/api/traces` or the CLI	A capture is not the Observability span store and may contain sampled event data
OpenTelemetry input/output	Receiving or exporting supported observability signals	It is a runtime transport, not product phone-home telemetry
Product telemetry	Reviewing installation and usage summaries sent to a telemetry receiver	It does not carry the workload trace contract

For product reporting and privacy controls, see Product Telemetry.

Inspect Observability spans

Open Observability → Traces and filter by the attributes and time range available in your installed release. Open a trace to inspect its spans, waterfall, timing, status, and attributes.

Observability traces must already have been emitted and ingested. If a trace is missing, verify instrumentation, OpenTelemetry input or collector health, sampling, tenant/environment filters, and retention before concluding that the operation did not run.

Inspect bounded job trace captures

The /api/traces surface stores a different record: a bounded capture for a selected job and worker, including its sample and execution snapshot. Use the installed UI or API operation that explicitly creates a capture. Trace-start entry points are release-specific, so verify the installed OpenAPI document instead of constructing a legacy /traces?start=... URL.

Choose the smallest sample count and window that answer the question. Smaller captures reduce storage and the chance of retaining sensitive event fields. Only one active capture is accepted for the same job and worker pair.

The CLI inspects existing bounded captures:

lyftdata traces list --job-name orders_ingest --worker-id <worker-id>
lyftdata traces show <trace-id>

Add --json when passing the result to another tool. A multi-tenant administrator may also need --tenant <tenant-id>.

Pin a trace that must survive normal trace retention:

lyftdata traces pin <trace-id> --pinned true

Delete a trace after the investigation when it no longer needs to be retained:

lyftdata traces delete <trace-id>

Deletion is destructive and the CLI asks for confirmation. Retention and deletion of the primary trace record do not make a claim about copies already exported to another observability system or retained in backups.

Sampling, retention, and sensitive data

Sampling is deliberately bounded. Collection can omit activity because of probability, rate, time, sample-budget, instrumentation, feature, connectivity, or topology limits. Read a trace as positive evidence for the spans or samples present, not as a complete audit of everything that happened.

Trace samples and attributes can contain workload metadata or captured event fields. Apply the same access, retention, and export controls you apply to customer workload data:

grant trace access only to operators who need it;
avoid capturing more samples than the investigation requires;
pin only traces with a named retention reason;
review downstream OpenTelemetry storage and deletion separately; and
redact trace material before attaching it to a support request.

Completed bounded captures follow the configured capture-retention policy, while pinned captures are retained until unpinned or deleted. Observability spans have their own storage and retention configuration. Review both stores before making a deletion or capacity claim.

Export and correlation

OpenTelemetry inputs and outputs can move supported logs, metrics, and traces between LyftData and external observability systems. Configure those components as normal jobs and confirm their component-specific retry, authentication, and delivery behavior:

Keep job name, worker identity, trace identifier, environment, and the investigation time range together when correlating systems. Do not treat an exporter acknowledgement as proof that the downstream store indexed every span.

Troubleshooting

Symptom	Check
Bounded capture cannot start	Confirm the exact job is deployed to the selected worker, no active capture already exists for that pair, and the installed release exposes a supported start operation
Bounded capture stays in progress without samples	Exercise the workload, check worker connectivity, and review logs for sampling or message-delivery warnings
Expected Observability spans are missing	Check instrumentation, ingest, sampling bounds, topology, and the time range before concluding the step did not run
Trace detail exposes more data than expected	Stop collection where supported, review sampling and captured fields, then delete or restrict the record
Trace storage keeps growing	Review bounded captures and Observability span retention separately

Pair trace evidence with Logs and Issues, Monitoring, and Runtime Evidence and Receipts when investigating an end-to-end outcome.