Worker Recovery and Serviceability

A worker can be online while one of its assigned jobs is unable to run. Recovery starts by identifying which layer failed rather than immediately restarting or replacing the worker.

Applies to

Worker control and health are beta capabilities across Community and Enterprise deployments. External and hybrid workers are Enterprise capabilities for distributed-worker and hybrid topologies. Verify edition and topology support in your selected release.

Caution

Worker heartbeat and placement are not serviceability guarantees. The current worker topology does not provide a universal automatic-failover or clustered control-plane high-availability guarantee.

The serviceability ladder

Check these layers in order:

Layer	Question
Server	Is the control plane reachable and healthy?
Worker heartbeat	Is the worker connected and reporting recent status?
Runtime authorization	Can an external worker obtain and refresh a valid auth lease?
Placement	Is the exact job intended for this worker or worker group?
Deployment	Did the exact rendered job reach a runnable state?
Runtime requirements	Are credentials, bindings, worker settings, database ACLs, and source prerequisites satisfied?
Capacity	Does the worker have the resources and component capabilities required by the job?
Outcome	Did a canary run and, where required, produce an external receipt?

Stop at the first failed layer. A restart can hide the original evidence and will not repair a missing job binding, credential scope, worker database ACL, or unavailable destination.

Capture evidence first

lyftdata doctor --json
lyftdata workers status <worker-id>
lyftdata workers sysinfo <worker-id>
lyftdata workers logs <worker-id> --limit 200

Also capture:

• server and worker versions;
• target worker ID and exact job name;
• target job deployment state and failure reason;
• runtime-requirements response;
• recent server and worker logs; and
• the last known-good deployment or receipt identity.

Write an explicit support-bundle path when working with support:

lyftdata doctor --bundle ./lyftdata-support.zip

Review the bundle before sharing it and handle it as sensitive operational data.

Common recovery paths

Connectivity or TLS

Confirm LYFTDATA_URL, DNS, TCP reachability, certificate trust, proxy behavior, and clock synchronization from the worker host. Use insecure TLS only for bounded evaluation with a self-signed certificate.

License or runtime authorization

An external worker needs a licensed deployment and a valid signed auth lease. If the UI or logs report community_edition_no_external_workers or license_expired_or_invalid, restore a valid license and wait for the next lease-refresh cycle. A normal license recovery does not require a worker restart.

If runtime requirements report a worker-auth trust-root problem, use the product trust-health and trust-reset operations. Do not edit auth-cache or settings databases.

Missing runtime prerequisites

For a worker that is online but not serviceable, inspect the exact runtime-requirements report. Repair the named blocker:

• bind the required credential to the job;
• add the worker to the credential’s allowed workers;
• update worker database settings or the exact job ACL;
• stage and deploy the required job; or
• repair the source or Provider Pack setup journey.

Rerun readiness after the repair. Do not treat a successful settings write as proof until the report and target job both converge.

Corrupt local state

Stop the worker service before a one-time state repair:

lyftdata-worker --reset-state repair

The repair mode checks worker SQLite state and rebuilds unhealthy files from a logical dump; it requires sqlite3 on the worker host. Each reset mode asks for confirmation. Add --yes only in deliberately reviewed, non-interactive automation.

Use narrower auth repair when only cached authorization is affected:

lyftdata-worker --reset-state auth

Use --reset-state full only as a last resort after recording the worker identity, local state path, deployed jobs, and recovery plan. Full reset recreates local runtime state and can require the worker API key or a fresh auto-enrollment.

Run the worker normally, without --reset-state, after the one-time repair.

Upgrade or replace a worker

External-worker network, credential, upgrade, and recovery design belongs to the deployment operator.

1. Record the worker identity, labels, settings, assigned jobs, and current evidence.
2. Pause upstream schedules or remove the affected job deployments when the workload cannot tolerate two active copies.
3. Stop the service and back up only the state required by your recovery policy.
4. Install a worker version supported by the server.
5. Restore or re-enroll the intended identity through product-supported paths.
6. Confirm heartbeat and runtime authorization.
7. Reapply or reconcile job placement.
8. Rerun runtime requirements and a bounded canary.
9. Confirm downstream receipts before restoring normal schedules.

Do not start a replacement with the same active identity and local state while the original worker can still run. That can create ambiguous ownership and duplicate effects.

Remove a worker safely

Before removing a worker record:

• stop new work at the source or scheduler;
• identify in-flight and checkpointed work;
• remove or relocate its job deployments;
• preserve required logs, traces, runtime artifacts, and receipts;
• confirm the service is stopped; and
• confirm the intended replacement is serviceable.

Then remove the worker through the UI or CLI:

lyftdata workers remove <worker-id>

Removal does not automatically replay lost work or prove that a replacement has assumed it.

Recovery acceptance

A recovered worker should pass all of these checks:

• recent heartbeat and system information;
• valid runtime authorization where external;
• exact intended placement and deployment identity;
• no unresolved runtime-requirements blockers;
• adequate measured capacity for the target job;
• successful bounded canary; and
• external receipt when the job’s purpose includes an external effect.

For bootstrap credentials, see Worker Authentication. For delivery after restart, see Delivery Semantics.