Skip to content
LyftData Docs

Workers

Workers execute jobs. The server stores job definitions, schedules runs, and deploys staged jobs to workers.

Built-in vs external workers

Section titled “Built-in vs external workers”
  • Built-in worker: Runs alongside the server and is the default for evaluation. If you followed the Evaluation Quickstart, you already have a built-in worker and do not need to install anything else to run jobs end-to-end.
  • External workers: Run on additional hosts so you can scale out, isolate workloads, or place execution closer to sources/destinations. External worker support depends on your license; see Licensing and Community Edition.

What a worker needs to connect

Section titled “What a worker needs to connect”

External workers need four core values:

  • LYFTDATA_WORKER_ID: the worker’s unique identity.
  • LYFTDATA_WORKER_API_KEY: the API key used to authenticate to the server.
  • LYFTDATA_URL: the base URL of the server (how the worker reaches it).
  • LYFTDATA_JOBS_DIR: the worker’s local data directory.

You typically create the worker ID in Workers, then create an API key in Settings → API Keys, and finally install the worker service on the target host.

Install an external worker

Section titled “Install an external worker”

Choose the guide that matches the worker host OS:

After the service starts, confirm the worker appears as Online in the UI. If it does not, the fastest checks are the server URL, API key, and firewall rules. See Networking & TLS and Troubleshooting.

After the worker is online

Section titled “After the worker is online”
  • Deploy a staged job to the worker to confirm end-to-end execution.
  • Enable metrics collection for new workers (disabled by default) via Next steps.
  • If you are scaling out, review System Requirements and the Scaling playbook.

Online means the server can see the worker heartbeat. It does not guarantee that every deployment or Provider Pack target job is serviceable. If a specific runtime reports degraded readiness, inspect the runtime-requirements report and target job status before restarting the worker. The common repairs are worker auth trust reset, worker settings or worker database ACL updates, and deployment runtime-requirements repair proposals. Follow Worker Recovery and Serviceability for the complete diagnostic and recovery sequence.