Workers

Workers execute pipelines. The server stores job definitions, schedules runs, and deploys staged jobs to workers.

Built-in vs external workers

Built-in worker: Runs alongside the server and is the default for evaluation. If you followed the Evaluation Quickstart, you already have a built-in worker and do not need to install anything else to run jobs end-to-end.
External workers: Run on additional hosts so you can scale out, isolate workloads, or place execution closer to sources/destinations. External worker support depends on your license; see Licensing and Community Edition.

What a worker needs to connect

External workers need four core values:

LYFTDATA_WORKER_ID: the worker’s unique identity.
LYFTDATA_WORKER_API_KEY: the API key used to authenticate to the server.
LYFTDATA_URL: the base URL of the server (how the worker reaches it).
LYFTDATA_JOBS_DIR: the worker’s local data directory.

You typically create the worker ID in Workers, then create an API key in Settings → API Keys, and finally install the worker service on the target host.

Install an external worker

Choose the guide that matches the worker host OS:

After the service starts, confirm the worker appears as Online in the UI. If it does not, the fastest checks are the server URL, API key, and firewall rules. See Networking & TLS and Troubleshooting.

After the worker is online

Deploy a staged job to the worker to confirm end-to-end execution.
Enable metrics collection for new workers (disabled by default) via Next steps.
If you are scaling out, review System Requirements and the Scaling playbook.