Overview

Lyftdata is a data integration and data engineering solution for creating and maintaining performant data pipelines. Data integration means getting data from sources to destinations reliably. Data engineering means intermediate processing of data, as it moves from sources to destinations. This generally includes data transformation, filtering, validation, enrichment and conversion.

Lyftdata simplifies the creation, deployment and management of robust data pipelines. Lyftdata features a distributed architecture and simple deployment model, making it a flexible platform that also scales well.

In essence, Lyftdata consists of three basic components; the server, jobs and workers. Lyftdata server is the central and minimal component of a Lyftdata deployment. The server provides centralized management of jobs and workers. Multiple workers can be added alongside the server, to add more processing capacity to Lyftdata deployment. Jobs are created and tested on the server, and then deployed to, and executed by workers.

Each job defines a configuration of a specific data engineering workload. A job consists of an input, an output, and zero or more actions. Jobs can read and write data from/to a variety of sources and destinations, as well as perform data transformation tasks.