Overview
LyftData connects to common object stores, protocols, analytics platforms, and delivery destinations. Start by deciding whether you need a runtime primitive, a recipe, or a governed Provider Pack.
Choose the right integration surface
Section titled “Choose the right integration surface”| Surface | Use it when | Verify before production |
|---|---|---|
| Runtime primitive | You want to build directly with a typed input or output such as S3, HTTP, Kafka, or OpenTelemetry | Component fields, credentials, network access, retry behavior, and delivery boundaries |
| Integration recipe | You want a documented composition of primitives for a recognizable use case | That the recipe matches your provider API, topology, and release |
| Provider Pack | Your Catalog contains a versioned package for the complete provider capability | Publication, installation, credentials, worker readiness, policy, and execution evidence |
Learn how packaged provider capabilities work in Provider Packs.
Provider Packs are a Beta Enterprise capability in the selected documentation snapshot. Pack availability is release- and environment-specific. Use the Catalog in your LyftData installation to see which Pack versions are published, then complete the Pack’s configuration and readiness checks.
Find a starting point
Section titled “Find a starting point”| Outcome or category | Start with |
|---|---|
| Object storage and file exchange | S3, Azure Blob, GCS, or FileStore |
| HTTP ingestion, polling, or delivery | HTTP endpoints |
| Observability and analytics delivery | Splunk, Grafana JSON, or the OpenTelemetry DSL components |
| Governed provider workflow | Provider Packs, then the installed Catalog |
| Product control-plane automation | Product API, not a pipeline connector |
Terms such as storage, object store, file exchange, telemetry, monitoring, and observability can describe nearby outcomes. The generated Pack projection normalizes Catalog categories and aliases for predictable lookup without turning those synonyms into independent readiness claims. Exact edition badges for primitives remain in the generated DSL reference.
Primitive and recipe guides
Section titled “Primitive and recipe guides”| Guide | Direction | Runtime primitive |
|---|---|---|
| Amazon S3 and compatible object storage | Input / Output | s3 |
| Azure Blob Storage | Input / Output | azure-blob |
| FileStore | Input / Output | file-store |
| Google Cloud Storage | Input / Output | gcs |
| HTTP endpoints | Input / Output | http-server, http-poll, http-post, http-get |
| Grafana JSON datasource | Visualization | HTTP delivery |
| Splunk Cloud or Enterprise | Output | splunk-hec |
The generated DSL reference is the source for exact component fields. These guides add task context and troubleshooting; they do not change the component contract.
Product API versus pipeline HTTP
Section titled “Product API versus pipeline HTTP”The HTTP integration components above move workload data into or out of a job. They are not the LyftData control-plane API used to manage the product. For product automation, use the Product API overview.
Where to go next
Section titled “Where to go next”- Start with storage recipes such as S3 or GCS when you are moving files.
- Use Credential Manager instead of placing production secrets in ordinary context values.
- Review delivery semantics before relying on retries, ordering, replay, or duplication behavior.
- For a connector not listed here, use the generated DSL reference to identify the underlying primitive, then confirm whether your Catalog contains a governed Provider Pack.