Known issues
This page lists known problems in the current LyftData 3.0 builds. We keep it up to date. An entry leaves this page only once its fix is in a published release and we have confirmed the fix on an installed copy of that release.
Each entry says what you may see, who is affected, how to avoid it, how to recover, and where it is fixed. Check your version with lyftdata --version.
Deployments and Workers
Section titled “Deployments and Workers”Undelivered objects are left behind when a deployment with a file-store or S3 edge is torn down
Section titled “Undelivered objects are left behind when a deployment with a file-store or S3 edge is torn down”What you may see: after you tear down a deployment, or retire it and create it again (for example to change how many readers an edge has), objects that were not yet delivered are never delivered. LyftData does not warn you.
Who is affected: deployments that hand data between workflow steps through a file-store or S3 edge.
How to avoid it: drain the edge before you tear the deployment down. See Tear down for the steps.
How to recover: there is no recovery yet. The objects stay in the edge’s store until you remove them.
Fixed in: fix planned for the first patch release after 3.0.0.
A producing Job cannot be removed when the Job it sends to is gone for good
Section titled “A producing Job cannot be removed when the Job it sends to is gone for good”What you may see: removing the producing Job is refused with 409 retarget_settlement_missing. From 3.0.0-beta.18, resetting the deployment reports that it is incomplete and keeps the Jobs. You can repeat the reset, but it does not complete.
Who is affected: deployments where a producing Job still holds undelivered records for a consumer Job that is gone for good, for example because the consumer’s Worker was deleted.
How to avoid it: before you delete a Worker for good, drain or reset the deployments that send data to it.
How to recover: if the consumer’s Worker is only offline, bring it back; the deliveries then complete and the producing Job can be removed. If the Worker has been deleted, there is no recovery yet.
Fixed in: fix planned for the first patch release after 3.0.0.
A deployment on a lost external Worker must be reset and created again
Section titled “A deployment on a lost external Worker must be reset and created again”What you may see: after you remove an external Worker whose host is gone, applying the same deployment again is refused. Removing a single Job from the lost Worker is refused while the Worker is still registered. From 3.0.0-beta.18, a deployment reset also cannot finish while that Worker is still registered, and the deployment cannot be applied again until you remove the Worker and repeat the reset.
Who is affected: deployments that still use an external Worker whose host will not return.
How to avoid it: once a host is lost, this cannot be avoided. When you retire a Worker on purpose, reset its deployments while its host is still running, then remove the Worker.
How to recover: if a producing Job sends data to a Job on the lost Worker, read A producing Job cannot be removed when the Job it sends to is gone for good before you start. Then:
- Remove the lost Worker from the Workers page, or with
POST /api/workers/remove/{id}. - Reset the deployment.
- If a static Worker group names the removed Worker, edit the group so it no longer names it.
- Create the deployment again and apply it on another Worker.
Data that the lost host had not sent is lost. A replacement Worker that enrolls automatically needs a new name, because auto-enrollment cannot reuse the removed Worker’s name.
Fixed in: fix planned for the first patch release after 3.0.0.
Process execution mode on the Built-In Worker can make Jobs fail
Section titled “Process execution mode on the Built-In Worker can make Jobs fail”What you may see: a Job on the Built-In Worker can fail when the Worker’s Job execution mode is set to Process. The error can include json-rpc frame too large followed by subprocess exited with code 1. It was seen with a Job that uses Variables and Secrets.
Who is affected: Jobs on the Built-In Worker when its Job execution mode is set to Process. Process mode is opt-in; Thread is the default.
How to avoid it: leave the Built-In Worker’s Job execution mode set to Thread, the default.
How to recover: on the Workers page, open the Built-In Worker, edit its settings, and set Job execution mode back to Thread. The Job recovers immediately, and no data is lost.
Fixed in: a fix that reports the cause of the failure is in testing for 3.0.0-beta.18.
On 3.0.0-beta.17, a stock Server cannot deploy a file-store or S3 edge
Section titled “On 3.0.0-beta.17, a stock Server cannot deploy a file-store or S3 edge”What you may see: planning a deployment with a file-store or S3 edge fails on a Server installed from the published archive, because the edge’s transport blueprints are not on the Server.
Who is affected: 3.0.0-beta.17 Servers.
How to avoid it: use a Worker Channel, or a broker through an output Job and an input Job, for hand-offs between steps.
How to recover: not needed. Nothing is deployed when the plan fails.
Fixed in: a fix is in testing for 3.0.0-beta.18.
A file-store or S3 edge keeps the same number of readers for the life of a deployment
Section titled “A file-store or S3 edge keeps the same number of readers for the life of a deployment”What you may see: changing the reader count of a running file-store or S3 edge is refused.
Who is affected: deployments that read a backlog of objects with several readers.
How to avoid it: choose the reader count when you create the deployment. To change it, drain the edge, retire the deployment, and create it again with the new count. See Tear down.
How to recover: not needed. Nothing is lost when the change is refused.
Fixed in: fix planned for 3.1.
Deployment readiness can show a working Job as blocked
Section titled “Deployment readiness can show a working Job as blocked”What you may see: the deployment screen reports a Job as blocked, and variable inspection, manual Job placement or a test run can return an error, while the deployed Job runs and delivers.
Who is affected: Jobs whose secret name comes from a context variable, as in the Catalog’s mail and Axiom stored-secret blueprints.
How to avoid it: nothing to change. Deploying and running are not affected.
How to recover: not needed. Check delivery on the Job itself rather than the readiness result.
Fixed in: fix planned for the first patch release after 3.0.0.
The Portal catalog cannot be synced on a standard installation
Section titled “The Portal catalog cannot be synced on a standard installation”What you may see: on a standard installation, Sync catalog from portal in the Catalog fails with an authentication error, even with an active license. Planning or deploying a deployment that uses a Portal catalog item fails with the same error. In 3.0.0, the Portal source is hidden on a standard installation.
Who is affected: standard installations that use, or want to use, items from the LyftData Portal catalog. Provider Packs and edge transports included with LyftData are not affected and work as usual.
How to avoid it: use the catalog items included with LyftData (Source → Local).
How to recover: there is no recovery for Portal catalog items yet. Build the deployment from items included with LyftData instead.
Fixed in: syncing the catalog from the Portal is planned as a licensed capability in a later release.
A test run can fail once with “database table is locked: variables”
Section titled “A test run can fail once with “database table is locked: variables””What you may see: a test run of a Job that expands a variable fetched during the run fails once with database table is locked: variables.
Who is affected: test runs only. Deployed Jobs are not affected.
How to avoid it: nothing to change.
How to recover: run the test again.
Fixed in: fix planned for the first patch release after 3.0.0.
Changing a context value fails while a deployed Job uses a feature the license no longer includes
Section titled “Changing a context value fails while a deployed Job uses a feature the license no longer includes”What you may see: saving a global, Worker or Job context value returns an error. The context value is saved before LyftData checks deployed Jobs, so if that check fails, the request returns an error even though the current value, and for a multi-value request possibly earlier values, has already been saved. Jobs after the unlicensed Job in the check may not pick up the change. The Server’s regular background check stops at the same Job and logs “Failed to scan for dirty jobs”.
Who is affected: Servers where a deployed Job uses a feature that the current license does not include, for example after a change to a license with fewer features. That Job keeps running.
How to avoid it: before you change to a license with fewer features, identify the deployed Jobs that use features the new license does not include, and plan a normal, supported stop or undeploy for them. Do not remove a running Job only as a workaround without confirming the impact on that workload.
How to recover: restore or activate a license that includes the feature the Job uses, then save the context change again. If you must undeploy the Job instead, follow your normal change procedure and confirm the workload can be stopped safely first.
Fixed in: fix planned for a release after 3.0.0.
Storage and telemetry
Section titled “Storage and telemetry”Cold-exported metrics can still count against the metric budget
Section titled “Cold-exported metrics can still count against the metric budget”What you may see: After LyftData moves older metrics to cold storage it can keep counting them against the metric budget. Once that budget is reached, new metric data is refused, although recent storage has room. Workaround: none.
Who is affected: Installations that move older metrics to cold storage.
How to avoid it: There is no workaround in 3.0.0-beta.18.
How to recover: There is no recovery path in the current build.
Fixed in: Not fixed in 3.0.0-beta.18; a later release has not been specified.
Before 3.0.0-beta.18, the Server does not start when its telemetry volume is nearly full
Section titled “Before 3.0.0-beta.18, the Server does not start when its telemetry volume is nearly full”What you may see: the Server exits at start with database or disk is full.
Who is affected: Servers with less free space than the telemetry disk floor on the volume that holds the Server’s telemetry stores, by default its staging directory. By default the floor is the smaller of 10% of that volume and 10 GiB. This is known for a newly installed Server. An existing installation may also be affected if a pending migration needs to enlarge a store.
How to avoid it: keep free space on that volume above the floor.
How to recover: free up space on the volume, or lower the floor with --telemetry-disk-floor (LYFTDATA_TELEMETRY_DISK_FLOOR), then start the Server again.
Fixed in: fix planned for 3.0.0-beta.18.
The observability store grows faster than it should
Section titled “The observability store grows faster than it should”What you may see: on a default installation, the store that holds the Server’s metrics, logs and traces takes more disk than expected, even on a small installation.
Who is affected: default installations.
How to avoid it: watch free space on the volume that holds the Server’s telemetry stores (by default its staging directory) and give it headroom.
How to recover: add disk to the volume. Do not delete store files by hand.
Fixed in: fix planned for the first patch release after 3.0.0.
Per-step throughput reads too low after a late Worker report
Section titled “Per-step throughput reads too low after a late Worker report”What you may see: if a Worker reports step metrics more than 24 hours late, for example after a day offline, the per-step throughput figures for that period read too low. This shows in the home dashboard’s throughput charts, the deployment flow overlay’s per-step figures, the database explorer’s Jobs target, and Operator, MCP and CLI answers about Worker metrics.
Who is affected: Workers that report step metrics more than 24 hours late. Stored data is intact, and license volume and usage reporting are not affected.
How to avoid it: keep Workers connected to the Server.
How to recover: there is no recovery yet for the affected period’s figures.
Fixed in: fix planned for the first patch release after 3.0.0.
Backup and recovery
Section titled “Backup and recovery”A backup restored on a rebuilt host needs the same service account uid
Section titled “A backup restored on a rebuilt host needs the same service account uid”What you may see: a backup restored onto a rebuilt host does not start, and the Server reports that the persisted runtime identity does not match the current runtime account. On the Linux install guide’s layouts, lyftdata secret-storage status, export and restore do not run, so you cannot make or restore a key recovery bundle there.
Who is affected: Linux installations restored onto a new or rebuilt host.
How to avoid it: record the service account’s numeric uid and the staging path, so you can recreate them.
How to recover: restore from the state backup with the service account recreated with its original numeric uid, at the same staging path. See Backup & Recovery.
Fixed in: fix planned for the first patch release after 3.0.0.
Install, licenses and editions
Section titled “Install, licenses and editions”3.0 beta builds ship no Docker image
Section titled “3.0 beta builds ship no Docker image”What you may see: beta builds ship no Docker image. The version folder of a current 3.0 beta, such as 3.0.0-beta.19, has no lyftdata-server-docker-*.tar.xz archive, so the Docker instructions cannot be followed for that beta. The earlier beta.13 and beta.15 images described on the Docker page were one-off exceptions.
Who is affected: anyone who wants to run a 3.0 beta in Docker or Docker Compose.
How to avoid it: run beta builds with the native Linux server install. For Docker, use an EAP or stable release once its image is published.
How to recover: nothing needs recovering. Install the beta natively, or move to an EAP or stable image when it is available.
Fixed in: 3.0.0-eap.1, the first build with a Docker image; stable releases also ship one. Betas stay without an image.
Changing plan with an activation token fails when the Server already runs a plan
Section titled “Changing plan with an activation token fails when the Server already runs a plan”What you may see: in Settings → License → Activate license, with Subscription (Recommended) selected, you paste the Portal token for your new plan and the Server shows “The activation token could not be redeemed. It may have expired or been consumed; return to your Portal account for help”. In this case the token has not expired or been used up, and you can still use it.
Who is affected: Servers that already run a plan, when you activate a different plan bought in the LyftData Portal.
How to avoid it: activate the new plan with its license file instead of the token, as below.
How to recover:
- In the Portal, open Account → Licenses & installations and, on the new plan, select Download license.
- On the Server, open Settings → License → Activate license → Advanced: Full License Key. Use Load from file, or paste the key into License key, then activate.
- Only after the new license is active on the Server, stop the earlier plan’s billing. In the Portal’s Billing page, select Manage billing on the earlier plan and cancel it there. Cancelling stops future renewals of the earlier plan. Until you cancel it, the earlier plan keeps billing.
If downloading or cancelling is not available to you, contact LyftData support from your Portal account.
Fixed in: fix planned for a later hosted Portal release.
A fresh install of exactly 3.0.0 is refused after 90 days
Section titled “A fresh install of exactly 3.0.0 is refused after 90 days”What you may see: a first-time install of exactly 3.0.0 on a host where it was never installed is refused once the first signed generation of its release metadata has expired, 90 days after release.
Who is affected: first-time installs of exactly 3.0.0. Installations that are already running are not stopped.
How to avoid it: install a newer release.
How to recover: install a newer release.
Fixed in: this is how 3.0.0 is signed. Install a newer release instead.
On Community Edition, lyftdata deployments list answers “404 Not Found”
Section titled “On Community Edition, lyftdata deployments list answers “404 Not Found””What you may see: on a Community Server with no license, lyftdata deployments add works, but lyftdata deployments list fails with a bare 404 Not Found. The message says nothing about the edition.
Who is affected: Community Edition Servers.
How to avoid it: there is no workaround yet.
How to recover: not needed. Nothing is lost.
Fixed in: fix planned for the first patch release after 3.0.0.
Audit events are not forwarded until the Server restarts after license activation
Section titled “Audit events are not forwarded until the Server restarts after license activation”What you may see: you activate a Standard, Enterprise, or Enterprise evaluation license on a running Server, and audit events are still not forwarded; they stay in the local Server log. Audit event forwarding is switched on or off when the Server starts, so it follows the license that was in force at the last start. If that license did not include audit forwarding, the Server log from that start shows “Audit event forwarding is not included in this license”.
Who is affected: Servers that activate or change a license while running. A new Standard, Enterprise, or Enterprise evaluation install that activates its license on a running Server forwards no audit events until the Server restarts. If the Server is still running with audit forwarding active, renewing that license does not require a restart. If the Server was restarted while the license was expired or did not grant audit forwarding, restart it after renewal.
How to avoid it: restart the Server after activating or changing a license for audit forwarding to follow it, unless you are renewing a license on a Server that is still running with audit forwarding active.
How to recover: restart the Server. Audit events from before the restart remain in the local Server log.
Fixed in: fix planned for a release after 3.0.0.
The license API shows Worker and volume limits that the license does not have
Section titled “The license API shows Worker and volume limits that the license does not have”What you may see: on a license with no Worker limit or no daily volume limit, the licensed_workers and licensed_volume fields of the Server’s license details (GET /api/license/show) still show a number, such as a daily volume of 1 TB. Tools that read those two fields show the same figures.
Who is affected: Servers on a license that sets no Worker limit or no daily volume limit. The license page in the Server’s UI shows the correct limits.
How to avoid it: read the capacity field of the same response, or the license page in the UI. Both show the limits your license sets. Worker admission follows your license, not these two fields.
How to recover: not needed.
Fixed in: fix planned for a release after 3.0.0.