Placement and Scaling

Placement selects eligible worker groups. Replicas express desired concurrency. Neither control proves that an input can be consumed concurrently or recover on another worker.

Place by runtime requirement

Use worker groups to express real constraints: network reachability, data residency, connector capability, credentials, storage visibility, resource class, or failure domain. Before plan/apply, confirm the group has live members and each member can meet the stage’s requirements.

Placement remains sticky after apply unless an operator re-plans and applies a change or explicitly requests placement recomputation. This avoids silent movement, but it is not a high-availability guarantee.

Co-locate local state

Keep jobs together when they share a worker channel, local file path, in-memory state, local cursor, or worker-local KV contract. Split them only after moving the boundary to shared/durable storage and proving both groups can access it.

Safe replica checklist

Before increasing replicas, answer every row:

Boundary	Required evidence
Input ownership	Broker/connector contract for partition or record assignment
State	Local/shared/reconstructable state and checkpoint location
Ordering	Required ordering scope and how concurrency preserves it
Delivery	Ack, retry, duplicate, and loss behavior
Capacity	Eligible workers and downstream capacity
Connectivity	Source, destination, credential, and storage access per worker
Recovery	Restart, reassignment, and scale-in/drain procedure
Proof	Baseline and post-change counts, latency, errors, and receipts

If state, ordering, delivery, or ownership is unknown, stop. Do not turn an unknown into a safe-scale recommendation.

flowchart TD
  accTitle: Evidence gate for a replica increase
  accDescr: Capture a baseline, prove input ownership, understand state ordering delivery and recovery, and verify worker and downstream capacity before planning and applying a small replica increase. If any gate fails, stop and resolve the gap. After apply, verify convergence and compare evidence with the baseline.
  baseline["Capture healthy baseline"] --> ownership{"Input ownership proven?"}
  ownership -- No --> stop["Stop and resolve the evidence gap"]
  ownership -- Yes --> semantics{"State, ordering, delivery, and recovery understood?"}
  semantics -- No --> stop
  semantics -- Yes --> capacity{"Worker connectivity and downstream capacity verified?"}
  capacity -- No --> stop
  capacity -- Yes --> plan["Plan smallest replica increase"]
  plan --> apply["Review and apply fresh plan"]
  apply --> converge["Verify convergence"]
  converge --> compare["Compare counts, latency, errors, and receipts"]

Scale out

1. Capture a healthy throughput, latency, backlog, error, and resource baseline.
2. Confirm input ownership and downstream capacity.
3. Plan the smallest replica increase for the affected runtime group.
4. Review generated assignments and warnings.
5. Apply the fresh plan.
6. Observe readiness and compare conservation, duplicates, ordering, backlog, latency, and external receipts with the baseline.

Adding workers without changing eligible placement/replicas may change no running workload. Increasing replicas without an ownership contract can create duplicates or competing readers.

Scale in

Drain or reassign input ownership according to the source contract, wait for in-flight work and checkpoints, reduce replicas through plan/apply, and confirm the removed worker owns no remaining assignment. Watch for replay, duplicate, and ordering effects. Never terminate capacity solely because its CPU is low.

Failure and failover

A connected spare worker is capacity, not automatic failover. Recovery also needs eligible placement, reachable sources/destinations, credentials, shared or reconstructable state, correct ownership reassignment, and a reconciler path proven for the topology. Use Worker Recovery and Serviceability.

Evidence after a change

Separate plan freshness, apply acceptance, desired/observed convergence, worker readiness, runtime counts, and destination acceptance. See Runtime Evidence and Receipts and Pipeline Integrity.