Scaling, Failure, and Capacity Controls

Scale the measured bottleneck, not the diagram

Use latency distributions, queue time, utilization, memory headroom, bandwidth, cache behavior, request sizes, concurrency, rejection, and cost to identify the limiting resource. Decide whether to use a larger device or host, more replicas, model or tensor parallelism, request batching, a queue, caching, routing by workload, or a smaller evaluated model.

Each option changes failure domains, startup, consistency, quality, latency, utilization, cost, and operating complexity. Scaling compute cannot fix an unbounded queue, slow artifact store, identity outage, bad prompt template, incompatible runtime, or quality regression.

Control admission and backpressure

Set request body, context, output, concurrency, rate, queue, deadline, and resource limits by workload and priority. Reject early with stable retry guidance when work cannot complete inside its deadline or budget. Propagate cancellation and deadlines through gateway, queue, server, runtime, and dependent tools.

Use bounded retries with jitter only for classified transient failures and only when the request is safe to repeat. Apply circuit breaking and load shedding before saturation cascades across replicas or dependencies. Define degraded modes that remain safe, such as reduced limits, lower-priority rejection, or routing to a separately evaluated fallback.

Place, warm, drain, and replace capacity deliberately

Schedule against real device, memory, topology, storage, network, and driver requirements. Kubernetes resource requests influence placement and limits bound use; extended devices cannot be overcommitted in the same way as fractional CPU. Verify the actual scheduler, device plugin, topology, and runtime behavior in the target environment.

Account for artifact download, verification, model load, compilation, cache warm-up, and readiness before adding a replica to service. On removal or update, stop new work, drain bounded in-flight requests, preserve telemetry, and revoke the route before reclaiming resources.

Test shared-resource and failure isolation

Inject process crash, out-of-memory termination, device loss, host loss, storage delay, network partition, identity failure, telemetry failure, registry outage, queue pressure, malformed work, and a bad model release. Observe which requests, replicas, nodes, sites, and management functions are affected.

Confirm that health and readiness remove unsafe capacity, retries do not amplify load, queues remain bounded, one workload cannot starve another, exact model identity is preserved, and recovery does not accept stale or unverified artifacts. Multiple replicas are useful only when their shared dependencies and placement match the availability claim.

Gate autoscaling and scale changes

Choose scale signals that lead demand without reacting to noise: queue delay, concurrency, work units, saturation, or a tested combination. CPU or accelerator utilization alone may lag queue growth or encourage thrashing. Set minimum safe capacity, maximum cost and quota, stabilization windows, warm-up allowance, scale-down drain, and alerting.

Compare scale policy against representative steady, burst, ramp, skewed, and failure traffic. Approve only when objectives, quality, resource isolation, cost, startup, recovery, and rollback gates pass. Preserve the prior policy and capacity floor for rapid reversal.

Practice activity

Design and challenge a scaling policy

Use a synthetic load result to identify one measured bottleneck and compare vertical, replica, sharded, batched, queued, cached, routed, and smaller-model options.
Define per-workload admission, queue, deadline, retry, cancellation, circuit, shedding, and degraded-mode behavior.
Write placement, artifact warm-up, readiness, draining, replacement, minimum capacity, maximum cost, and stabilization rules.
Create ten process, device, host, dependency, pressure, and bad-release failure cases with expected blast radius and recovery.
Compare the candidate scale policy to a baseline under steady, burst, ramp, skewed, and failure traffic, then issue a reversible decision.

What to produce

A bottleneck-to-pattern comparison with quality, latency, failure, cost, and complexity impacts.
An admission, backpressure, placement, warm-up, drain, autoscale, and degraded-mode policy.
A failure-isolation report and baseline-candidate scale decision with objectives, cost, recovery, and rollback gates.

Reflect before continuing

Which autoscaling signal would arrive too late for your workload, and what leading signal should accompany it?