Project 42Start learning
Module 4 of 12 · 58 min

Hardware, Runtime, and Capacity Planning

Select a compatible runtime and size compute, accelerator memory, host memory, storage, and concurrency from measurements of the exact serving build.

Core concept

By the end

You will be able to

  • Build a compatibility matrix across artifact format, model architecture, runtime, accelerator, drivers, operating system, and deployment target.
  • Estimate static model memory, dynamic cache and workspace memory, host memory, storage, bandwidth, and startup needs without treating an estimate as a benchmark.
  • Measure latency, throughput, concurrency, saturation, errors, and resource headroom with representative request distributions.
  • Produce a capacity decision with resource limits, admission control, overload behavior, cost assumptions, and re-benchmark triggers.
01

Prove compatibility across the complete stack

Record the model architecture, artifact and tensor formats, precision or quantization, tokenizer, template, maximum tested context, runtime and version, execution backend, operating system, CPU architecture, accelerator model, accelerator count, driver, compute library, container base, and required extensions. Verify support from current runtime and hardware documentation and then load the exact artifact in quarantine.

A runtime may support an architecture but not a specific conversion, quantization, kernel, device, platform, or feature combination. Automatic fallback can silently move work to a slower backend or change behavior. Make fallback visible, measured, and either accepted or rejected.

02

Estimate memory and storage before measuring

Begin with the exact artifact sizes and a rough weight-memory estimate from parameter count and bits per parameter, then add quantization metadata, embeddings, runtime workspace, graph or kernel caches, temporary buffers, loaded replicas, and safety headroom. Dynamic memory also depends on concurrent sequences, context and output lengths, cache representation, batching, and runtime behavior.

Include host memory for the runtime, tokenizer, request queues, observability, page cache, staging, and failure handling. Include local and remote storage for immutable artifacts, container layers, caches, logs, evaluation data, backups, and rollback bundles. Treat every calculation as a hypothesis to validate on the intended build.

03

Measure a representative request distribution

Version a load profile containing warm and cold starts, input and output token distributions, modalities, concurrency steps, bursts, cancellations, timeouts, malformed requests, maximum allowed context, and recovery after pressure. Measure time to ready, time to first output, inter-output latency where applicable, end-to-end latency percentiles, throughput, queue time, error and rejection rates, accelerator and host memory, compute utilization, storage and network behavior, power where relevant, and restart time.

Hold the artifact, runtime, configuration, hardware, prompt template, and cases constant when comparing builds. Published benchmark results can help frame methods and candidate hardware, but they do not replace measurements of the exact workload, software stack, quality target, and service policy.

04

Find saturation and design overload behavior

Increase concurrency and request size gradually until latency, queueing, memory, errors, power, or quality crosses a declared boundary. Record the first limiting resource and the failure mode. Repeat after restarts and with neighboring workloads if resources are shared.

Set container or scheduler requests and limits from evidence, leaving headroom for the host and recovery. Docker documents that containers have no resource constraints by default, and memory pressure can trigger out-of-memory termination. Admission control, bounded queues, maximum context and output limits, rate or concurrency limits, timeouts, load shedding, and clear retry behavior are part of capacity—not afterthoughts.

05

Issue a capacity and cost decision with expiry

State supported workload slices, tested concurrency, latency and throughput distributions, quality gates, resource reservations and limits, minimum headroom, overload behavior, replica assumptions, startup and recovery time, storage, power or cloud cost assumptions, monitoring thresholds, and excluded cases. Separate measured facts from extrapolation.

Re-benchmark after changing model revision, quantization, runtime, driver, accelerator, host, operating system, container, context policy, batching, template, feature flags, traffic distribution, telemetry, or service objective. Capacity evidence expires when the workload or stack it measured no longer matches production.

Practice activity

Build a measured-paper capacity plan

  1. Create a synthetic exact-stack compatibility matrix and mark every field as documented, measured, inferred, or unknown.
  2. Estimate model, dynamic cache, workspace, host, storage, and rollback capacity for three context and concurrency scenarios.
  3. Design a versioned load profile with warm-up, steady state, burst, maximum context, overload, cancellation, restart, and recovery cases.
  4. Declare pass thresholds and stop conditions before reviewing synthetic results, then identify the first saturated resource and safe admission policy.
  5. Write a capacity decision with supported slices, limits, headroom, costs, overload behavior, excluded cases, and re-benchmark triggers.

What to produce

  • An exact-stack compatibility matrix and calculation sheet with uncertainty labels.
  • A versioned load profile, metric contract, threshold table, and synthetic saturation results.
  • A dated capacity decision with resource and admission controls, recovery evidence, costs, exclusions, and expiry triggers.

Reflect before continuing

Which capacity assumption would cause the most harm if it were wrong, and what measurement resolves it?

Evidence

Sources and verification

Knowledge check

Make it stick.

Pass at 80%

Choose the strongest answer for each question. Your attempts become part of your device-local transcript.

01What belongs in an exact-stack compatibility record?
02Why is weight memory alone insufficient for sizing?
03Which load profile is most useful?
04What should happen as a service approaches saturation?
05What does a published benchmark establish for your endpoint?
06When should capacity be re-benchmarked?