Service / Scale
Scaling & reliability
Hold your latency target at ten times the traffic, and know what it costs per request.
GPU workloads break the rules your platform was built on
Standard autoscaling assumes cheap, fast, stateless replicas. A GPU replica is none of those: it is expensive, slow to start, and holding gigabytes of weights and cache. Scale on CPU utilisation and you will scale on the wrong signal, too late, into capacity that is not available.
Scaling language models is queueing theory with an unusually harsh cost function. The work is knowing which signal to scale on, how deep the queue may get, what to shed when it gets deeper, and what all of it costs per request.
Capabilities
What this covers.
Capacity planning
Tokens per second per GPU for your model and your traffic shape, turned into a headcount of hardware and a bill you can forecast.
Autoscaling on the right signal
Queue depth and time-to-first-token rather than CPU, with scale-up that accounts for how long a GPU node actually takes to become useful.
Admission control
Bounded queues, priority classes and load shedding, so an overloaded system degrades predictably instead of timing out for everyone at once.
Latency budgets
An explicit split of time-to-first-token and inter-token latency across the request path, so tuning targets the part that is actually slow.
SLOs that mean something
Percentile targets tied to user-visible behaviour, with error budgets and alerts that fire on symptoms rather than on causes.
Multi-region and failover
Regional capacity, health-aware routing and tested failover, including what happens when a provider has a bad day.
Load testing
Realistic traffic with realistic context lengths and arrival patterns. Uniform synthetic load will lie to you flatteringly.
Token economics
Cost per request, per feature and per customer, so the growth curve and the margin curve are visible in the same room.
Failure engineering
Timeouts, retries with jitter, circuit breakers and idempotency, so a slow model does not turn into a retry storm.
Deliverables
What you're left with.
- A capacity model tying traffic growth to hardware and spend
- Autoscaling configured on signals that reflect the real bottleneck
- SLOs, dashboards and alerts your on-call will actually trust
- Load-test results at target and breaking point, with the breaking point named
- A failover plan that has been tested rather than written
Stack
What we work in.
- Kubernetes HPA / KEDA
- Ray
- Envoy
- Prometheus
- Grafana
- OpenTelemetry
- k6
- Terraform
Tools follow the problem. If your team already runs something that works, we would rather extend it than replace it.
Questions
Asked often.
What if our traffic is spiky and unpredictable?
Then the honest answer is often a hybrid: owned capacity for the floor, burst to an API above it. Provisioning peak-shaped GPU capacity for a workload that idles most of the day is how teams end up with a bill they cannot defend.
Can you help us pass a load test we already failed?
Yes, and that is a common starting point. The first step is finding the actual bottleneck, which is frequently not the model — it is a queue, a connection pool, or a retry loop.
What does 'reliable' mean for a probabilistic system?
Availability and latency are engineered the way they always were. Output quality is a separate matter, handled with evaluation and monitoring. Conflating the two is how teams end up with a green dashboard and unhappy users.