Service / Deployment
Deployment & serving
Get the model into production on a path you can repeat, audit and roll back.
A notebook is not a deployment
The distance between a model that works and a model in production is mostly unglamorous: GPU scheduling, image sizes, cold starts, health checks, weight loading, version pinning, and a way back when the new checkpoint is worse than the old one.
Teams that skip it ship once, successfully, and then cannot ship again without a person who remembers how. The work here is making the second, tenth and hundredth release boring.
Capabilities
What this covers.
Serving stack selection
vLLM, SGLang, TGI, TensorRT-LLM or Triton, chosen against your model, latency target and operational appetite — with the reasoning written down.
GPU scheduling
Kubernetes device plugins, MIG partitioning, time-slicing and node pools, so expensive silicon is not sitting idle in the wrong namespace.
Model registry
Versioned, immutable weights with provenance, so the artefact in production is one you can name and reproduce.
Cold start engineering
Weight streaming, image slimming and warm pools, because a five-minute cold start makes autoscaling a fiction.
Release mechanics
Blue/green and canary releases with automated rollback on quality or latency regression — not just on HTTP 500s.
Shadow traffic
New models evaluated against live production traffic before they serve a single user.
Infrastructure as code
The whole path in Terraform and your CI, reviewable and repeatable, with no console clicks in the critical path.
Health and readiness
Probes that reflect whether the model can actually serve, rather than whether the process is running.
Multi-model hosting
Adapter hot-swapping and multi-tenant serving, so ten fine-tunes do not become ten idle clusters.
Deliverables
What you're left with.
- A serving stack running in your cloud, defined in your IaC
- A CI/CD pipeline from model artefact to production with gates and rollback
- A registry with versioning, provenance and a promotion path
- Load-test results at your expected and peak traffic
- Runbooks for the failures we can predict, and an on-call handover
Stack
What we work in.
- vLLM
- SGLang
- TensorRT-LLM
- Triton
- Ray Serve
- Kubernetes
- Terraform
- Argo
Tools follow the problem. If your team already runs something that works, we would rather extend it than replace it.
Questions
Asked often.
Do we need Kubernetes for this?
Often not. A managed endpoint or a couple of well-configured VMs serves a great many production workloads, and costs far less to operate. We will recommend Kubernetes when your scale or your platform standards actually call for it.
Can you work with our existing platform team?
Preferably. The best outcome is your team owning this fully, with us as the people who set it up and then left useful documentation behind.
How long does a first deployment take?
It depends on what exists already. A greenfield deployment onto an established platform is quick; one that has to wait on procurement, GPU quota and a security review is not. We will give you an honest estimate after the diagnostic, not before.