Service / Optimisation
Optimisation
Make the same model cheaper and faster, and prove the quality survived it.
Every speed-up spends quality — the question is how much, and where
Optimisation is a series of trades. Four-bit weights buy memory with accuracy. A distilled student buys latency with generality. Speculative decoding buys tokens per second with complexity. None of these are free, and most teams take the trade blind because they have no way to see the cost.
So the evaluation harness comes first. Once you can measure quality on your own task, every other lever becomes a decision with a number attached instead of a leap of faith.
Capabilities
What this covers.
Evaluation harness
A task-specific quality bar, built first, that your team can re-run in CI. Without it, everything below is guesswork.
Quantisation
GPTQ, AWQ, FP8 and INT4 weight and activation schemes, applied per model and verified against the bar rather than assumed safe.
Distillation
A smaller student trained on your traffic to carry the narrow task at a fraction of the cost, where the task is narrow enough to allow it.
Adapters and fine-tuning
LoRA and QLoRA where a prompt has stopped being enough — plus the judgement to say when it has not.
Speculative decoding
Draft-model and n-gram speculation for real tokens-per-second gains on latency-bound interactive work.
KV cache engineering
Paged attention, prefix reuse and cache-aware routing, so shared context is computed once instead of per request.
Prompt and context reduction
The cheapest optimisation available and the most often skipped: shorter contexts, fewer round-trips, less retrieved padding.
Batching and scheduling
Continuous batching tuned to your latency target, trading tail latency for throughput deliberately rather than by default.
Kernel and compiler work
TensorRT-LLM engines, torch.compile and CUDA graphs when the remaining cost is genuinely in the kernels.
Deliverables
What you're left with.
- An evaluation harness for your task, running in your CI
- A before-and-after report: quality, latency, throughput and cost per token
- The optimised model or serving configuration, reproducible from your repo
- A written record of every trade taken and every one rejected, with reasons
- The regression tests that stop the gains eroding next quarter
Stack
What we work in.
- vLLM
- SGLang
- TensorRT-LLM
- PyTorch
- GPTQ / AWQ
- PEFT
- Optimum
- Weights & Biases
Tools follow the problem. If your team already runs something that works, we would rather extend it than replace it.
Questions
Asked often.
How much can we expect to save?
We will not put a number on it before measuring your workload — anyone who does is guessing. Teams that have never optimised usually have significant headroom in caching, context length and batching before anything exotic is needed. The first week is measurement, and it sometimes ends with a report saying the model is already close to the floor.
Will quantisation hurt quality?
Yes, measurably, and usually less than people fear — but how much depends on the model and the task. That is the entire point of building the bar before touching the weights.
We have no evals. Is that a problem?
It is common, and it is where we start. Building the harness is usually the highest-leverage week of the engagement, and it outlives it.