What on-premise LLM deployment requires

The short answer

Running an LLM on your own hardware is not one decision. It is four decisions that are easy to confuse:

1. Can the model fit? Weights plus KV cache plus runtime overhead must fit in memory, with headroom for concurrency.
2. Is it fast enough? Throughput and p95 latency on the target hardware, at the context length you actually use.
3. Is the boundary real? Network egress, authentication, audit logging and model supply chain.
4. Who operates it? Updates, monitoring, capacity, fallback and the cost of an idle GPU.

Most failed on-premise projects get the first two roughly right and underestimate the last two. A running endpoint is not a deployment. A deployment is something your team can restart, upgrade, monitor and explain to a security reviewer.

The decision: API, private cloud or on-premise

The honest version of the on-premise case is usually one of two things: the data legally cannot leave, or you already own the hardware and it is underused. "It will be cheaper" is sometimes true, but it is a utilisation argument, not a hardware argument.

Run the numbers on cost per successful request, not cost per GPU hour. Include the people time for updates, monitoring and incident response. A GPU that runs at 5% utilisation is not cheaper than an API; it is a capital expense with a support ticket attached.

How do you size the hardware?

Parameter count is the starting point, not the answer. Three things consume memory.

1. Model weights

weights memory ≈ parameters × bytes per parameter

These are decimal gigabytes and exclude scales, embeddings and runtime buffers. Add 10 to 20 percent headroom.

2. KV cache

The KV cache stores the attention keys and values for every token in context. It grows with context length and with the number of concurrent sequences.

KV bytes ≈ 2 × layers × KV heads × head dimension × sequence length × batch × bytes per value

For an 8B model with 32 layers, 8 KV heads, head dimension 128, FP16, 8,192 tokens and a batch of 8:

• 2 × 32 × 8 × 128 × 8,192 × 8 × 2 bytes ≈ 8 GB

That is before the weights. Add 16 GB of FP16 weights and roughly 2 GB of runtime overhead, and the model needs about 26 GB. It fits on a 32 GB card only with little room for a larger batch; a 48 GB card is comfortable. At 4-bit weights, the same setup lands around 15 GB and fits on a 24 GB card, if the quality holds.

Grouped-query attention reduces the KV cache substantially compared with full multi-head attention. That is one reason two models with the same parameter count can have very different serving costs.

3. Activation and runtime overhead

Prefill of a long prompt creates temporary activations. The serving stack, CUDA context, tokenizer and batching buffers all take memory. Do not size to the last gigabyte.

Quantization is a quality trade, not a free win

Quantization reduces the number of bits used to store weights and, in some schemes, activations. It is the main lever for fitting a larger model on smaller hardware.

• FP16 or BF16 is the reference. Use it when memory allows.
• INT8 is often close to FP16 on many tasks, but not all. Measure.
• 4-bit (GPTQ, AWQ, GGUF variants) makes consumer hardware viable. Quality loss depends on the task, the calibration data and the layers that are quantized.

The mistake is to quantize, run a few prompts, and declare it fine. Quantization can degrade exactly the behaviour that makes the system useful: strict JSON, numeric reasoning, rare vocabulary, long-context recall. Put the quantized model through the same frozen evaluation set as the full-precision model and compare per category.

A practical rule: if the 4-bit model loses more than the task can tolerate, use a smaller model at higher precision, or a bigger GPU. Do not hide the regression behind a better demo prompt.

Which serving stack should you use?

There is no single correct stack. The right one depends on hardware, concurrency and how much operational complexity you want.

TensorRT-LLM and Triton are areas where the NVIDIA certification covers the concepts, not a claim that I have run them in production. If your workload needs deep TensorRT engine work, say so early and we scope it as a specialist task. The same applies to CUDA kernel work.

What does the security boundary include?

An on-premise deployment moves the trust boundary. It does not remove it.

• Network egress. Prove what the process can reach. A model that phones home for telemetry is not air-gapped.
• Authentication and authorisation. Who can call the endpoint, from where, and with what rate limit?
• Audit logging. Prompts and completions may contain regulated data. Decide what is logged, where it is stored, who can read it and how long it is retained.
• Secrets and model supply chain. Verify weight checksums, pin container images, and control who can replace a model file.
• Multi-tenant isolation. If one deployment serves several customers, the cache, logs and retrieval index need isolation, not just separate API keys.
• Prompt injection. A private model is still vulnerable to instructions hidden in retrieved documents. The boundary protects data; it does not make the model obedient.

The output of this work is a short document: what leaves the network, what is logged, who can reach the endpoint and what happens on failure. If that document cannot be written, the deployment is not ready for a security review.

Operations: the part that decides whether it survives

A model endpoint has a lifecycle. Plan for it before launch.

• Updates. Model weights, serving stack and CUDA drivers all change. A pinned version is a decision, not an accident.
• Monitoring. Track p50 and p95 latency, throughput, error rate, GPU memory, queue depth and refusal rate. A model that silently starts refusing is a production incident.
• Capacity. Concurrency is not linear. Measure at the batch size you expect, with the context length you actually see.
• Fallback. Decide what happens when the GPU is saturated or the model fails. A queue with a timeout is better than an infinite spinner.
• Cost. Include idle time, power, hardware refresh and the engineer who owns it.
• Evaluation. Keep the frozen test set. Re-run it after every model, driver or serving change.

A four-week pilot shape

The checkpoint at the end of week 2 is deliberate. If the quality or the hardware numbers do not work, stop before the integration cost is spent.

What I have actually done

I have deployed NVIDIA AIQ on NVIDIA DGX Spark hardware and adapted it with tools and agent capabilities. I have also built the AI and application layer of a HIPAA-compliant assistant inside a controlled Azure environment, including the technical and security discussions with the customer.

I have not operated a 24/7 multi-site inference fleet, and I do not claim production TensorRT engineering. If your project needs either, I will tell you before the proposal, not after the invoice.

If you are sizing a private deployment, the AI integration service covers the scope and the deliverables. If the harder question is whether the model is good enough, a workflow audit comes first.