by 
Every week, another organisation announces it’s deploying a large language model. And every week, a Technical Architect or Pre-sales Engineer gets asked a version of the same question: “How much infrastructure do I actually need for this?”
In my days as a Data Centre Architect and Engineer, I’d size server clusters for databases and VMware environments. The maths was different, but the discipline was the same: understand the workload, match it to the hardware, justify the recommendation. Now the question I get asked is “How do I size for LLMs?” This blog series is all about answering that.
Before you can answer that — before GPUs, nodes, interconnects, or platform choices even enter the conversation — you need two concepts nailed down cold: tokens and parameters. They’re the two dials that drive every LLM server sizing decision you’ll ever make.
Think of it this way. Parameters tell you how big the engine is. Tokens tell you how hard you’re asking it to work. Get those two right, and the rest of the sizing conversation falls into place.
Tokens: The Currency of Language Models
LLMs don’t read sentences the way you do. They don’t even read words. They read tokens — small chunks of text that sit somewhere between a syllable and a word.
- Sometimes a token is a whole word: server
- Sometimes it’s a fragment: serv, er
- Sometimes it’s punctuation or whitespace: . , ,
For English text, a useful rule of thumb is:
1 token ≈ 3–4 characters, or roughly 0.75 of a word
So the sentence “This is a sizing test.” runs to about 6–7 tokens — not 5, because the model doesn’t count words.
When you see pricing or performance metrics quoted in the market, they’re always denominated in tokens:
- $X per 1,000 tokens
- Y tokens per second
- 4k / 8k / 32k / 128k context window
That last one matters a lot. The context window is the maximum amount of text — measured in tokens — the model can hold in view at once. It’s not just the question you asked; it includes everything: system instructions, conversation history, documents you’ve fed in, and the response being generated. Every token in that window costs compute and memory.
Why Tokens Drive Sizing
Tokens show up in three places in every sizing conversation:
1. Context length (the prompt window) Longer context means the model has to track more information simultaneously. That translates directly into more VRAM for the KV cache — the memory structure the model uses to keep track of what it’s already processed. A customer who wants 128k-token context windows needs significantly more memory per request than one running at 4k.
2. Throughput and concurrency Tokens per second is the fundamental throughput metric — per GPU, per node, per cluster. In practice, you’ll often work backwards from a customer’s requirements:
“We need to support 500 concurrent users, each generating responses of around 300 tokens, within 3 seconds.”
That’s a tokens-per-second and concurrency problem. Everything else follows from it.
3. Capacity and cost planning Whether on-premises or cloud, consumption is effectively input tokens + output tokens. On a Dell PowerEdge server deployment, higher sustained tokens per second means more GPU compute, more memory bandwidth, and — beyond a certain point — more nodes or a move to higher-end accelerators.
Parameters: The Size of the Brain
If tokens are the currency of language models, parameters are what you’re buying with your hardware budget.
A parameter is a learned numeric weight — a floating-point number — stored inside the model. Mathematically, an LLM is an enormous function, and parameters are the numbers that define it. When a model is trained, billions of these weights are adjusted, incrementally, until the model gets reliably good at predicting language.
This is why model names look the way they do:
- 7B → approximately 7 billion parameters
- 13B, 34B, 70B, 405B → and so on up the scale
More parameters generally mean greater model capability — the model can represent more complex patterns, handle more nuanced reasoning, and produce higher-quality output. But that capability comes at a direct hardware cost, because every parameter has to live somewhere.
The VRAM Equation
The first-order estimate for model memory is straightforward:
Model VRAM (GB) ≈ Parameters × Bytes per Parameter
In practice:
| Model | Precision | Weights-only VRAM |
|---|---|---|
| 7B | FP16/BF16 (2 bytes) | ~14 GB |
| 70B | FP16/BF16 (2 bytes) | ~140 GB |
That’s just for the weights themselves. In a real deployment you also need memory for:
- KV cache — grows with context length and batch size
- Activation memory — the working memory during computation
- Optimizer states — relevant if you’re fine-tuning, not just inferencing
- Runtime overhead — fragmentation, safety layers, serving framework
The practical consequence is clear:
- A 7B model can typically run on a single high-memory GPU (24–80 GB class, depending on precision and context requirements).
- A 70B model generally needs multiple high-VRAM GPUs — and demands fast interconnects between them, whether that’s NVLink, NVSwitch, or a high-bandwidth PCIe fabric.
As a pre-sales engineer, parameter count is what you’ll map to platform choices: how many GPUs per node, whether you need a GPU-dense platform with a high-speed fabric, and whether the workload fits in a 2U form factor or needs something more substantial.
How the Two Interact
Here’s the mental model worth keeping front of mind for every sizing conversation:
| What it measures | What it drives | |
|---|---|---|
| Parameters | How big the model is | VRAM requirement, compute per token, hardware footprint |
| Tokens | How much work you’re asking it to do | Throughput, latency, concurrency, context memory |
Given a fixed GPU budget, customers are always navigating a trade-off:
- Bigger model (more parameters) versus more throughput (more tokens per second)
- Longer context windows (more tokens per request) versus more concurrent requests
There’s no universally right answer — it depends on the use case. A legal document analysis platform that processes 100k-token contracts needs a very different configuration from a customer service chatbot handling hundreds of short, concurrent sessions.
Turning This Into a Sizing Conversation
When you strip away the jargon, most customer LLM questions reduce to this:
“Given a model size (parameters) and an expected usage pattern (tokens), how many GPUs and servers do I need to hit my latency and concurrency targets?”
The discovery questions that unlock that answer fall into two groups:
Model side:
- Are you targeting a 7B, 13B, 70B, or larger class model?
- Are you planning full precision, mixed precision, or quantized deployment?
- Is this inference only, or do you also plan fine-tuning?
Usage side:
- What’s the average prompt size per request (in tokens)?
- What’s the expected response length?
- What’s the maximum context length required — 8k, 32k, 128k?
- How many concurrent users do you need to support, and at what latency?
Once you have those answers, you can map them to Dell platforms — GB10, GB300, PowerEdge XE and XE+ GPU servers, interconnect choices, cluster configurations — in a structured and defensible way. That’s exactly what we’ll build up in the posts that follow.
Next up: Part 2 — from tokens per second to GPU count: the maths that drive inference sizing.
excellent material. 10x