In Part 1 we established the two fundamentals: parameters define how big the model is, and tokens define how much work you’re asking it to do. Now we make it practical.

This post is about the bridge between those concepts and actual hardware — specifically, how you translate a customer’s real-world requirements (“we need to support 500 users”) into a GPU count you can put in a proposal.

The key metric that connects the two sides is tokens per second (TPS). To use it properly, you need to understand what’s actually happening inside the GPU when a model generates a response — because not all tokens are created equal.

Two Phases, Two Different Problems

When an LLM handles a request, it does so in two distinct phases. They look similar from the outside — text goes in, text comes out — but they have fundamentally different performance characteristics under the hood.

Phase 1: Prefill This is where the model reads and processes the entire input prompt.

• All the input tokens in your prompt are processed in parallel.
• This phase is compute-intensive — the GPU is doing a lot of simultaneous maths.
• It largely determines Time to First Token (TTFT): how long the user waits before they see any response at all.

Phase 2: Decode This is where the model generates the response, one token at a time.

• Each new token depends on the previous ones, so this phase is inherently sequential.
• And here’s the critical insight for sizing: the decode phase is often not limited by the GPU’s raw FLOPS.
• It’s limited by memory bandwidth — how fast the GPU can stream the model’s weights from high-bandwidth memory (HBM) to generate each token.

A quick note on FLOPS

You’ll see FLOPS quoted constantly in GPU spec sheets, so it’s worth understanding what it actually means — and where it does and doesn’t tell the full story.

FLOPS stands for Floating-Point Operations Per Second. It measures how much numerical computation a processor can perform per second. LLMs are essentially enormous stacks of matrix multiplications on floating-point numbers, so FLOPS is a natural unit for describing raw GPU compute power.

Vendors typically quote performance in:

• TFLOPS (tera-FLOPS = 10¹²) or PFLOPS (peta-FLOPS = 10¹⁵)
• Often broken down by precision: FP32 TFLOPS, FP16/BF16 TFLOPS, INT8 TOPS

So when you see “H100: X PFLOPS (FP16)”, that’s the peak theoretical compute at 16-bit precision — not what you’ll observe in a real LLM workload once memory access patterns, batching, and framework overhead come into play.

Here’s how FLOPS maps to the two inference phases:

• Prefill is FLOPS-hungry. Processing all prompt tokens in parallel is a heavy matrix multiplication workload — this is where raw compute throughput matters most. Higher FLOPS directly improves prefill speed and reduces TTFT.
• Decode is not FLOPS-bound. Generating tokens sequentially doesn’t saturate the GPU’s arithmetic units. The bottleneck shifts entirely to memory bandwidth — how fast the GPU can stream model weights from HBM for each token generated.

This distinction matters enormously in practice: a GPU with impressive FLOPS but modest memory bandwidth can underperform for LLM inference compared to one with higher bandwidth, even if the spec sheet comparison looks favourable. It’s why memory bandwidth is often the first number to check when evaluating accelerators for inference workloads — and why the H100 SXM, with its multi-TB/s HBM3 bandwidth, consistently outperforms lower-bandwidth alternatives for decode-heavy deployments.

The Core Metric: Tokens Per Second

Tokens per second (TPS) is your fundamental unit of inference throughput. Everything in a sizing conversation eventually traces back to it.

There are two ways to look at TPS, and you need to keep them separate:

• Per-user TPS — how fast tokens are delivered to a single user.
  • This drives the perceived experience.
  • Rough guide: below 10–15 tokens/sec starts to feel sluggish; above 30 tokens/sec it feels near-instant for most chat use cases.
• System TPS — the total token output across all concurrent users.
  • This is what you’re actually sizing the hardware to sustain.

The relationship is simple in principle:

System TPS = Concurrent Users × Tokens per Second per User

In practice, batching is what makes this efficient:

• Rather than serving each user’s request on dedicated GPU resources, a well-configured inference server groups multiple requests together and processes them as a single batch.
• This significantly improves GPU utilisation — particularly during the memory-bandwidth-bound decode phase.
• Batching is the primary mechanism that lets you serve many users from a relatively small GPU footprint.

Working Backwards: From Users to GPUs

Here’s the sizing workflow that turns a customer conversation into a hardware recommendation.

Step 1: Define the workload

Start with the usage-side discovery questions from Part 1:

• How many concurrent users?
• What’s the average prompt length (input tokens)?
• What’s the expected response length (output tokens)?
• What’s the acceptable latency — both time to first token (TTFT) and total response time?

A worked example

A customer wants to deploy an internal assistant. Together you define:

Step 2: Calculate required system TPS

From the example:

• 300 output tokens in 10 seconds = 30 tokens/sec per user
• 200 users × 30 tokens/sec = 6,000 tokens/sec system throughput

So the platform needs to sustain ~6,000 TPS of decoded tokens under load.

Step 3: Establish per-GPU TPS for your chosen model

This is where model size and GPU choice meet. As a rough reference for inference at FP16 (actual figures vary with batch size, framework, and optimisation):

Note: these are illustrative ranges. Always validate against benchmark data for your specific model, serving framework, optimisation level (TensorRT-LLM, vLLM, etc.), and batch configuration.

Step 4: Calculate GPU or node count

Continuing the example, assume:

• You choose a 70B model hosted on 4× H100 80GB nodes.
• Based on benchmarks, you take a conservative estimate of 1,000 TPS per node (decode, batched).

Then:

• 6,000 system TPS ÷ 1,000 TPS per node ≈ 6 nodes

Add a headroom buffer (typically 20–30% for burst traffic, uneven load, and future growth):

• 6 nodes × 1.25 ≈ 8 nodes as a starting recommendation.

At this point, you have a defensible answer to “how many GPUs/nodes do we need?” that’s grounded in user requirements, not just “bigger is better.”

Reference Sizing: Two Common Scenarios

The worked example above walks through the methodology. The table below applies it to two reference architectures you’ll encounter regularly — a 7B internal assistant and a 70B RAG system — to give you a practical feel for how the numbers land.

Figures assume FP16 or INT8 precision, batched inference, and a well-optimised serving framework such as TensorRT-LLM or vLLM. Treat these as directional reference points, not guaranteed benchmarks — validate against your specific model, configuration, and workload before quoting.

Notice the counterintuitive result: the smaller 7B model actually demands nearly four times the system throughput of the 70B RAG system (≈12,500 TPS vs ≈3,300 TPS). That’s not a contradiction — it’s the concurrency effect. Serving 500 chat users simultaneously generates far more aggregate token output than 100 users running deep reasoning queries, even though each individual 7B response is shorter. Bigger model doesn’t always mean bigger infrastructure footprint; workload pattern matters just as much as parameter count.

A few additional assumptions behind these figures worth keeping front of mind:

• No fine-tuning overhead — these are inference-only configurations. If the customer plans on-premises fine-tuning, GPU and memory requirements increase substantially.
• Steady-state load — the node counts include a 20–30% headroom buffer but assume reasonably predictable peak concurrency. Highly bursty workloads (e.g. end-of-day batch spikes) may warrant additional headroom or an autoscaling strategy.
• Single-tenant deployment — figures assume dedicated GPU resources per workload. Multi-model or multi-tenant deployments require separate sizing treatment.
• Retrieved context included — the 70B RAG prompt size of ~2,000 tokens already includes retrieved document chunks. If retrieval quality improves and chunk sizes grow, prompt tokens — and therefore TTFT — will increase accordingly.

The Trade-offs Every Customer Faces

Once you’ve run this exercise with a customer, three trade-off conversations typically follow.

1. Model quality vs. throughput

• A 70B model usually produces higher-quality outputs than a 7B.
• But it also serves far fewer users per GPU.
• For some use cases — summarising legal documents, writing complex code, specialised reasoning — the quality premium is worth it.
• For a high-volume customer service assistant, a well-tuned 7B model might deliver better economics with acceptable quality.

2. Latency vs. concurrency

• Larger batch sizes improve GPU utilisation and system throughput, but they increase the time an individual request spends waiting to join a batch.
• If TTFT is critical (live chat, voice interfaces), you’ll accept lower utilisation to keep batches small and responsive.
• If the application is asynchronous (batch document processing, offline analytics), you can run large batches, push utilisation higher, and drive down cost per request.

3. Precision vs. memory footprint

• Running a model at FP16 gives you full quality but also the full VRAM cost.
• Quantising to INT8 or INT4 roughly halves or quarters the memory footprint, allowing either a larger model to fit in the same GPUs, or the same model to fit in fewer GPUs.
• There is a quality trade-off, but for many inference workloads, well-done INT8 quantisation offers an excellent quality-to-cost ratio and is worth including in the conversation.

What This Means for Platform Selection

By this point in a customer conversation, you have enough to make an informed platform recommendation.

Single-node, lower concurrency, 7B–13B models A PowerEdge server with 2–4 high-memory GPUs will typically cover the requirement, with room to scale up or out as usage grows.

Multi-node, higher concurrency, or 70B+ models You’re looking at GPU-dense platforms where high-speed interconnect between GPUs (NVLink, NVSwitch) and network fabric between nodes become as important as raw GPU count. These directly affect prefill and decode performance, and therefore both latency and throughput.

Mixed workloads (inference + fine-tuning) Fine-tuning demands significantly more memory per GPU than inference alone (optimizer states, gradient storage, larger activations). If a customer plans both, size for fine-tuning — the inference requirement is typically covered as a result.

The specific Dell platform mapping — PowerEdge XE series, GPU configurations, and interconnect options — is what we’ll build out in Part 3.

Next up: Part 3 — Platform and GPU selection: mapping your sizing to Dell PowerEdge XE configurations. We’ll take the TPS-based sizing approach from this post and show how it translates into concrete server configs you can quote.

Every week, another organisation announces it’s deploying a large language model. And every week, a Technical Architect or Pre-sales Engineer gets asked a version of the same question: “How much infrastructure do I actually need for this?”

In my days as a Data Centre Architect and Engineer, I’d size server clusters for databases and VMware environments. The maths was different, but the discipline was the same: understand the workload, match it to the hardware, justify the recommendation. Now the question I get asked is “How do I size for LLMs?” This blog series is all about answering that.

Before you can answer that — before GPUs, nodes, interconnects, or platform choices even enter the conversation — you need two concepts nailed down cold: tokens and parameters. They’re the two dials that drive every LLM server sizing decision you’ll ever make.

Think of it this way. Parameters tell you how big the engine is. Tokens tell you how hard you’re asking it to work. Get those two right, and the rest of the sizing conversation falls into place.

Tokens: The Currency of Language Models

LLMs don’t read sentences the way you do. They don’t even read words. They read tokens — small chunks of text that sit somewhere between a syllable and a word.

• Sometimes a token is a whole word: server
• Sometimes it’s a fragment: serv, er
• Sometimes it’s punctuation or whitespace: . , ,

For English text, a useful rule of thumb is:

1 token ≈ 3–4 characters, or roughly 0.75 of a word

So the sentence “This is a sizing test.” runs to about 6–7 tokens — not 5, because the model doesn’t count words.

When you see pricing or performance metrics quoted in the market, they’re always denominated in tokens:

• $X per 1,000 tokens
• Y tokens per second
• 4k / 8k / 32k / 128k context window

That last one matters a lot. The context window is the maximum amount of text — measured in tokens — the model can hold in view at once. It’s not just the question you asked; it includes everything: system instructions, conversation history, documents you’ve fed in, and the response being generated. Every token in that window costs compute and memory.

Why Tokens Drive Sizing

Tokens show up in three places in every sizing conversation:

1. Context length (the prompt window) Longer context means the model has to track more information simultaneously. That translates directly into more VRAM for the KV cache — the memory structure the model uses to keep track of what it’s already processed. A customer who wants 128k-token context windows needs significantly more memory per request than one running at 4k.

2. Throughput and concurrency Tokens per second is the fundamental throughput metric — per GPU, per node, per cluster. In practice, you’ll often work backwards from a customer’s requirements:

“We need to support 500 concurrent users, each generating responses of around 300 tokens, within 3 seconds.”

That’s a tokens-per-second and concurrency problem. Everything else follows from it.

3. Capacity and cost planning Whether on-premises or cloud, consumption is effectively input tokens + output tokens. On a Dell PowerEdge server deployment, higher sustained tokens per second means more GPU compute, more memory bandwidth, and — beyond a certain point — more nodes or a move to higher-end accelerators.

Parameters: The Size of the Brain

If tokens are the currency of language models, parameters are what you’re buying with your hardware budget.

A parameter is a learned numeric weight — a floating-point number — stored inside the model. Mathematically, an LLM is an enormous function, and parameters are the numbers that define it. When a model is trained, billions of these weights are adjusted, incrementally, until the model gets reliably good at predicting language.

This is why model names look the way they do:

• 7B → approximately 7 billion parameters
• 13B, 34B, 70B, 405B → and so on up the scale

More parameters generally mean greater model capability — the model can represent more complex patterns, handle more nuanced reasoning, and produce higher-quality output. But that capability comes at a direct hardware cost, because every parameter has to live somewhere.

The VRAM Equation

The first-order estimate for model memory is straightforward:

Model VRAM (GB) ≈ Parameters × Bytes per Parameter

In practice:

That’s just for the weights themselves. In a real deployment you also need memory for:

• KV cache — grows with context length and batch size
• Activation memory — the working memory during computation
• Optimizer states — relevant if you’re fine-tuning, not just inferencing
• Runtime overhead — fragmentation, safety layers, serving framework

The practical consequence is clear:

• A 7B model can typically run on a single high-memory GPU (24–80 GB class, depending on precision and context requirements).
• A 70B model generally needs multiple high-VRAM GPUs — and demands fast interconnects between them, whether that’s NVLink, NVSwitch, or a high-bandwidth PCIe fabric.

As a pre-sales engineer, parameter count is what you’ll map to platform choices: how many GPUs per node, whether you need a GPU-dense platform with a high-speed fabric, and whether the workload fits in a 2U form factor or needs something more substantial.

How the Two Interact

Here’s the mental model worth keeping front of mind for every sizing conversation:

Given a fixed GPU budget, customers are always navigating a trade-off:

• Bigger model (more parameters) versus more throughput (more tokens per second)
• Longer context windows (more tokens per request) versus more concurrent requests

There’s no universally right answer — it depends on the use case. A legal document analysis platform that processes 100k-token contracts needs a very different configuration from a customer service chatbot handling hundreds of short, concurrent sessions.

Turning This Into a Sizing Conversation

When you strip away the jargon, most customer LLM questions reduce to this:

“Given a model size (parameters) and an expected usage pattern (tokens), how many GPUs and servers do I need to hit my latency and concurrency targets?”

The discovery questions that unlock that answer fall into two groups:

Model side:

• Are you targeting a 7B, 13B, 70B, or larger class model?
• Are you planning full precision, mixed precision, or quantized deployment?
• Is this inference only, or do you also plan fine-tuning?

Usage side:

• What’s the average prompt size per request (in tokens)?
• What’s the expected response length?
• What’s the maximum context length required — 8k, 32k, 128k?
• How many concurrent users do you need to support, and at what latency?

Once you have those answers, you can map them to Dell platforms — GB10, GB300, PowerEdge XE and XE+ GPU servers, interconnect choices, cluster configurations — in a structured and defensible way. That’s exactly what we’ll build up in the posts that follow.

Next up: Part 2 — from tokens per second to GPU count: the maths that drive inference sizing.