Which GPU do you need to serve AI models

Choosing GPUs for inference is not about buying the most expensive one. It is about buying the one that understands the format your model lives in.

Every generation adds hardware support for smaller number formats (FP8, NVFP4, FP6), and each new format translates into one of three things: bigger models in the same VRAM, more throughput, or fewer GPUs to serve the same load. But not every GPU supports every format. If your model uses a format the GPU does not accelerate in silicon, you pay the penalty of the software-emulated path, which is usually 2x to 10x slower.

Here are the families that matter today, July 2026, what each one accelerates, and how to translate that into a concrete buying decision: from a compact MoE like Qwen3.6-35B-A3B up to a massive one like GLM-5.2.

1. The map of NVIDIA architectures

NVIDIA has shipped six datacenter architectures in six years. Each one introduces new-generation Tensor Cores that understand smaller precision formats.

One detail people constantly get wrong: the RTX PRO 6000 is Blackwell, not Ada. It shares the B200's 5th-gen Tensor Cores (NVFP4, FP6 and FP8 in hardware) with 96 GB of GDDR7. Don't confuse it with the old RTX 6000 Ada Generation, which has 48 GB and does not accelerate FP4. That difference is exactly why we run NVFP4 in production, as we covered in the NVFP4 migration post .

2. Precision formats: what each GPU accelerates in hardware

Not every format runs the same on every GPU. A model quantized to NVFP4 does not automatically run at full speed on an H200, because Hopper has no 4-bit Tensor Cores: it emulates them over FP8 or FP16 and you lose the whole throughput and memory advantage.

One important clarification, because it is a common mistake: INT8 is native on the Tensor Cores since Ampere (A100). It is not emulated. What changed with the latest generations is not INT8, it is 4-bit. INT4 peaked with Ampere, Hopper set it aside, and Blackwell replaced it with NVFP4, which keeps far more numerical precision at the same size thanks to per-block scaling.

Three things follow from that table:

Hopper (H100/H200) is the queen of FP8. If your model is in FP8, and today most open models publish an official FP8 checkpoint, Hopper serves it at full speed with its Transformer Engine. What Hopper does not understand is 4-bit floating point.

Blackwell unlocks FP4 and FP6 in hardware. NVFP4 is not just another quantization: they are 5th-gen Tensor Cores that operate natively at 4 bits. Our NVFP4 checkpoint of Qwen3.6-35B-A3B takes half the VRAM of FP8 and delivers 11% more throughput on the same card, with 12 points less KV cache usage.

Loading NVFP4 on Hopper works, but it is pointless. The GPU runs it with software kernels over FP8 or FP16. The model responds, the advantage disappears. It is the most expensive buying mistake on this list, because it does not fail: it just performs as if you had done nothing.

3. How much VRAM your model needs

This is the exercise every company should do before buying. Let's take the three models we run in production and see which GPU holds them.

✅ = fits in a single current GPU with room for KV cache and serving.

The pocket rule: FP16 is ~2 bytes per parameter, FP8 ~1 byte, NVFP4 ~0.5 bytes. Multiply by the number of parameters and you have the model weight. Then add the KV cache, which grows with context and batch, plus the serving engine overhead.

What each row means for buying decisions:

Qwen3.6-35B-A3B is tiny in VRAM. In NVFP4 it takes about 18 GB. It fits in any current GPU, including an RTX PRO 6000 with 96 GB, with plenty of room for KV cache and concurrency. It is exactly the model we serve on that card.

DeepSeek-V4-flash changes category with the format. 284B total, 13B active. In NVFP4 it is about 142 GB and fits in a single B200 (192 GB) or B300 (288 GB) comfortably. In FP8 it is 284 GB and you already need two cards if you want long context. Even though only 13B activate per token, all 284B must be loaded in memory because any expert can be invoked.

GLM-5.2 is the extreme case. 753B total, 40B active. In FP8 it is about 753 GB, so we are talking 4 B200 or 3 B300. In NVFP4 you drop to 377 GB and two cards are enough. That is why format is critical in large MoEs: it does not reduce compute per token, but it does reduce the number of GPUs you need just to hold the model loaded.

4. Throughput: tokens per second in the real world

A model that fits is not enough. You have to serve requests at a useful speed, and there memory bandwidth (HBM) and Tensor Cores rule.

The short read of that chart: the A100 to H100 jump is explained by the Transformer Engine and native FP8, and the Hopper to Blackwell jump is explained by bandwidth plus 4-bit. Between adjacent generations the difference is not in catalog TFLOPS, it is in whether the GPU speaks the language of your checkpoint.

The real case: Qwen3.6 in NVFP4

We validated these figures with production data. Migrating our Qwen3.6-35B-A3B checkpoint from FP8 to NVFP4 on an RTX PRO 6000 (96 GB, Blackwell sm_120) gave us this:

Same hardware, same model, more throughput. Only the checkpoint format changed.

The full write-up of the migration is in the previous post , including the cuDNN bug we hit along the way: a kernel that picked datacenter sm100 tilings and wrote out of bounds on sm120's 99 KB of shared memory.

5. Which GPU to buy or rent for your use case

There is no single best GPU. There is a GPU that fits each workload.

Research, fine-tuning and proofs of concept

• H100. Enough for almost everything and the natural choice if usage is intermittent.
• RTX PRO 6000. Blackwell in workstation form. Local capacity without depending on the cloud, with native NVFP4 and 96 GB.

Production at scale with compact models

Serving a compact MoE like Qwen3.6-35B-A3B:

• H200. Its 141 GB give room for long KV cache, 256K contexts included. Still the FP8 workhorse.
• B200 in NVFP4. If your model has an NVFP4 checkpoint, it is the highest throughput per GPU.
• RTX PRO 6000. Native NVFP4 with 96 GB, plenty for models of this size locally.

Massive models: DeepSeek-V4-flash, GLM-5.2

• B200 cluster, 2 to 4 cards. With NVFP4, DeepSeek-V4-flash fits in 1 or 2 and GLM-5.2 in 2. Without NVFP4 you need twice as many.
• B300 (Blackwell Ultra). 288 GB per GPU, so you need fewer cards for the same model.
• H200 cluster. A solid alternative if the model is in FP8 and you are not going to migrate.

Ultra-low latency: coding agents, real-time chat

Many small requests that need an immediate response:

• B200 and B300. The lowest TPOT thanks to 8 TB/s of bandwidth. Each token arrives sooner.
• H200. Also excellent if the model is FP8.
• Avoid A100. Its ~2 TB/s are an obvious bottleneck in the decode phase.

Mixed load: AI, graphics and virtualization

• L40S and RTX 6000 Ada. They combine Tensor Cores with RT Cores and video output. For engineering, design or vGPU workstations they are still the natural choice. If you also want native FP4 on the desktop, the RTX PRO 6000 Blackwell is the evolution.

6. What's coming: Blackwell Ultra and Vera Rubin

Blackwell Ultra (B300 / GB300), already available

• 288 GB of HBM3e per GPU at 8 TB/s, thanks to 12-high stacks
• 15 PFLOPS FP4 dense per GPU
• GB300 NVL72: 1.1 ExaFLOPS FP4 per rack, 1.5x the GB200 NVL72
• 1,400 W TDP, with liquid cooling mandatory

In practice, a ~200B parameter model in NVFP4 fits in a single GPU, and for a massive one in FP8 with 2 or 3 B300 you have plenty of room even with long KV cache.

Vera Rubin (R100), in production since June 2026

• TSMC 3 nm, 336B transistors across two dies
• HBM4: 288 GB per GPU at 22 TB/s, 2.8x over Blackwell
• 50 PFLOPS FP4 inference per chip
• Rubin NVL144: the new rack reference, about 3.3x the performance of the GB300 NVL72

NVIDIA entered production in June 2026 and cloud availability (AWS, GCP, Azure, Oracle) arrives through the second half of the year. If you are signing a 3-year GPU contract this quarter, that calendar is part of the negotiation.

7. How to make the decision, step by step

1. Pick your target model. Not "a GPU", but "I am going to serve GLM-5.2 at 200 requests per second".
2. Pick the inference precision. FP8 is the standard today. NVFP4 is the edge if your hardware supports it.
3. Compute the VRAM. Model weight, plus KV cache (context by batch), plus serving overhead.
4. Pick the GPU with native Tensor Cores for that precision. NVFP4 on Blackwell and NVFP4 emulated on Hopper are not the same.
5. Compare real throughput in tokens per second and effective VRAM, not catalog specs.
6. Add a 20-40% margin for traffic spikes and growth.

The right GPU lets you serve more model with less hardware. The wrong one ties you to a format, a platform or a VRAM capacity that limits what you can offer during the three years the depreciation lasts.

At Helmcode we have processed more than 574 billion tokens, with peaks of about 14 billion a day. The three models in this post are exactly the ones we serve most, and the numbers are open on our public data page . We have gone through A100, H100, Ada and now Blackwell, and every generational jump has given us more capacity without changing the number of servers. Optimization is not only software: it is knowing which hardware understands your model.

Frequently asked questions

How much VRAM does a 70B parameter model need?

In FP16 about 140 GB, in FP8 about 70 GB and in NVFP4 about 35 GB, for the weights alone. On top of that comes the KV cache, which depends on context and batch, and the serving engine overhead. With a 20-40% margin on top, a 70B in FP8 asks for an H200 (141 GB) or larger.

Can I run an NVFP4 model on an H100 or H200?

Yes, but you should not. Hopper has no 4-bit Tensor Cores, so it emulates the format over FP8 or FP16. The model responds, but you lose the throughput and memory advantage that justified using NVFP4.

What is the difference between FP8 and NVFP4?

FP8 uses 8 bits per number and is the current standard for open-model inference. NVFP4 uses 4 bits with block scaling, takes half the space and keeps far more numerical precision than the old INT4. Only Blackwell accelerates it in hardware.

Is the RTX PRO 6000 Ada or Blackwell?

Blackwell. It carries 5th-gen Tensor Cores with NVFP4, FP6 and FP8 in hardware, and 96 GB of GDDR7. The confusion comes from the RTX 6000 Ada Generation, the previous generation, which has 48 GB and does not accelerate FP4.

Which GPU do I need to serve a large MoE like GLM-5.2?

Even though only 40B parameters activate per token, all 753B must be loaded in memory. In FP8 that is about 753 GB, meaning 4 B200 or 3 B300. In NVFP4 you drop to 377 GB and two cards are enough.

Is it worth buying GPUs or renting inference through an API?

It depends on usage. With intermittent or variable load, a flat-rate API avoids amortizing hardware that sits idle. With sustained, predictable load, or data-residency requirements that force on-premise, buying starts to make sense. Real utilization decides it, not the price of the card.

---

*Need GPUs for your workloads? Talk to us and we'll tell you about our availability of dedicated Blackwell GPUs, or we serve you the same models directly from our inference API without you having to buy anything.*