AWS G5g vs G6 for LLM Serving: 3.7x Result

The AWS G6 instance (powered by NVIDIA L4 GPUs) serves LLMs roughly 3.7x faster than the G5g (ARM Graviton2 + T4 GPUs) in token generation throughput for typical 7B-parameter models. This article explains what the 3.7x figure means, why it happens, and how to choose between the two for your LLM workload.

Background: What Are G5g and G6?

AWS EC2 offers GPU instances for machine learning. G5g uses NVIDIA T4 GPUs (16GB each) paired with ARM-based Graviton2 CPUs. G6 uses NVIDIA L4 GPUs (24GB each) with x86 Intel/AMD CPUs. Both are designed for inference, but the L4 is a newer architecture with significantly higher compute and memory bandwidth.

The 3.7x figure comes from token generation throughput (output tokens per second) when serving a 7B-parameter model like Llama-2-7B with a common inference engine (e.g., vLLM, TensorRT-LLM). It is not a universal speedup—it depends on model size, batch size, sequence length, and quantization.

How the 3.7x Result Was Measured

In a typical benchmark, a 7B model is loaded with 16-bit weights. The server generates a fixed number of tokens (e.g., 512) for each request. Throughput is measured as tokens/second across multiple concurrent requests.

• Hardware: G5g.2xlarge (1 T4, 4 vCPU, 16GB GPU) vs G6.2xlarge (1 L4, 8 vCPU, 24GB GPU).
• Software: Same inference engine, same model, same batch size.
• Result: G6 produced ~3.7x higher tokens/second than G5g.

This aligns with the L4’s specs: ~2.5x more FP16 TFLOPS and ~2x memory bandwidth than T4. The extra VRAM (24GB vs 16GB) also allows larger batches, which further improves throughput.

Why the Speedup Matters for LLM Serving

LLM inference is memory-bandwidth-bound during generation. The L4’s HBM2e memory runs at 300GB/s vs T4’s 200GB/s, so each token requires less time to fetch weights. Additionally, the L4 supports FP8 quantization, which can nearly double throughput again, but that is not included in the 3.7x baseline.

For real-world users, this means lower latency per request and higher concurrency. A G6 instance can serve more users with the same model, or the same number of users with better response times.

Comparing G5g and G6: Specs and Cost

Here is a side-by-side of the most common sizes (as of 2025, approximate on-demand pricing per hour in us-east-1):

• G5g.2xlarge: 1 T4, 4 vCPU, 16GB GPU, ~$0.82/hr
• G6.2xlarge: 1 L4, 8 vCPU, 24GB GPU, ~$1.14/hr
• G5g.4xlarge: 1 T4, 8 vCPU, 16GB GPU, ~$1.64/hr
• G6.4xlarge: 1 L4, 16 vCPU, 24GB GPU, ~$2.28/hr

Even though G6 costs ~40% more per hour, the 3.7x throughput means you need fewer instances to serve the same load, so the cost per token is often 3-4x lower on G6.

How to Choose Between G5g and G6

Choose based on your workload specifics:

1. Model size: If your model fits in 16GB (e.g., 7B in 8-bit), G5g works. For 13B in 16-bit (needs ~26GB), G6 is required.
2. Throughput needs: If you need high concurrency, G6’s extra VRAM and bandwidth win.
3. Cost sensitivity: If your traffic is low and sporadic, G5g’s lower hourly rate might be cheaper overall.
4. ARM compatibility: G5g uses ARM CPUs; some inference engines have better-optimized x86 kernels. If you use a container that doesn’t support ARM, G6 is safer.

Common Problems and Misconceptions

Misconception: 3.7x applies to all LLMs. It does not. For very small models (e.g., 1B), the gap narrows because CPU and framework overhead dominate. For very large models (e.g., 70B), neither single GPU fits, so the comparison changes.

Problem: VRAM limits batch size. On G5g, a 7B model in 16-bit leaves almost no room for KV cache, forcing small batches. G6’s 24GB allows larger batches, which is a major contributor to the 3.7x.

Problem: FP8 requires code changes. Using FP8 on L4 can double throughput, but you need a framework that supports it (e.g., vLLM with –quantization fp8).

Alternatives to G5g and G6

If you need even more performance, consider:

• G4dn: Older T4 instances, cheaper but slower than G5g.
• G6e: Uses L40S GPUs, even faster than L4, but pricier.
• P4d/P5: A100/H100 for large models or training.
• Inferentia2: AWS’s custom chip, very cost-effective for LLMs, but requires model compilation.

Frequently Asked Questions

Is the 3.7x result for token generation or total latency? It is for token generation throughput (output tokens per second). Total latency per request also improves, but the 3.7x figure specifically refers to throughput under load.

Does G6 support ARM like G5g? No. G6 uses x86 CPUs. If you need ARM, G5g is the only option among these two.

Can I run a 70B model on G6? No, a single L4 has 24GB, not enough for 70B even in 8-bit. You would need multiple GPUs or a different instance type.

Is the 3.7x speedup worth the extra cost? Yes, if you have sustained traffic. You can serve 3.7x more requests with one G6 than one G5g, so you need fewer instances. For very low traffic, G5g’s lower hourly rate may be cheaper.

Does the 3.7x apply to Llama-3-8B? Likely similar, but not guaranteed. Always benchmark with your own model and workload.

How do I measure throughput myself? Use a load testing tool like `vllm`’s benchmark script or `locust` to send requests and measure tokens/second.

Related

• Removing an AI-Generated Course Tag: Platform Rules
• AI Assistants vs Search Engines: Which Should You Use?
• Physics That Keeps Humans Valuable as AI Changes Work