How to Optimize GPU Utilization? 5 Steps to Find the Real Cause of Compute Idling

Optimizing GPU utilization isn't about inflating the metric; it's about opening the monitoring dashboard, aligning GPU utilization, request queue time, and prefix cache hit rate side-by-side, then methodically checking in this order: utilization number → queueing → bandwidth → duplicate computation → engine selection. Looking only at nvidia-smi's static occupancy is no longer sufficient; you need a systematic assessment that incorporates request distribution and engine parameters.

Bottom Line Up Front: The Order of Investigation for GPU Utilization Optimization

First, verify what the "utilization" number you see actually represents; second, check if requests are queued and causing GPU idle time; third, check if memory bandwidth is the bottleneck, leading to high utilization but no increase in effective token output; fourth, check for duplicate computation (e.g., prefix recomputation and preemption re-computation); only last, consider whether to switch inference engines. Benchmarks and observability practices from the second half of 2026 (e.g., from DeepInfra, Spheron, and others) consistently prove that relying solely on nvidia-smi utilization percentages cannot distinguish between three classes of problems: "true idle," "fake saturation," and "compute without output."

Three Flavors of "Poor Utilization": True Idle, Fake Saturation, Compute Without Output

Many mistakenly believe that maximizing GPU utilization to 100% will linearly increase throughput, but when KV cache insufficiency triggers preemption (preemption and recomputation), the GPU may be fully loaded while effective token output doesn't improve.

Step 1: Confirm What the Utilization Number Means and Which Metrics to Watch

The first thing in GPU utilization optimization is to confirm what the number displayed by nvidia-smi actually represents. The GPU utilization displayed by nvidia-smi reflects whether any kernel is running on the SMs (streaming multiprocessors), not whether the compute power is being used effectively. To determine if there's "compute idling," cross-validate with these finer-grained metrics:

These metrics can be collected via Prometheus; for example, vLLM has built-in metrics. If you find high queue times but low GPU utilization, it indicates uneven request arrival or low concurrency settings; if both high, it might be a scheduling issue.

Step 2: Estimate the Prefix Overlap of Request Distribution—Why the 60% Line Is Critical

Open your production logs and calculate the proportion of shared prefixes among requests—such as system prompts, RAG retrieval blocks, and multi-turn conversation history that frequently repeat. Compute the rate of prefix repetition across all requests. When this value exceeds 60%, it's worth considering an engine with prefix reuse capabilities. This 60% quantitative boundary comes from controlled benchmarks in 2026 (e.g., comparative evaluations by DeepInfra and Spheron), not marketing claims.

Step 3: How to Optimize When Prefix Overlap Is High, and What Gains to Expect

If your scenario has prefix overlap above 60% (e.g., typical RAG applications), using SGLang's RadixAttention can reduce time-to-first-token (TTFT) by 20%–40% compared to vLLM. Note this is an improvement in TTFT, not a doubling of overall throughput. This result is from controlled benchmarks that keep model, precision, and concurrency constant, and it holds only when prefix overlap >60%, reflected solely in TTFT, not a proportional gain in overall throughput. In scenarios with high prefix overlap like multi-turn dialogue and RAG, first-token latency significantly impacts user experience. For practical deployment details, refer to SGLang deployment.

Step 4: When Prefix Overlap Is Low, Don't Rush to Switch Engines—Tune Parameters First

If prefix overlap is below 60%, or independent prompts dominate, the throughput difference between the two engines is within 5%, and migrating engines may cost more than the benefit. This difference is based on controlled comparisons; it's advisable to replay your own request distribution to verify before concluding.

Prioritize tuning vLLM parameters: For example, --gpu-memory-utilization defaults to 0.9, allocating 90% of GPU memory for model weights and KV cache, leaving a 10% buffer to prevent OOM from CUDA graphs and temporary tensors spilling over. If KV cache space is insufficient and preemption recomputations are frequent, you can increase this ratio appropriately, or decrease max_num_seqs and max_num_batched_tokens to reduce concurrency pressure. Note that max_num_seqs and max_num_batched_tokens have opposite adjustment directions: Increasing max_num_seqs improves batching capability and reduces queue time, but increases memory pressure and may trigger more preemptions; conversely, decreasing these parameters reduces concurrency and preemption counts but may lengthen queue times. You need to balance queue time and preemption count based on your request arrival rate and memory headroom.

Step 5: Translate Throughput Gains into Cost per Million Tokens to Decide if It's Worth It

Calculation: hourly rental cost ÷ effective token output. Effective token output = output tokens per request × successful requests per unit time. Migration engineering effort and stability risks should also be factored into the benefit side. If throughput improves significantly after tuning, consider long-term resource allocation; otherwise, spot instances for short-term validation are more cost-effective. Use on-demand instances (e.g., same-spec GPUs readily available on NexGPU) for brief validation, converting the effective throughput difference before and after optimization directly into rental cost difference. GPU utilization optimization ultimately comes down to cost per million tokens.

Conducting a Dual-Engine A/B Test with the Same Request Distribution: What Metrics to Record for Comparability

When running A/B tests, you must maintain the same model, precision, request distribution, and concurrency gradient, and record these metrics: TTFT, per-output-token latency, effective throughput, prefix cache hit rate, and preemption count. It's worth noting that this comparison needs to run on identical instances. NexGPU offers on-demand GPU resources and pre-built model templates, allowing you to spin up identical instances for two rounds of testing and release them afterward. If you have questions about cluster planning, refer to GPU cluster planning. For multi-GPU scenarios, consider tensor parallel and data parallel splitting, and the impact of inter-GPU communication overhead on throughput; see recommendations in Multi-GPU inference optimization.

Four Common Misconceptions and a Troubleshooting Checklist

1. Treating nvidia-smi full utilization as full compute usage—needs cross-validation with fine-grained metrics.
2. Treating TTFT improvement as overall throughput improvement—lower TTFT doesn't imply higher throughput.
3. Blindly trusting non-controlled claims like "Engine X is 29% faster across all scenarios"—check if version, parameters, and request distribution are consistent.
4. Increasing memory ratio too much and causing OOM—the 10% buffer is for stable operation.

Troubleshooting Checklist

• [ ] Confirm the source of the GPU utilization metric (nvidia-smi vs engine metrics)
• [ ] Check request queue duration to determine idle time
• [ ] Calculate prefix overlap ratio to estimate duplicate computation
• [ ] Inspect KV cache hit rate and preemption count
• [ ] Adjust gpu-memory-utilization and max_num_seqs parameters as needed
• [ ] If necessary, run a dual-engine comparison under the same request distribution

FAQ

How to troubleshoot low GPU utilization?

First, look at request queue time: if there's a lot of queuing but the GPU is idle, concurrency settings might be too low or request arrival is uneven. Then check the prefix cache hit rate; if it's low, many prefixes are being recomputed, wasting compute. Finally, check if the memory ratio is too small, causing frequent preemption. This order should help you locate the issue.

Why is GPU utilization 100% but throughput not increasing?

This is usually due to memory bandwidth limits or preemption recomputation. The decode phase is memory bandwidth-bound; even if SMs are fully loaded, token output can't increase. Additionally, KV cache insufficiency triggers RECOMPUTE, consuming compute without producing results. In this case, increase gpu-memory-utilization or reduce concurrency.

What is an appropriate value for gpu-memory-utilization?

The default 0.9 is a good starting point; it allocates 90% of GPU memory to model and KV cache, leaving a 10% buffer to prevent OOM. If preemption occurs frequently, you can adjust upward slightly, but after each adjustment, observe if CUDA graphs and runtime temporary tensors trigger OOM; the upper limit depends on the model and GPU memory. If issues persist, lower max_num_seqs.

How to interpret the prefix cache hit rate?

Look for the prefix cache hit rate metric in vLLM's Prometheus metrics. If it's low, many prefixes are being recomputed; consider enabling prefix reuse. You'll need to calculate the shared prefix proportion of your requests to determine if it's a high-overlap scenario.

How to choose an inference engine for RAG scenarios?

If prefix overlap exceeds 60%, SGLang is preferred; its RadixAttention can reduce first-token latency by 20%–40%. If overlap isn't high, the throughput difference between vLLM and SGLang is within 5%, so choose the more mature vLLM and focus on tuning memory parameters.

How to optimize high time-to-first-token in multi-turn dialogue?

First, measure the prefix overlap caused by multi-turn history; if >60%, evaluate prefix reuse. Also check if KV cache insufficiency is triggering preemption and recomputation. You can increase gpu-memory-utilization or lower concurrency parameters to reduce preemption events.