How to Choose Batch Sizes to Minimize Cost per Token in LLM Serving

Running large language models (LLMs) at scale isn’t just about having powerful hardware-it’s about squeezing every bit of efficiency out of each inference. If you’re paying for API calls or hosting your own models, you’ve probably noticed how quickly costs add up. One of the most effective ways to slash your cost per token is by batching requests. But choosing the right batch size isn’t guesswork. It’s a trade-off between speed, memory, and money-and getting it wrong can cost you thousands.

Why Batch Size Matters More Than You Think

Think of your GPU like a kitchen. If you cook one meal at a time, you’re wasting heat, electricity, and time. But if you cook 20 meals together in the same oven, you spread the fixed cost over more servings. That’s batching. Instead of sending one user’s prompt to the model and waiting for a response, you group 10, 50, or even 200 prompts and run them all at once. The model loads once. The GPU computes once. The cost per token drops.

Companies like Scribd and First American cut their LLM bills by over 40% just by switching from single-request calls to smart batching. One fintech team reduced their monthly OpenAI bill from $18,500 to $9,200-no model change, no new hardware. Just better batching.

What’s the Best Batch Size?

There’s no universal number. The optimal batch size depends on your model, your input length, and your latency tolerance.

• Text generation (like summarizing documents or writing emails): 10-50 requests per batch
• Classification (like tagging support tickets or sentiment analysis): 100-500 requests
• Q&A systems (like customer chatbots): 50-200 requests

For example, a 7B-parameter model saw its latency drop from 976ms at batch size 1 to just 126ms at batch size 8. But push past batch size 64, and gains flatten out. Why? Memory. Larger batches need more space to store intermediate calculations (the KV cache). If you exceed your GPU’s memory, you get crashes-not savings.

Hardware Limits: The Hidden Bottleneck

Your GPU isn’t just a processor-it’s a memory machine. A 70B model like LLaMA2 needs about 140GB of VRAM just to load. Add a batch size of 32 with long prompts, and you’re quickly hitting 200GB+. That’s why many teams max out at batch size 64-even if their GPU could technically handle more.

Here’s the kicker: consumer-grade GPUs like the RTX 4090 often offer better memory bandwidth per dollar than enterprise chips like the H100. A study from February 2025 showed consumer GPUs deliver 1.9x more memory bandwidth per unit price. For smaller models (7B-13B), you can run larger batches on cheaper hardware and still outperform expensive cloud GPUs.

Static vs. Dynamic vs. Continuous Batching

Not all batching is created equal.

• Static batching groups requests ahead of time. Great for scheduled jobs like overnight document processing. But if your traffic is unpredictable, you’ll have idle GPUs or long queues.
• Dynamic batching waits for requests to arrive, then groups them as they come in. Ideal for chatbots or APIs with spiky traffic. Most teams use this.
• Continuous batching is the next level. As one request finishes generating text, the GPU immediately starts processing a new one in the same slot. No waiting. No gaps. Tools like vLLM and TensorRT-LLM use this to achieve up to 24x higher throughput than standard Hugging Face setups.

Continuous batching isn’t magic-it’s math. It keeps the GPU busy 95% of the time instead of 60%. For high-volume services, that’s the difference between 3 servers and 1.

Streaming vs. Batch: The Cost Trap

If you’re streaming responses-sending tokens one by one as they’re generated-you’re paying a premium. Streaming keeps the GPU in constant motion, uses more memory, and prevents batching. It’s great for interactive chatbots where users expect instant replies. But if you’re generating reports, translating documents, or analyzing logs? Don’t stream. Batch. You’ll save 20-40% just by turning off streaming.

Combine Batching With Other Cost Hacks

Batching alone won’t solve everything. Pair it with these proven tactics:

• Model cascading: Send 90% of simple queries to a tiny model like Mistral 7B ($0.00006 per 300 tokens). Only escalate complex ones to GPT-4. Combined with batching, this can cut costs by 87%.
• Early stopping: Configure your model to stop generating once it’s confident. This cuts output tokens by 20-40% without hurting quality.
• Quantization: Run models with 4-bit or 8-bit precision. You lose almost no accuracy but gain 2-3x larger batch sizes on the same GPU.
• Caching: Store responses to repeated queries (like “What’s our return policy?”). Retrieval-augmented generation (RAG) with caching can reduce redundant calls by 15-25%.

Real-World Numbers: What You Can Actually Save

OpenAI slashed GPT-4 pricing from $36 to $5 per million tokens between March 2023 and September 2024. That’s an 86% drop. But with batch processing, you can go even lower-to $2.50 per million tokens. That’s another 50% off.

One team serving 5 million tokens per day with GPT-4 paid $150/day before batching. After switching to batch size 35 with continuous batching and early stopping? $65/day. That’s $2,550 saved per month.

Another team running LLaMA2-70B on A100s was spending $2,500/day. After switching to RTX 6000 Ada GPUs with dynamic batching and 4-bit quantization? $900/day. And their latency improved.

How to Start

You don’t need to overhaul your system overnight. Here’s how to begin:

1. Identify your most frequent use case. Is it classification? Summarization? Q&A?
2. Start with a batch size of 20-30 for that use case.
3. Monitor GPU memory usage and latency. Use tools like NVIDIA Nsight or Prometheus.
4. Gradually increase batch size until memory hits 80% or latency exceeds your SLA.
5. Turn off streaming if it’s not needed.
6. Test model cascading: route 50% of simple queries to a smaller model.
7. Measure cost per token before and after. Track savings over 30 days.

Most teams need 2-4 weeks of tuning. One engineer on Reddit spent three weeks testing batch sizes from 5 to 100 before landing on 35. The result? A 58% cost cut.

What Not to Do

• Don’t assume bigger is always better. Batch size 128 might crash your GPU.
• Don’t batch requests with wildly different input lengths. A 500-word prompt and a 10-word prompt in the same batch forces the GPU to pad everything to 500 words-wasting compute.
• Don’t ignore monitoring. Without tracking memory, latency, and cost per token, you’re flying blind.

The Future Is Automated

The next wave of LLM serving won’t rely on engineers manually tweaking batch sizes. Tools like Anthropic’s upcoming 2025 Q4 release will auto-optimize batching based on real-time traffic, model load, and cost targets. But until then, the best tool you have is data-track, test, and iterate.

Final Thought

LLMs aren’t getting cheaper. They’re just getting smarter about how they’re used. The companies winning aren’t the ones with the biggest GPUs. They’re the ones who know how to pack them full.