Benchmarking 6 LLM Providers for Latency and Cost (December 2024)
Why I ran this benchmark
So I was building a feature at work that need to call LLM thousands of times per day. The use case was structured JSON generation, basically generating a structured analysis for each incoming item, where the prompt was around 2,000 tokens of input and the expected output is a JSON object with scored criteria. Pretty repetitive task but at high volume.
When you’re doing this kind of volume, the question is not really about which model is the smartest. It’s more about latency, cost per request, and how consistent the output are. I didn’t want to just pick provider based on vibes or someone else’s benchmark that maybe using different prompt and different use case. I want to run my actual production prompt through each provider and see what I would actually pay.
“Let me just test all of them with my real workload and compare the number myself.”
That’s basically my thinking at that time.
The setup
I took 10 representative inputs and ran each one through 6 LLM providers, all on my personal credits. The test ran through my self-hosted Dify instance with LangSmith connected for observability, so I could see exact token counts, latencies, and cost per request for every single call.
Same prompt template, same input data, same expected output structure. The only variable was the model and provider. I was trying to make sure the comparison is as fair as possible, no other thing that could affecting the result besides the model itself.
The results
Here’s the average per request across 10 runs:
| Model | Latency | Tokens | Cost | Cost / 1000 req |
|---|---|---|---|---|
| Groq Llama 3.3 70B | 4.1s | 2,005 | $0.0014 | $1.37 |
| OpenAI GPT-4o | 6.4s | 1,630 | $0.0075 | $7.50 |
| OpenAI o1-mini | 14.3s | 2,953 | $0.0290 | $29.00 |
| Gemini Experimental 1206 | 14.9s | 1,815 | $0.0045 | $4.45 |
| Claude 3.5 Sonnet (OpenRouter) | 18.2s | 2,008 | $0.0145 | $14.51 |
| Claude 3.5 Haiku (OpenRouter) | 19.5s | 1,950 | $0.0035 | $3.50 |
What stood out
Groq Llama 3.3 70B was the fastest and cheapest by a wide margin. Like, 21x cheaper than o1-mini and 10x cheaper than Claude 3.5 Sonnet. For high-volume structured output where you don’t need heavy reasoning, this was pretty much the clear winner. Not bad, right?
GPT-4o had the best balance of everything. It produces the tightest output, lowest token usage among all model, consistent latency, and moderate cost. If I only needed few hundred calls per day, I would probably just go with this one.
Reasoning model like o1-mini is the wrong tool for this kind of task. The latency and cost both explodes when you’re doing high-volume scoring. The worst case in my test was pretty wild, a single request hit $0.548 and took 35.5 seconds. At 1000 requests per day, o1-mini would cost $29/day versus $1.37 for Groq. That’s a huge difference.
Claude via OpenRouter added routing overhead. Claude 3.5 Haiku peaked at 70.5 seconds latency for single request, I’m guessing it was a cold-start or routing issue on OpenRouter side. The model itself is fine, but the intermediary layer adds variance that you can’t really control.
Gemini Experimental 1206 was actually a good surprise for me. Competitive price, decent latency, and this was the model that was topping Chatbot Arena leaderboard at that time. I wasn’t expecting much from it but the result Actually pretty solid.
The caveat
These numbers are from December 2024, so keep that in mind. Providers are changing their pricing, routing, and model performance constantly. Groq was extremely aggressive on pricing at that time, that may or may not still be the case when you’re reading this.
The methodology matters more than the specific number here. If you’re planning to do high-volume LLM call, just run your actual workload through each provider before committing to one. That’s basically what I learned from this whole tinkering around.