Splitting LLM Workload Between Strong Model and Cheap Model

The consistency problem with single-model scoring

So I was building a system that need to score and classify items at high volume using LLMs. The straightforward approach was pretty simple, just send each item to single LLM, ask it to evaluate and score everything. Should be easy enough, right?

The problem appeared immediately. The scores was inconsistent between runs. I send the exact same item twice and I would get different scores. Not slightly different, sometimes wildly different. The reason is that the model was inventing its own scoring criteria on the fly for every single request. Each time it would reason about what matters, weight things differently, and arriving at different conclusion.

For a system that processing thousands of item per day, this kind of variance is not acceptable. Users would see their items scored differently depending on when the request happened to be processed. That’s really not something I could just ignore.

The insight: separate reasoning from application

The fix came from realizing that scoring actually have two distinct phase:

1. Deciding what to score on: this requires deep reasoning about the category, understanding nuance, weighing what matters more than what. This is the hard, expensive thinking part.
2. Applying those criteria to each item: this is mechanical. Given a rubric with clear criteria and weights, even a simple model can follow instruction consistently.

The first phase needs to happen only once per category or template. The second phase happens thousands of times per item. So why I’m using expensive reasoning model for both?

“Do I really need the big model to do this repetitive work every single time?”

That question basically changed how I approach the whole architecture. I was thinking to splitting the work between two model, one that do the heavy thinking once, and one that just apply the result over and over. It’s like having senior engineer write the standard operating procedure, then junior engineer just follow it. The senior doesn’t need to sit there watching every single execution.

The two-tier architecture

Tier 1: Criteria Generation (strong model, runs once)

A frontier model with chain-of-thought reasoning analyzes the task definition and generates structured, weighted scoring criteria. The key here is that different task types need completely different rubric. Like imagine you’re building an essay grading platform. An argumentative essay and a narrative essay need to be scored on very different thing. The strong model reads the assignment description and produces the rubric:

{
  "assignment": "argumentative_essay_climate_policy",
  "criteria": [
    {
      "name": "thesis_clarity",
      "weight": 0.25,
      "description": "Clear central argument stated early, specific position taken",
      "scoring_guide": "0.0 = no identifiable thesis, 0.5 = vague position, 1.0 = specific arguable claim"
    },
    {
      "name": "evidence_quality",
      "weight": 0.25,
      "description": "Claims supported with concrete data, statistics, or credible sources",
      "scoring_guide": "0.0 = no evidence, 0.5 = anecdotal only, 1.0 = multiple credible sources cited"
    },
    {
      "name": "counterargument_handling",
      "weight": 0.20,
      "description": "Acknowledges and addresses opposing viewpoints",
      "scoring_guide": "0.0 = ignores opposition, 0.5 = mentions but dismisses, 1.0 = engages and rebuts"
    },
    {
      "name": "logical_structure",
      "weight": 0.15,
      "description": "Arguments flow logically, each paragraph builds on previous",
      "scoring_guide": "0.0 = disconnected paragraphs, 0.5 = basic flow, 1.0 = clear logical progression"
    },
    {
      "name": "writing_mechanics",
      "weight": 0.15,
      "description": "Grammar, spelling, sentence variety, vocabulary appropriate for level",
      "scoring_guide": "0.0 = frequent errors impede reading, 0.5 = occasional errors, 1.0 = polished"
    }
  ]
}

Now compare that with what the same strong model would generate for narrative essay assignment. Completely different criteria: character development, sensory detail, narrative arc, dialogue usage, point of view consistency. The strong model is doing real reasoning work here, deciding what matters for each specific assignment. These criteria are stored and reused for every essay submission in that assignment. The strong model only runs again when new assignment is created.

Tier 2: Item Scoring (cheap fast model, runs per item)

A small, fast, cheap model receives the pre-generated criteria and scores each incoming essay against them. The prompt for cheap model is basically just:

Score the following essay against the provided criteria.
For each criterion, give a score from 0.0 to 1.0 based on the scoring guide.

ASSIGNMENT: {{ assignment_name }}
CRITERIA: {{ stored_criteria_json }}

ESSAY TO SCORE:
{{ student_essay }}

Return JSON:
{
  "scores": [
    { "criterion": "thesis_clarity", "score": 0.85, "reason": "Clear thesis in paragraph 1..." },
    { "criterion": "evidence_quality", "score": 0.60, "reason": "Two sources cited but..." }
  ],
  "total_weighted": 0.73,
  "feedback_summary": "..."
}

No criteria invention, just following the rubric. The cheap model doesn’t need to figure out what makes a good argumentative essay, that reasoning was already done once by the strong model. It just need to check: does this essay have a clear thesis? Give it a number between 0 and 1. Does it cite evidence? Score 0 to 1. Very mechanical, very consistent.

Why this works at scale

The economics is pretty dramatic when you look at the number. Using the numbers from my LLM benchmark, imagine a grading period where 50 assignments each receive 200 essay submissions:

• Single-tier approach: 10,000 essays × $0.015 (frontier model) = $150.00 per grading period
• Two-tier approach: 50 × $0.015 (criteria generation) + 10,000 × $0.0014 (cheap scoring) = $14.75 per grading period

That’s roughly 10x cheaper. And the criteria generation for all 50 assignments cost less than a dollar total. Not bad, right?

Processing speed also improves significantly. The cheap model responds in about 4 seconds versus 18 seconds for the frontier model, so throughput is 4.5x faster. When you have 200 essay coming in for same assignment, the scoring tier uses async batch processing with retries through queue system, so failures on individual essay do not block the rest.

I was actually surprised how much difference it makes when you start multiplying those small number across thousands of request. At first I thought the saving would be marginal but when I see the actual cost comparison it’s pretty clear that I was wrong.

The real argument: score variance

Cost and speed are nice, but the real reason I split the tiers is consistency. When single model handles both reasoning and scoring, it re-derives its criteria every time. Subtle changes in prompt context, temperature, or even token sampling can shift what the model considers important. I was struggling with this for a while before I realize the root cause, the model was basically starting from scratch on every request.

With the two-tier approach, the criteria are fixed. The cheap model cannot invent new criteria because it’s only given the rubric. Two identical essays will always receive same score, regardless of when they are processed. That consistency is what actually making the system trustworthy. Imagine student complaining “my friend submitted almost the same argument and got a different score.” With single-model scoring, that could actually happen. But using two-tier, it can’t.

Still tinkering around with some edge case on how the criteria generation handles ambiguous assignment description, but so far it’s been working pretty well for my use case.