Interactive evaluator calibration

LLM-as-a-Judge
Bias Lab

The verdict should survive the AB swap. Stress-test a synthetic pairwise judge, expose order and verbosity bias, then decide when a human should take over.

Run the stress test See the protocol

Teaching simulator. Its ballots are deterministic synthetic outputs, not benchmark measurements.

Evaluation task

Decision protocol

Prompt under evaluation

Candidate A

Shown first

Rubric quality0

Candidate B

Shown second

Rubric quality0

Bias controls

Turn the evaluator's hidden preferences up or down. The response qualities remain fixed.

Position bias10 pts

Adds an advantage to whichever answer appears first.

Verbosity bias8 pts

Rewards the longer answer even when length adds no task value.

Ballot variance3 pts

Models repeated-run instability around the same comparison.

Repeated ballots9

More ballots reveal consistency; they do not remove systematic bias.

Factuality weight45

Task-fit weight25

Clarity weight15

Completeness weight15

Blind namesHide identity-like labels

Require evidenceReward grounded support

Candidate-identity margin

Every lane maps the displayed answer back to candidate A or B. Positive margins favor A; negative margins favor B.

Synthetic deterministic ballots

Protocol resultCandidate ASingle pass

Order consistencyNoSame winner after swap

Position swing0 ptsDistance between orders

Repeat stability0%Ballots matching lane majority

Order-sensitive verdict

A score is an observation, not a ground truth.

Pairwise evaluation combines the quality signal you care about with presentation effects and run-to-run noise. Calibration separates the pieces instead of trusting one polished number.

The observed verdict has extra terms.

A reliable design makes those terms visible, measurable, and removable where possible.

quality + position + verbosity + noise

Swap the order, preserve identity.

Run A/B and B/A, map each ballot back to the same candidate identity, then average the margins.

Position bias

The first or second slot can attract preference independent of content. A swap exposes the directional effect.

Verbosity bias

More detail can look like more quality. Swapping does not cancel this bias; the rubric must penalize irrelevant length.

Prompt sensitivity

Rubric wording and output format change what the judge attends to. Lock prompts, versions, and evidence before comparing runs.

A calibration protocol you can audit.

Keep the judge useful without pretending it is impartial. Each stage leaves an artifact that a reviewer can inspect later.

Blind identity. Randomize presentation. Repeat the vote. Swap both answers. Average by candidate identity. Escalate close or inconsistent results.

Lock the rubric

Define factuality, task fit, clarity, and completeness before seeing the candidate identities. Store the judge model, prompt, temperature, and response text.

Run both orders

Evaluate A-first and B-first with repeated ballots. Convert every margin back to stable candidate identity before averaging.

Gate uncertainty

Send close margins, swap-dependent winners, low repetition stability, and high-stakes decisions to a blinded human reviewer.

Know what the signal is telling you.

Open each failure mode to see the diagnostic pattern and the correction that matches it.

Winner flipsPosition dominates.

A wins when shown first and B wins when shown first. The content signal is smaller than the slot effect.

Correction: pair both orders and average by identity. Long answer winsLength masquerades as quality.

The longer response wins in both orders despite weaker factuality or relevance.

Correction: make relevance explicit and cap unsupported detail. Votes scatterThe judge is unstable.

Repeated ballots disagree even when order and rubric are fixed.

Correction: report stability, not only the majority winner. Margin is closeAutomation has reached its limit.

A calibrated difference near zero is not permission to force a winner.

Correction: preserve the evidence and escalate to a human.

Read the primary research.

The lab's concepts come from peer-reviewed or preprint research on pairwise judging, position effects, evaluator inconsistency, and calibration.

Shi et al., 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-JudgeOpen paper Wang et al., 2023Large Language Models are not Fair EvaluatorsOpen paper Zheng et al., 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot ArenaOpen paper Stureborg et al., 2024Large Language Models are Inconsistent and Biased EvaluatorsOpen paper

Questions worth asking before launch.

Does swapping remove every evaluator bias?

No. It estimates and cancels a directional position effect when you average by candidate identity. It does not remove verbosity, self-preference, rubric ambiguity, knowledge gaps, or correlated errors.

Why repeat the same comparison?

Repeated ballots estimate stability. A large majority can still be systematically biased, so stability and fairness must be reported separately.

When should a human review the result?

Escalate when the order changes the winner, the calibrated margin is small, repeated votes scatter, evidence is disputed, or the outcome carries meaningful safety, legal, financial, or personal consequences.

Are the numbers in this lab benchmark results?

No. They are deterministic synthetic outputs designed to make evaluator mechanics inspectable. The response attributes, bias controls, and ballot variance are teaching inputs, not empirical claims about any named model.

Ship the protocol, not just the score.

Use this checklist to turn the experiment into an evaluation contract. Then explore how Superpowers can run repeatable agent workflows, build inspectable web tools, and cache controlled computer-use steps.

Build agent web tools Cache computer-use runs

0 of 4 ready

LLM-as-a-JudgeBias Lab