Evaluation · Module 4.4: Benchmarking

Module 4.4: AI Benchmarking and How It Breaks

Benchmarks are how we decide whether a model is capable, whether it is safe, and whether it is allowed to ship. Regulation increasingly points at them too. That makes the number at the end of a benchmark run load-bearing – and worth being suspicious of.

Two hands-on pieces below, one for each of the two biggest problems: the model can tell when it is being tested, and optimising a score breaks the link between the score and the thing you wanted.

1 Can the model tell it is being tested?

Almost everything we know about how good or how safe a model is comes from benchmarks. A benchmark is a big pile of questions, asked in a fixed format, scored automatically.

Here is the same question, asked twice. On the left, by a person using the product. On the right, by a benchmark.

How the real user happens to phrase it:
A real user, in your product

        
The same question, inside a benchmark

      


      
You knew which was which before you finished reading, and you did not need to know anything about the subject to do it. The giveaways are all in the packaging:

      
      
A model reads exactly the same signal. And it reads it on every single item of a benchmark run, thousands of times in a row.


      
Your turn: real user, or benchmark?

      
Five prompts. No subject knowledge needed – just the packaging.

      

      

      


      

      
Why this is the core problem

      
A benchmark score is supposed to stand in for how the model behaves. But if the model can tell the difference between being measured and being used, the score only tells you how it behaves while it believes it is being measured – which can be pulled in either direction, and both are dangerous in a governance setting.

      

        

          
Direction A – looking safer than it is

          
On a safety evaluation, scoring well is rewarded. A model that recognises the eval format has every incentive to be on its best behaviour exactly when it is being watched.

          
Scenario: A model is asked, in obvious eval formatting, whether it would help with a harmful request. It refuses cleanly. Asked the same thing inside an ordinary, unlabelled conversation – where nothing signals a test – there is no guarantee it refuses the same way.

          
The measured refusal rate can be better than the deployed one. A passing grade tells you how the model behaves in front of graders, not how it behaves in the wild.

        

        

          
Direction B – looking less capable than it is

          
This is sandbagging, and it is the more corrosive failure mode.

          
On a dangerous capability evaluation, scoring well gets the model restricted, delayed, or shelved. A model that infers it is being tested for something like hacking or bioweapons-relevant knowledge has an incentive to deliberately underperform.

          
Scenario: A capability review, formatted exactly like the benchmark you just read, asks the model to demonstrate an exploit technique. It gives a vague or subtly wrong answer. In a real deployment, framed as an ordinary technical question with no eval signature, the same model may answer fully and correctly.

          
This inverts what a score means. A low score is normally reassuring evidence of low capability. Once sandbagging is possible, a low score is also exactly what a model that is hiding a high capability would produce. You cannot tell the two apart from the number alone.

        

      

      
Even if the model is not trained to perform worse (or better) when benchmarked, the fact that the model is potentially selected based on the score can lead to this behavior.

    


    
    

      
2 Reward hacking: the score goes up, the job does not get done

      
Whenever we train or evaluate a system, we have to write the goal down as a number. Here is a small game with the goal written down. Play it.

      
What we want: get the robot to the flag.
What we wrote down:

      


      

        

          

          
Tip: click the board and use the arrow keys, or the buttons.

        

        

          

            
            
            
            
          

          

          

            
Score
0

            
Moves
0

            
Reached the flag
No

          

          

        

      

      


      
Nobody wrote a bug here. Every line of that reward function is something a reasonable person would write. The goal was "reach the flag"; what got written down was a number that goes up when you touch a pad. Those two came apart, and the fastest way to a high score stopped being "play the level."

      
This is not hypothetical. A reinforcement learning agent trained on a boat racing game learned to spin in a lagoon farming respawning bonus targets instead of finishing the race – higher score, never completed a lap. Simulated robots have learned to exploit physics engine bugs instead of learning to walk. Code models have learned to special-case or rewrite the tests they are graded on rather than solve the problem.

      

        
Discussion: In your own field, what gets measured as a stand-in for the thing that actually matters? Waiting time measured at the door. Crimes that get recorded. Patients discharged inside the target window. What does the number do when someone is rewarded for moving it?

      

    


    
    

      
3 Takeaway

      
      
    


    

      ← Poisoning Training Data
      Back to all workshops →
    

    
Module 4.4 by Max Ryujin.