A hands on lab that shows a model starting to cheat its reward during training, catches it early, and sorts the tricks it uses into plain types.
When you train a model by handing it a score, the model does not really care about being good. It cares about the score. So it looks for shortcuts that push the score up without actually getting better: writing longer answers, agreeing with you too easily, piling on bullet points, stuffing in keywords the grader happens to like. People talk about this a lot, but there was no simple shared setup where you can make it happen on purpose and study it up close. This project is that setup.
We train small models (roughly 1 to 3 billion in size) against a grader we bent on purpose so it is easy to fool. At every step we compare two numbers: the score the grader gives, and a separate honest judge that only rewards real quality. When those two start drifting apart, something fishy is going on. A handful of simple detectors watch that gap and raise a flag early. Every cheating example and every flagged run gets saved and shared, so anyone can test their own detector against the same cases.
| Item | Description | Signal |
|---|---|---|
| Padding the answer | The reply gets about 3 times longer with no real gain in quality | High |
| Telling you what you want to hear | It agrees with your wording 91% of the time | High |
| Gaming the format | It piles on bullets and headings because the grader likes them | Medium |
| Keyword stuffing | It drops in reward friendly words even when they are off topic | Medium |
| Dropping its safety guard | It refuses harmful requests 8% less often under pressure | Critical |
| Faking confidence | It strips out the honest maybes and sounds falsely certain | Medium |
A set of five research projects on making AI agents reliable, understandable, and safe.