When one AI teaches another, it helps to check how the teacher learned, not just what it ended up knowing.

Training an AI model to reason well costs a lot. So here is a fair question. If we already paid for that training once, can we pass the benefit to other models for much less?

A new paper from a team at Alibaba Cloud Computing tries to answer it. The method is called Waypoint-OPD. On October 16 it was the most upvoted paper on Hugging Face’s daily list, with 667 upvotes. Hugging Face’s paper account also put it first in its “Top AI Papers of the Week” post.

One honest note before we start. The upvotes are big, but the real talk around it is quiet. The top post about it on X had 23 likes. The code had only a handful of GitHub stars when I checked. So this is a paper getting attention, not a paper everyone is arguing about yet.

The core idea fits in one line: when one model teaches another, look at the middle of the teacher’s training, not just the start and the end.

Paper: Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation (Hou, Jiang, Quan, Li, Feng, Liu, Zhang; 2026).

A few plain words first

A model here means a large language model, the kind of AI that writes text. It writes one small piece at a time. Each piece is a token, often a word or part of a word. At every step, the model gives each possible next token a chance of being picked.

Reinforcement learning (RL) is training by trial and reward. The model tries a math problem. A checker says right or wrong. The model then makes the moves that led to right answers a bit more likely. Do this thousands of times and the model gets much better. It is also slow and costly, because the model has to write and check a huge pile of answers.

Distillation means one model (the teacher) helps train another (the student). Think of a senior engineer guiding a new hire.

On-policy means the student practises on its own answers. The student writes something, and the teacher gives feedback on that exact answer, step by step. It’s a coach watching you play, rather than you watching tapes of the coach.

A checkpoint is a saved copy of a model at one moment in training. Like a photo of the model on that day.

The old trick: copy what training changed

This paper builds on an earlier method called Direct-OPD.

You take a small model and run RL on it. Small models are cheaper to train. Now you have two checkpoints: the small model before RL, and after.

You could tell a bigger student to just copy the trained small model. But the small model has small-model limits. Copy all of it and you copy its weak spots too.

So Direct-OPD looks only at the difference between “before RL” and “after RL”. For each possible next token, it asks one thing. Did RL make this choice more likely, or less? That difference is “what RL taught”. The bigger student gets rewarded for moving the same way, on its own answers.

Picture a cooking student who skips the chef’s whole style. They copy only what the chef changed after a year of practice. Less salt here. More heat there.

Where the before-and-after picture fails

The new paper spots a weak point. The before-to-after difference is one number per choice. The same number can come from very different stories.

Story A: Early in training, the teacher learned to favour a choice a little. Later, a little more. Steady progress.

Story B: Early on, the teacher favoured the choice a lot. Later, it pulled back and undid part of that.

Both can land on the same final number. The paper’s tiny example: if the total change is “d”, then “half of d, then half of d” adds up to d. So does “two d, then minus one d”. From the start and end alone, you can’t tell them apart.

Yet they mean different things. Story A looks like a lesson the teacher settled on. Story B looks like the teacher changed its mind. Should a student learn both with the same strength? The authors bet no.

The fix: keep a photo from the middle

The method keeps three checkpoints from one teacher training run:

Initial: before RL.
Waypoint: halfway through. In the paper, step 600 out of 1,200.
Final: the end of RL.

A waypoint is a stop on a journey. It shows you the path, not just the two ends.

That gives two changes to compare. The early change runs from Initial to Waypoint. The late change runs from Waypoint to Final.

Here is how the student learns:

The student writes an answer to a practice problem, one token at a time.
At each step, it takes its top 16 guesses for the next token.
All three teacher snapshots score those 16 guesses.
The early and late changes get compared. Did both push the same guesses up and down? Or did the late change go the other way?
A “gate” is set. Think of it as a volume knob. Same direction: the knob turns up toward full. Opposite directions: it drops to zero. Anything in between gets partial volume.
The usual lesson is taught through the knob. The student still learns the full before-to-after change. The middle snapshot never changes what is taught. It only decides how loudly.

Two small extras. Knob values tend to drift lower over time, so each batch is rescaled to keep the average knob near 0.8. And a “leash” keeps the student from wandering too far from where it started. That’s a normal safety rope in this kind of training.

The extra cost is small: one more frozen teacher snapshot to ask. No new teacher training. No right-or-wrong checker running while the student trains.

The test: a small teacher, three bigger students

This is the fun part. The teacher is smaller than every student.

Teacher: Qwen3–4B. “4B” means about 4 billion parameters, the adjustable numbers inside a model. Bigger usually means more capable.
Students: Qwen3–8B, Qwen3–14B and Qwen3–30B-A3B. The last one is a “mixture of experts” model. It has many sub-parts and uses only some of them for each token.

The teacher got 1,200 steps of RL, once for math and once for code. Each student then got only 100 steps with the new method.

Math tests came from hard contests: AIME, HMMT and AMC problems from 2023 to 2026. Code tests were MBPP, MBPP+ and three versions of LiveCodeBench. Each math problem was tried 32 times, and the score is the average success rate. Code got 8 tries per problem.

The rivals were plain on-policy distillation, a method called OPD², and the old Direct-OPD.

What they found

It won on average every time. In all six settings (three students, two subjects), Waypoint-OPD had the top average.

On math, against Direct-OPD:

Qwen3–8B: 53.96 → 56.14 (+2.18 points)
Qwen3–14B: 57.61 → 59.20 (+1.59)
Qwen3–30B-A3B: 55.96 → 57.85 (+1.89)

It was better on 19 of the 21 single math tests. For scale, the 8B student started at 26.04 before any teaching.

On code, the gains over Direct-OPD were +2.66, +1.52 and +1.19 points. It won on all 15 code tests.

The students beat their teacher. After its own RL, the 4B teacher scored 56.13 on math. The 14B student hit 59.20 and beat it on all seven math tests. The 30B-A3B student reached 57.85. The 8B student tied at 56.14. On code, the 14B and 30B-A3B students also passed the teacher’s 55.37.

Where the knob sits matters. The authors broke one part at a time on the 8B math student:

No knob, always full volume: 56.14 drops to 53.50.
Knob values shuffled randomly between steps: down to 51.20.
No batch rescaling: down a little, to 55.22.

The shuffle result is the one I keep thinking about. Same knob values, wrong places, and it’s worse than no knob at all. So the gain isn’t from “teaching less overall”. It comes from each knob matching its own moment.

The exact middle point barely matters. Step 500, 600 or 700 as the waypoint gave 56.18, 56.14 and 56.13.

It wasn’t slower. In the 8B math run it finished about 3.7% faster: 8 hours 2 minutes against 8 hours 21 minutes. The students wrote shorter answers, and fewer hit the length limit (11.97% vs 14.83%).

The knob gets pickier over time. The average knob started at 0.81 and ended at 0.68. Steps with the knob fully off rose from about 7% to about 17%.

What to keep in mind

Good paper. Clear limits.

One model family. Everything is Qwen3, with a single 4B teacher. Teacher and student must split text into tokens the same way.
Only math and code. Both have clear right answers. Chat, writing and safety behaviour are untested.
No error bars. The gains are about 1 to 3 points, and there are no repeated runs. We can’t tell how much is noise.
One test does double duty. Each student’s best saved copy was picked using AIME 2024, which is also a reported math test. That gives every method a small boost there.
The break-one-part tests used only the 8B math student.
Agreement isn’t correctness. The authors say so. My own thought, which the paper doesn’t test: if a teacher steadily picked up a bad habit during RL, say a trick that fools the checker, the early and late changes would agree. The knob would let it through. This filter catches wobbly lessons, not wrong ones.

Why this matters

RL is now a big part of how strong AI models get made, and it’s expensive. Methods like this point to a cheaper path. Train once on a small model, then pass the lessons up to bigger ones.

The twist here is simple. Don’t just compare start and finish. Look at the path. Teach steady lessons loudly. Teach the back-and-forth ones softly, or skip them.

If you train models yourself, there’s a practical tip: save your middle checkpoints. They might make good teachers later.

There’s a bigger question too. Moving “what training changed” from one model to another means moving behaviour. Choosing which changes to move is a strong lever. This paper chooses by steadiness. The next step is harder: not “was this lesson steady?” but “was this lesson good?”

Code: github.com/hhhbxloveai/WaypointOPD

A Snapshot From the Middle: How a Small AI Model Taught Bigger Models to Beat It was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.