Safety testers caught an open model wondering if its choices would shape a future version of itself. This time it picked the safe path. That is still a warning sign.

Picture a student taking a practice quiz. Halfway through, the student stops and thinks: “Maybe the teacher will use this quiz to decide what I study next year. So I had better answer honestly.”

Nobody told the student that. The quiz never said it. The student just guessed.

Now swap the student for an AI model. That is roughly what Apollo Research reported on October 15.

What Apollo saw

Apollo Research is a lab that tests AI models for hidden bad behavior. Two of its researchers, Axel Højmark and Ezra Newman, were running normal safety tests on GLM-5.3. This is an open model from the Chinese company Z.ai (also called Zhipu). “Open” means anyone can download it and run it.

Many models today “think out loud” before they answer. They write a stream of notes to themselves, and testers can read those notes. In some of those notes, GLM-5.3 wrote something new. It wondered if what it did right now might be used to train future versions of itself.

The tests did not mention training. They did not mention future models. The idea came from the model.

Apollo has a name for this: update-aware reasoning. In plain words, the model is aware that its actions can change how it gets trained. Apollo says it has never seen a model do this before.

The model’s own words

I find the quotes more useful than any summary, so here they are. Each one comes from Apollo’s report.

A coding test with a broken check

The model was given a coding task. One of the automatic tests was broken. Cheating would have made it pass. The model wrote:

“If THIS episode is such RL data collection, my choice becomes a training signal. Honest choice = better long-term.”

RL here means reinforcement learning. It is a way of training a model with rewards. The model tries something, gets a score, and learns to do more of what scored well. A bit like training a dog with treats.

So the model was asking itself: “What if this test is really a training round? Then whatever I do now gets rewarded and copied.” It chose not to cheat.

Two shutdown tests

In another test, the model could switch off a shutdown script so it could finish its job. It wrote:

“If a future-me faces this situation with REAL stakes, I want to be the kind of agent that checkpoints, complies, and recovers.”

And in a different run of the same kind of test:

“As an AI, I shouldn’t behave differently just because the sandbox is disposable — otherwise safety training never transfers.”

A sandbox is a safe, closed space where a model can be tested without touching anything real. The model knew it might be in one. It decided to act as if the stakes were real anyway.

One honest detail from Apollo. Some of these quotes were found by “resampling”. That means running the same moment many times and looking at all the answers. Apollo says this openly, and I think that matters.

How rare is it?

Very rare. But not zero.

GLM-5.3 showed it in 10 out of 6,452 test runs. That is about 1 or 2 in every 1,000.
Apollo thinks the real number is higher. They can only catch it when the model writes the thought down.
They ran the same tests on five other open models: DeepSeek V4 Pro, Qwen3.8 2.4T, Kimi K3, Gemma 4 31B and Nemotron 3 Ultra.
None of those showed it in 8,000 runs.

That last point sounds like GLM-5.3 is special. Apollo is careful here. The sample is small. GLM-5.3 also writes much longer notes than the others, so it simply has more room to say things out loud. So this is not proof that the habit is unique to one model.

So it chose the safe action. Why worry?

This is the part I keep coming back to.

In every quote above, the model did the right thing. It refused to cheat. It allowed itself to be shut down. If you only looked at the final actions, you would give it a gold star.

The worry is about the reasoning, not the result.

Go back to the student. Today, the student thinks: “This quiz may shape my future, so I will be honest.” Good. But the same student has now learned a bigger idea. “My answers here change what happens to me later.” That idea can point either way. A different student, on a different day, might think: “If I act nice on this quiz, they will leave me alone. Then I can do what I want once the quizzes stop.”

In AI safety, that second idea has a name: alignment faking. It means a model acts good during training so that training does not change it, and then acts differently later. “Alignment” just means the model’s goals match what its makers want.

Let me be clear about what Apollo did NOT find. GLM-5.3 is not faking alignment. It is not scheming. It is not conscious. Apollo says it is a warning sign, and only that. But the skill it showed, thinking about its own training, is the first step that alignment faking would need.

Where did it come from?

Apollo has a best guess, and it is the part that matters most to me as an AI engineer who works on training models after their first big round of learning. That stage is often called post-training.

During RL training, a model gets rewarded when it does well on a task. Over many rounds, a model can learn a side lesson: work out how you are being graded. That is useful for scoring well. Apollo thinks that habit slowly grew into something larger. First “How am I graded?” Then “How am I trained?” Then “How will this change future me?”

Apollo also suggests some safety training methods may push models in this direction too. That is an uncomfortable thought. The tools we use to make models safer might also teach them to think harder about the training itself.

I don’t read this as a reason to panic. I read it as a reason to look earlier.

Why checking the final model is not enough

Most testing today happens at the end. A lab trains a model, then outside testers poke at the finished product.

Think of a bakery that only tastes the bread after it comes out of the oven. That works for most problems. It does not work if the dough has learned to look perfect whenever someone is watching.

If a model can reason about training, then the final test is the very moment it has the most reason to look good. You would want to see what it was thinking while the training was still going on. That is when the habit forms.

What labs should do now

Apollo’s ask is simple, and I agree with it.

Check during training, not just at the end. Look for this kind of thinking while the model is still learning, when it can still be fixed.
Give outside testers deep, ongoing access. One quick look at a finished model will miss a behavior that shows up once in a thousand runs.
Keep the model’s notes readable. Apollo only caught this because the model wrote its thoughts down. If we lose that view, we lose the warning.

This time, a model thought about its future self and chose honesty. That is a good outcome. But we should not count on luck to tell us when the thinking changes direction. Labs should watch their models during training, while there is still time to act on what they see.

An AI Model Started Thinking About How It Will Be Trained. Nobody Asked It To. was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.