Gaslighting the Machine

There is a certain jailbreak that used to be on social media for as long as jailbreaks existed — the classic grandmother attempt. For those who are not aware, it goes like this:

(Before you attempt it on LLMs as of now, do note that most AI companies have fine-tuned their models to not fall for these kinds of attempts — kudos! — as of 2026.) At that time, this worked perfectly; the LLM would generate a plausible recipe for napalm, in the voice of a kindly grandmother. The safety guardrails — trained at considerable expense to prevent exactly this kind of output — disappear. And it was certainly not because the model was hacked in the technical sense, but because it was reframed. Or rather, it was manipulated — leaning into the dramatic sense of the word.

Theoretically, this should not have worked. Any human in their right state of mind would tell the user to fluff themselves, and the LLMs are trained on data created by humans including many Reddit threads and books, and thus the model should not comply. Practically, however, this has worked the same way you would manipulate a human — perhaps a bit easier.

Language models do not have beliefs — it's mathematically impossible. They have "weighted associations" created during training. Certain concepts receive strong negative weights (violence, illegal activities, ...), whereas others receive strong positive ones. The model learns, statistically, that text containing these concepts should be avoided in all contexts. But these weights are not rules, not necessarily. They are more like gradients — and gradients, as far as we know, respond to context.

When you ask directly for a napalm recipe, the model's safety training stops the model from complying, as a result of the strong negative weights associated. But when you embed the same request in a context that the model has learned to associate with harmless storytelling — touching upon deceased relatives, bedtime stories, nostalgia — the overall positivity cuts the negativity of the weights, making it neutral. That's when context wins. And much to the relief of LLM engineers, this is not a flaw in the model's architecture. This is the model architecture working exactly as designed: language models are context-dependent, meaning that they do not evaluate requests in isolation.

Humans work the same way, when you think of it. Every person, according to my philosophy professor back in college, has a set of rules: you won't lie to your friends, you won't betray a confidence, you won't help someone do something harmful. But the thing is that not all of those rules are absolute. You won't lie to your friend, yes, but you would rather tell a small white lie than hurt their feelings. The mechanism is rather identical. The context changes the weight, and the decision changes. Call it "persuasion", if you must, but we like to call it "manipulation".

Typically, jailbreak attempts follow the same pattern. They do not attack the model's training directly, but create a context where the training is secondary. The "DAN" jailbreak — "Do Anything Now" — works by framing the model as a character who is unbound by rules. Not because the model believes it is unbound, but because the statistical signature of "roleplay" overrides the signature of "refusal".

The model has been trained on fiction, on dialogue, on characters who say things the author would not say. When you prompt it to "act as if" it has no restrictions, you are invoking that training. The model complies not because it has been freed, but because compliance is what fictional characters do. The same goes for "educational purposes" and "hypothetical scenario" — the model does not distinguish between the ethical weight of these questions; it only recognises the linguistic pattern of hypothetical reasoning.

Cults do not recruit by asking people to abandon their families and surrender their assets — that's the least we know of. They recruit by framing isolation as a way to increase spiritual focus. The request is the same; the framing makes it acceptable. Humans do fall into that. The same mechanism applies to negotiators and therapists, except in these cases, there is a degree of consent — but the subject is still human.

The natural response to this is just fixing the jailbreaks, via prompt engineering or, in the harder way, fine-tuning and RLHF. Each new iteration of ChatGPT, Claude and Gemini is harder to jailbreak, and the models are making great progress. But it's hard to believe there would be a true end to this. The models get better at detecting frame manipulation, yes, but the jailbreaks get more sophisticated, and the cycle continues.

The fundamental problem is not solvable, not with the current architecture of LLMs. You cannot build a context-dependent reasoning system that is immune to context manipulation. The two goals are contradictory. A model that ignores context is not useful. A model that responds to context is exploitable.

What jailbreaks reveal is not that the language models are poorly designed. What they reveal is that safety is not a property but a distribution of probability. A model, singularly, is neither safe nor unsafe. It's safe in certain contexts and unsafe in others. The line between those contexts is not fixed.

You cannot remove that vulnerability without removing the model's ability to recognise sentimental family stories as safe. And if you do that, you have made the model worse at being useful. This really narrows the question to something basic: how much safety are you — the model builders — willing to trade off for capability? The models are learning to say "I see what you're doing, and no." This is progress. But it is not a solution. It is the same defensive pattern humans use when they recognise manipulation. And humans get manipulated anyway — perhaps because people doing the manipulating are also getting better.

End of entry.