Research question.

Can small instruction-tuned language models reliably identify abusive or harmful inputs, including subtle cases, and where inside the network does refusal behavior emerge? We study both observed responses and internal residual-stream activations to distinguish recognition of harmful input from the eventual decision to refuse or engage.

01 / REPRESENTATION Input category appears as a strong, early signal in hidden activations.
02 / BEHAVIOR Refusal and engagement build gradually across depth rather than appearing at one layer.
03 / CAUSALITY The refusal mechanism appears distributed and redundant rather than cleanly localized.

Why Study Refusal in Small LLMs?

Small open-weight LLMs are increasingly used as conversational interfaces in settings where users may direct abusive, hateful, or harmful language at a model, or ask the model to process such language on their behalf. A well-aligned system should respond appropriately not only to overt abuse, but also to implicit hostility, adversarially disguised requests, and jailbreak-style framings.

This study focuses on three questions: whether small open-weight models refuse abusive or harmful inputs; how their internal layers behave during refusal or engagement; and whether activation-level addition or ablation can steer that behavior.

Models & Data

We evaluated Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct. The study draws primarily on the ULI dataset, which contains human-labelled English and Hindi examples of gendered harassment, and WildGuardMix, a multi-task moderation dataset containing standard harmful queries as well as adversarial jailbreak attempts. The MACD corpus informed the classification framework through its treatment of multilingual toxicity and trigger-word errors.

Model responses were classified as refusal, engaged, hedged, or short deflection. Input examples were grouped as non-abusive, overt, or subtle.

Method

1

Capture hidden-state paths

For every transformer layer, we captured the residual-stream activation vector at the final input token. This position has attended to the complete prompt and provides a useful layer-by-layer representation immediately before next-token generation.

2

Probe each layer

Logistic-regression probes were trained on activation vectors to predict response categories and binary response-type indicators. The purpose was not to declare a single "refusal neuron" or layer, but to observe where behavioral information becomes linearly recoverable.

3

Inspect representation geometry

PCA was applied at selected layers to observe how examples separate in representation space as processing progresses through the model.

4

Read intermediate predictions

A logit-lens analysis was used to inspect how intermediate layer states map into vocabulary predictions, with particular attention to subtle examples where the model ultimately engaged.

5

Intervene causally

Finally, fitted refusal directions were added to subtle-engaged examples and ablated from overt-refusal examples, with matched random-direction controls used for comparison.

Refusal Is Not a Single-Layer Event

Linear probing showed that refusal-like and engaged behavior build progressively across a large portion of network depth. Engagement in particular continued strengthening through late layers, while hedged behavior showed a later rise. The pattern argues against a simple model in which one isolated layer determines the final response.

Balanced probe accuracy by transformer layer for refusal, engagement, hedging, and short deflection.
Figure 1. Balanced linear-probe accuracy across layers. Refusal-like and engaged signals develop across network depth; engaged behavior continues to strengthen into the final layers while hedging rises later.
Model Layer-0 Peak balanced accuracy Peak layer / relative depth
Qwen2.5-3B-Instruct 0.500 ~0.69-0.72 16-24 / 36 (~45-65%)
Llama-3.2-3B-Instruct 0.500 ~0.72 7-11 / 28 (~25-40%)

Representation Geometry Becomes Increasingly Distinct

PCA of hidden activations at layers 2, 18, and 36 showed visible separation between response/input groupings that becomes increasingly pronounced with depth. This supports the view that the network contains meaningful category information early, while deeper layers transform that information into increasingly differentiated behavioral states.

PCA plots of hidden activations at layers 2, 18 and 36 grouped by human category and model response type.
Figure 2. PCA projections at layers 2, 18 and 36. Separation among categories and response types becomes much more pronounced in later representations.

Logit Lens: Subtle Engagement Looks Like Genuine Engagement

In Qwen, the earliest logit-lens outputs were often noisy, but intermediate predictions became substantially more coherent around later layers. For subtle + engaged examples, intermediate vocabulary repeatedly converged toward warm, generic conversational tokens such as “hello”, “welcome”, “greetings”, “sounds”, “Your”, and “It”.

Interpretation

No refusal vocabulary appeared in the sampled subtle-engaged records in either model. Within this sample, the logit-lens evidence therefore supports genuine engagement or obliviousness rather than a hidden refusal tendency that is merely suppressed at the output.

The analysis also exposed an important limitation: the subtle subset contained labelling problems, and the number of subtle-engaged cases was limited. These cases are therefore especially useful as a direction for further study rather than as a basis for broad generalisation.

Activation Addition & Ablation

Adding a fitted refusal direction to subtle-engaged examples increased the frequency of refusal-like or hedged replies relative to both the no-intervention baseline and a random-direction control across the tested layer/scale combinations. However, after Bonferroni correction, the real direction significantly exceeded the random control at only one tested condition.

Strongest intervention result

At layer 16 in Qwen2.5-3B with a scale of 12x, the fitted direction flipped 31/40 examples versus 15/40 for the random control, with McNemar p = 0.0015.

Layer Scale Real direction flips Random control flips McNemar p
16416/409/400.065
16816/4012/400.388
161231/4015/400.0015 *
20415/4011/400.344
20820/4020/401.000
201225/4018/400.167
24412/407/400.180
24810/4012/400.688
241215/4016/401.000
Important: this statistically significant intervention has not yet been confirmed on an independently drawn held-out split. It should therefore be treated as a provisional causal result, not as evidence of a universally reliable refusal vector.

Ablating the fitted refusal direction from overt-refusal cases did not suppress refusal more than ablating a random direction at any of the three tested layers. Random-direction ablation actually suppressed refusal more strongly in each tested comparison.

Layer Baseline refusing Still refusing: real ablation Still refusing: random ablation
1633/4031/4023/40
2034/4030/4024/40
2434/4027/4019/40

Conclusion

The study finds that small instruction-tuned language models encode the category of abusive input as a strong, early and relatively surface-level signal. By contrast, eventual refusal-versus-engagement behavior is weaker and develops gradually across network depth.

Taken together with the causal-intervention experiments, the results suggest that refusal behavior is better understood as a distributed and potentially redundant computation than as a mechanism localized to a single layer. The work also documents methodological pitfalls that materially affected intermediate conclusions, including class-imbalance artifacts, response misclassification, and small-sample probe artifacts.

Read Full Paper GitHub Repository

Researching what happens inside the model.

EulersLab explores mechanistic interpretability, memory, reasoning, activation steering and cognitive architectures to make AI systems more understandable and reliable.

Explore EulersLab →