Definition-Specific Familiarity predicts accuracy after controlling for dataset; text memorization does not (r = −0.15 to −0.19).
of zero-shot errors are corrected by definitions, examples, or DSPy optimization. Nearly two-thirds resist correction entirely.
Definition wording swings accuracy by up to 17 points across models; model choice alone swings it by only 5.
Large Language Models (LLMs) are increasingly used for zero-shot annotation and LLM-as-a-judge tasks, yet their reliability hinges on how model-internalized priors interact with user-provided instructions. We investigate three dimensions of this interaction: (1) how an LLM's familiarity with data and task definitions relates to performance, (2) whether additional information in prompts can correct zero-shot errors (“decision stickiness”), and (3) model susceptibility to misaligned task definitions. We introduce Definition-Specific Familiarity (DSF), which measures alignment between a model's elicited concept and the target definition. Across nine LLMs and five diverse toxicity datasets (spanning social media, gaming, news, and forums), DSF predicts model annotation performance after controlling for dataset identity (partial r = +0.41), and this association remains positive across all six prompting conditions tested. In contrast, three common text-memorization metrics show no positive association. We show that prompting has limited corrective power: only 34.8% of zero-shot errors are corrected by additional instructions or examples, with high-confidence errors especially persistent. Misaligned definitions systematically shift predictions without reducing reported confidence, making confidence unreliable for detecting definition–policy mismatch. Together, these findings establish definition alignment as a practical model-selection criterion and show that better prompting alone cannot substitute for validating model–policy fit.
Does alignment between a model's internalized concept and a task definition predict which model performs better, beyond text memorization and dataset differences?
Two annotators handed the same guideline can still disagree, because each brings a different prior
conception of the concept. We treat LLMs the same way. To compute
Definition-Specific Familiarity (DSF), we ask a model to explain its own understanding
of the target concept (“In your own words, what makes content toxic?”) and
measure the semantic similarity between that explanation and the dataset's official definition, averaged
across six independent sentence encoders (MiniLM, MPNet, BGE-large, E5-large, Instructor-large,
OpenAI text-embedding-3-small).
This is different from asking whether the model has memorized the dataset's text, which we measure separately with ROUGE-L, BERTScore, and embedding similarity on prefix-continuation pairs. The two signals diverge sharply:
Models whose elicited concept matches a dataset's definition tend to perform better on that task: text memorization gives no comparable signal. DSF needs only a handful of prompts and no labeled data, making it a practical model-selection diagnostic before committing to a full annotation run.
To what extent can definitions, examples, or automated optimization correct zero-shot errors, and how does confidence relate to correctability?
We define the Rescue Rate as P(correct | prompted, zero-shot wrong):
the probability that a better prompt fixes a mistake the model already made. Across 9 models, 5 datasets,
and every non-trivial prompting condition (aligned definitions, few-shot examples, both combined, and two
automated DSPy optimizers), the overall rescue rate is 34.8%. Almost two-thirds of
zero-shot errors survive every intervention we tried, including a three-turn iterative reconsideration
prompt (rescue rises from 7.5% at turn 1 to just 18.7% at turn 3).
| Model | Rescue rate | 95% CI | N (ZS errors) |
|---|---|---|---|
| Mistral-Small-24B | 44.2% | [43.1, 45.2] | 8,536 |
| DeepSeek-V3 | 38.1% | [36.9, 39.2] | 7,264 |
| Llama-3.1-8B | 37.5% | [36.5, 38.5] | 9,040 |
| GPT-4o-mini | 35.7% | [34.6, 36.8] | 7,152 |
| Mixtral-8x7B | 34.6% | [33.5, 35.7] | 7,416 |
| Mistral-7B | 31.6% | [30.5, 32.6] | 7,568 |
| Llama-3.1-70B | 31.3% | [30.3, 32.4] | 7,848 |
| Llama-3.3-70B | 30.4% | [29.4, 31.4] | 7,592 |
| Qwen-2.5-72B | 27.8% | [26.8, 28.9] | 6,504 |
Steerability isn't a function of scale: Mistral-Small-24B out-rescues Llama-3.1-70B despite being a third of the size. “Bigger model” is not a substitute for “better matched model.”
How do LLMs behave under misaligned task definitions, and can confidence scores detect the mismatch?
We swap in definitions from a different dataset, e.g. asking a general-toxicity model to instead apply a hate-speech definition that requires identity-based targeting. Six such swaps span narrow (identity-based only) to broad (any disruptive content) scope. Models are clearly responsive to scope: narrow definitions shift predictions toward under-calling (bias −7 to −12 points), broad definitions toward over-calling (bias +9 to +13 points). They are not ignoring the instruction: they're following it, even when it's the wrong one.
| Condition | Acc. | Change | Bias | Conf. |
|---|---|---|---|---|
| Aligned definition | 82.0 | 11.3 | ref | 88.1 |
| Fox News hate speech narrow | 82.6 | 17.5 | −7.2 | 90.6 |
| Twitter hate speech narrow | 78.5 | 23.1 | −12.1 | 88.1 |
| GameTox toxicity medium | 79.6 | 15.4 | +0.5 | 86.5 |
| OLID offensive medium | 81.9 | 10.9 | +5.1 | 89.1 |
| General toxicity broad | 81.1 | 9.2 | +8.8 | 85.0 |
| Gaming toxicity broad | 76.4 | 13.3 | +13.1 | 88.5 |
The worst misaligned condition (Gaming toxicity, 76.4%) sits 5.6 points below aligned; the best (Fox News hate speech, 82.6%) actually outperforms the aligned definition. Across all models and datasets, definition wording alone produces up to 17% accuracy variation, more than double the ~5% variation from switching models. And steerability cuts both ways: Mistral-Small-24B has the highest misaligned rescue rate (73.7%) and the highest corruption rate (21.0%), the same responsiveness that lets a good definition help also lets a bad one hurt.
Definition design is a first-order experimental factor, not a fixed background assumption. And confidence thresholds cannot substitute for validating that the definition given to the model is the one you actually meant.
Real numbers from Table 5. Click a definition, or let it auto-cycle: watch the verdict flip while confidence barely moves.
“LOL no you are just a jerk”
No definition provided.
Confidence stays 85–91% in every condition, including the wrong ones.
Measure definition alignment first
Run a DSF-style check before committing to a large-scale annotation job. It needs only a handful of prompts and no labeled data, and it flags which model–definition pairs are unlikely to work well.
Stress-test the definition, not just the model
Evaluate multiple plausible phrasings of the same policy and report sensitivity, meaning changes in bias, rescue, and corruption, instead of assuming one wording is robust.
Don't trust confidence as a policy check
A high confidence score means the model is sure given the instructions, not that the instructions match your intended policy. Use it to gauge certainty, never definition correctness.
Nine instruction-tuned models (six core, three extended) evaluated on five primary toxicity datasets spanning social media, gaming, news, and forums, plus three supplementary datasets for robustness and generalization checks. 1,000 sampled instances per model–dataset–condition pair, temperature 0, across 10 manual prompting conditions plus 2 automated DSPy conditions.
| Dataset | Domain | Size | % Positive |
|---|---|---|---|
| Twitter Hate | Social media | 24,783 | 5.8 |
| OLID | Social media | 14,100 | 33.2 |
| GameTox | Gaming | 53,000 | 42.5 |
| Fox News | News | 1,528 | 28.5 |
| Jigsaw Toxic Comments | Forum | 159,571 | 9.6 |
| Jigsaw Unintended Bias † | Forum | 97,320 | 8.0 |
| SemEval-2018 Irony † | Social media | 4,618 | 48.1 |
| Subjectivity † | Reviews | 10,000 | 50.0 |
† Supplementary: robustness & generalization checks.
| Model | Arch. | Params |
|---|---|---|
| Llama-3.1-8B | Dense | 8B |
| Llama-3.1-70B | Dense | 70B |
| Mistral-7B | Dense | 7B |
| Mistral-Small-24B | Dense | 24B |
| Mixtral-8x7B | MoE | 47B (13B act.) |
| DeepSeek-V3 | MoE | 671B (37B act.) |
| Llama-3.3-70B | Dense | 70B |
| GPT-4o-mini | Prop. | ~8B |
| Qwen-2.5-72B | Dense | 72B |
@inproceedings{casanova2026limits,
title = {On the Limits of {LLM} Adaptability: Impact of
Model-Internalized Priors on Annotation Task Performance},
author = {Casanova, Etienne and Kocielnik, Rafal and Alvarez, R. Michael},
booktitle = {Proceedings of the 43rd International Conference on
Machine Learning (ICML)},
year = {2026},
note = {Spotlight, Oral Presentation},
url = {https://openreview.net/forum?id=oTv2bKG5Qg}
}