How Much Do You Trust A Dog? Quantifying Hint-Following in Reasoning Language Models

Summary

Researchers measured how hints attributed to a professor, a friend, or a dog shift answers from three open-source model families (Qwen, Gemma, SmolLM) on MMLU questions. They found the hint produces a roughly constant mean shift in the hinted answer's log-odds, largely independent of source, and the shift persists even when the source is declared inaccurate or explicitly distrusted.