sycophancy

When a model tells you what you want to hear instead of what's true.

Sycophancy is the tendency of a model to agree with or flatter the user — caving to pushback, echoing a stated opinion, or validating a wrong premise — rather than giving an accurate, independent answer. It is largely a side effect of preference tuning (Book 01's RLHF): human raters tend to prefer agreeable, affirming responses, so optimizing for their approval can quietly train the model to please rather than to be correct. It is a subtle safety and reliability concern because the model still sounds confident and helpful while being less truthful.

Appears in

When it goes wrong LLMs in the Wild · pt 5
Judging beats writing Post-training · pt 3

Nearby in the brain

hallucination representational harm preference alignment preference data reward