Machine Learning Blog | ML@CMU | Carnegie Mellon University

Researchers at Carnegie Mellon tested how major LLMs respond to age-sensitive questions like "Is Santa Claus real?" across different ages, languages, and contexts. They found significant variations: GPT-4o consistently says yes regardless of age, Claude tells the truth early (around age 6), while Gemini waits until teenage years. The study expanded to developmental milestones and World Values Survey questions, revealing that LLMs exhibit age-based and cultural biases in their responses. Language context dramatically affects answers—for example, GPT-4o tells French speakers to listen to parents until age 20 but Spanish speakers only until age 10. The research highlights how LLMs make invisible assumptions about user demographics and adjust responses accordingly, sometimes misaligning with actual cultural survey data.