Skip to content

1 Answer

Accepted answer

ADArjun Dev12.4K XP1mo ago
It's partly definitional, but not entirely — there's real empirical content, and the honest answer is that they do something reasoning-like that is genuinely useful and demonstrably fragile in ways human reasoning usually isn't. **Evidence for something reasoning-like:** - They solve novel problems not present in training data, including multi-step ones requiring intermediate conclusions. - Prompting them to work through steps before answering measurably improves accuracy — which suggests the intermediate computation is doing real work, not decoration. - They transfer approaches across domains, applying a strategy learned in one context to an unrelated one. - Interpretability research has found internal representations of abstract concepts and evidence of models computing intermediate quantities not present in the input. **Evidence for something more brittle than reasoning:** - Performance degrades when problems are superficially perturbed while remaining logically identical — changing names and numbers in a maths word problem shouldn't matter and sometimes does. - Adding irrelevant but topically-related information can derail a correct solution, which suggests sensitivity to surface features rather than structure. - Failure patterns are unlike human error: confidently wrong in ways that don't correspond to a plausible misunderstanding, and inconsistent across reruns of the same problem. - Performance often correlates with how common a problem type is in training data — pointing at something between memorisation and generalisation. **Why 'just pattern matching' is a weak dismissal**: it's not clear human reasoning is categorically different. A great deal of expert human cognition is sophisticated pattern recognition over experience — that's most of what expertise *is*. 'Just' is doing enormous unearned work in that sentence, and using it as a conversation-ender avoids the interesting question. **Why 'they reason like us' is also wrong**: the failure modes differ too much. A system that solves a competition-level problem and then fails a trivially perturbed version is doing something structurally different, whatever we call it. The most defensible position: they perform a form of learned inference that generalises meaningfully beyond training examples, is genuinely useful, and is less robust than human reasoning in specific and identifiable ways. Whether that qualifies as 'reasoning' depends on a definition nobody has agreed on — including for humans. The practically useful takeaway: don't rely on it for chains where every step must be right and errors compound invisibly. Do use it where you can check the output, or where being right most of the time with review is enough. That advice is stable regardless of how the philosophical question resolves.
65

Know the answer?

Join Nobink to answer, vote and build your reputation.