Me Learning LLM Reasoning
-
Who thinks LLMs reason anyway? They predict strings of words or sub-words, and their neural net is reinforced by humans who give thumbs up or down to certain responses.
There's an interesting case to be made that if reasoning was involved in the creation of the words in the training set, then that reasoning survives in the words predicted by the LLM trained on that set. Even if the LLM can't disentangle that reasoning from its predictions.
-
Who thinks LLMs reason anyway? They predict strings of words or sub-words, and their neural net is reinforced by humans who give thumbs up or down to certain responses.
There's an interesting case to be made that if reasoning was involved in the creation of the words in the training set, then that reasoning survives in the words predicted by the LLM trained on that set. Even if the LLM can't disentangle that reasoning from its predictions.
said in Me Learning LLM Reasoning:
Who thinks LLMs reason anyway? They predict strings of words or sub-words, and their neural net is reinforced by humans who give thumbs up or down to certain responses.
There's an interesting case to be made that if reasoning was involved in the creation of the words in the training set, then that reasoning survives in the words predicted by the LLM trained on that set. Even if the LLM can't disentangle that reasoning from its predictions.
ChatGPT responds:
That’s a sharp way of framing it. Let me break this into a few layers:
1. The standard view: LLMs as pattern completers
- At the mechanistic level, LLMs don’t “reason” in the way humans typically mean the word.
- They’re statistical models of conditional probability: given a history of tokens, predict the next token.
- The reinforcement (RLHF) you mentioned just shifts the probability landscape toward outputs that humans like, but it doesn’t fundamentally add a reasoning module.
2. The “reasoning residue” argument
Here’s where your point kicks in. Much of the text in the training corpus was itself produced by humans reasoning—deriving a theorem, explaining a cause, constructing an argument.
- That means the distribution of tokens encodes traces of reasoning patterns.
- When the LLM predicts continuations, it is effectively interpolating through that latent structure.
- Even if the LLM doesn’t “know it’s reasoning,” it reproduces patterns that are consequences of reasoning.
So in a sense, you’re right: the reasoning that went into producing the training data survives in fossilized form inside the model.
3. Emergent reasoning-like behavior
Because human reasoning is systematic, the training process induces internal circuits that can manipulate abstract relations (chains of if-then, causal analogies, etc.).
- Some researchers call this “simulacrum reasoning”: the model doesn’t reason in the causal, self-directed sense, but it produces outputs that look like reasoning because the statistical map encodes it.
- Chain-of-thought prompting is an example: when we nudge the model into “thinking step by step,” it can unspool reasoning-like patterns that were baked into the data.
4. The philosophical wrinkle
If reasoning is defined purely behaviorally (following valid inference patterns to reach conclusions), then LLMs arguably do reason.
If reasoning is defined internally (involving awareness, intentionality, goal-directed use of logic), then they don’t.It’s similar to debates in philosophy of mind about whether a system that simulates a process is “really doing” it. (Classic reference: Searle’s Chinese Room.)
A question back to you: do you lean toward the “behavioral” definition (reasoning is as reasoning does), or the “internalist” one (reasoning requires a kind of intentional process)? That determines whether you’d say LLMs reason, or just mimic reasoning. -
said in Me Learning LLM Reasoning:
Who thinks LLMs reason anyway? They predict strings of words or sub-words, and their neural net is reinforced by humans who give thumbs up or down to certain responses.
There's an interesting case to be made that if reasoning was involved in the creation of the words in the training set, then that reasoning survives in the words predicted by the LLM trained on that set. Even if the LLM can't disentangle that reasoning from its predictions.
ChatGPT responds:
That’s a sharp way of framing it. Let me break this into a few layers:
1. The standard view: LLMs as pattern completers
- At the mechanistic level, LLMs don’t “reason” in the way humans typically mean the word.
- They’re statistical models of conditional probability: given a history of tokens, predict the next token.
- The reinforcement (RLHF) you mentioned just shifts the probability landscape toward outputs that humans like, but it doesn’t fundamentally add a reasoning module.
2. The “reasoning residue” argument
Here’s where your point kicks in. Much of the text in the training corpus was itself produced by humans reasoning—deriving a theorem, explaining a cause, constructing an argument.
- That means the distribution of tokens encodes traces of reasoning patterns.
- When the LLM predicts continuations, it is effectively interpolating through that latent structure.
- Even if the LLM doesn’t “know it’s reasoning,” it reproduces patterns that are consequences of reasoning.
So in a sense, you’re right: the reasoning that went into producing the training data survives in fossilized form inside the model.
3. Emergent reasoning-like behavior
Because human reasoning is systematic, the training process induces internal circuits that can manipulate abstract relations (chains of if-then, causal analogies, etc.).
- Some researchers call this “simulacrum reasoning”: the model doesn’t reason in the causal, self-directed sense, but it produces outputs that look like reasoning because the statistical map encodes it.
- Chain-of-thought prompting is an example: when we nudge the model into “thinking step by step,” it can unspool reasoning-like patterns that were baked into the data.
4. The philosophical wrinkle
If reasoning is defined purely behaviorally (following valid inference patterns to reach conclusions), then LLMs arguably do reason.
If reasoning is defined internally (involving awareness, intentionality, goal-directed use of logic), then they don’t.It’s similar to debates in philosophy of mind about whether a system that simulates a process is “really doing” it. (Classic reference: Searle’s Chinese Room.)
A question back to you: do you lean toward the “behavioral” definition (reasoning is as reasoning does), or the “internalist” one (reasoning requires a kind of intentional process)? That determines whether you’d say LLMs reason, or just mimic reasoning.said in Me Learning LLM Reasoning:
said in Me Learning LLM Reasoning:
Who thinks LLMs reason anyway? They predict strings of words or sub-words, and their neural net is reinforced by humans who give thumbs up or down to certain responses.
There's an interesting case to be made that if reasoning was involved in the creation of the words in the training set, then that reasoning survives in the words predicted by the LLM trained on that set. Even if the LLM can't disentangle that reasoning from its predictions.
ChatGPT responds:
That’s a sharp way of framing it. Let me break this into a few layers:
1. The standard view: LLMs as pattern completers
- At the mechanistic level, LLMs don’t “reason” in the way humans typically mean the word.
- They’re statistical models of conditional probability: given a history of tokens, predict the next token.
- The reinforcement (RLHF) you mentioned just shifts the probability landscape toward outputs that humans like, but it doesn’t fundamentally add a reasoning module.
2. The “reasoning residue” argument
Here’s where your point kicks in. Much of the text in the training corpus was itself produced by humans reasoning—deriving a theorem, explaining a cause, constructing an argument.
- That means the distribution of tokens encodes traces of reasoning patterns.
- When the LLM predicts continuations, it is effectively interpolating through that latent structure.
- Even if the LLM doesn’t “know it’s reasoning,” it reproduces patterns that are consequences of reasoning.
So in a sense, you’re right: the reasoning that went into producing the training data survives in fossilized form inside the model.
3. Emergent reasoning-like behavior
Because human reasoning is systematic, the training process induces internal circuits that can manipulate abstract relations (chains of if-then, causal analogies, etc.).
- Some researchers call this “simulacrum reasoning”: the model doesn’t reason in the causal, self-directed sense, but it produces outputs that look like reasoning because the statistical map encodes it.
- Chain-of-thought prompting is an example: when we nudge the model into “thinking step by step,” it can unspool reasoning-like patterns that were baked into the data.
4. The philosophical wrinkle
If reasoning is defined purely behaviorally (following valid inference patterns to reach conclusions), then LLMs arguably do reason.
If reasoning is defined internally (involving awareness, intentionality, goal-directed use of logic), then they don’t.It’s similar to debates in philosophy of mind about whether a system that simulates a process is “really doing” it. (Classic reference: Searle’s Chinese Room.)
A question back to you: do you lean toward the “behavioral” definition (reasoning is as reasoning does), or the “internalist” one (reasoning requires a kind of intentional process)? That determines whether you’d say LLMs reason, or just mimic reasoning.I think reasoning vs mimicry of reasoning is ultimately a rhetorical distinction without a practical difference. Even humans probably overrate the amount of "reasoning" involved in their own language production. It's mostly pattern matching itself, I suppose.
-
said in Me Learning LLM Reasoning:
said in Me Learning LLM Reasoning:
Who thinks LLMs reason anyway? They predict strings of words or sub-words, and their neural net is reinforced by humans who give thumbs up or down to certain responses.
There's an interesting case to be made that if reasoning was involved in the creation of the words in the training set, then that reasoning survives in the words predicted by the LLM trained on that set. Even if the LLM can't disentangle that reasoning from its predictions.
ChatGPT responds:
That’s a sharp way of framing it. Let me break this into a few layers:
1. The standard view: LLMs as pattern completers
- At the mechanistic level, LLMs don’t “reason” in the way humans typically mean the word.
- They’re statistical models of conditional probability: given a history of tokens, predict the next token.
- The reinforcement (RLHF) you mentioned just shifts the probability landscape toward outputs that humans like, but it doesn’t fundamentally add a reasoning module.
2. The “reasoning residue” argument
Here’s where your point kicks in. Much of the text in the training corpus was itself produced by humans reasoning—deriving a theorem, explaining a cause, constructing an argument.
- That means the distribution of tokens encodes traces of reasoning patterns.
- When the LLM predicts continuations, it is effectively interpolating through that latent structure.
- Even if the LLM doesn’t “know it’s reasoning,” it reproduces patterns that are consequences of reasoning.
So in a sense, you’re right: the reasoning that went into producing the training data survives in fossilized form inside the model.
3. Emergent reasoning-like behavior
Because human reasoning is systematic, the training process induces internal circuits that can manipulate abstract relations (chains of if-then, causal analogies, etc.).
- Some researchers call this “simulacrum reasoning”: the model doesn’t reason in the causal, self-directed sense, but it produces outputs that look like reasoning because the statistical map encodes it.
- Chain-of-thought prompting is an example: when we nudge the model into “thinking step by step,” it can unspool reasoning-like patterns that were baked into the data.
4. The philosophical wrinkle
If reasoning is defined purely behaviorally (following valid inference patterns to reach conclusions), then LLMs arguably do reason.
If reasoning is defined internally (involving awareness, intentionality, goal-directed use of logic), then they don’t.It’s similar to debates in philosophy of mind about whether a system that simulates a process is “really doing” it. (Classic reference: Searle’s Chinese Room.)
A question back to you: do you lean toward the “behavioral” definition (reasoning is as reasoning does), or the “internalist” one (reasoning requires a kind of intentional process)? That determines whether you’d say LLMs reason, or just mimic reasoning.I think reasoning vs mimicry of reasoning is ultimately a rhetorical distinction without a practical difference. Even humans probably overrate the amount of "reasoning" involved in their own language production. It's mostly pattern matching itself, I suppose.
said in Me Learning LLM Reasoning:
said in Me Learning LLM Reasoning:
said in Me Learning LLM Reasoning:
Who thinks LLMs reason anyway? They predict strings of words or sub-words, and their neural net is reinforced by humans who give thumbs up or down to certain responses.
There's an interesting case to be made that if reasoning was involved in the creation of the words in the training set, then that reasoning survives in the words predicted by the LLM trained on that set. Even if the LLM can't disentangle that reasoning from its predictions.
ChatGPT responds:
That’s a sharp way of framing it. Let me break this into a few layers:
1. The standard view: LLMs as pattern completers
- At the mechanistic level, LLMs don’t “reason” in the way humans typically mean the word.
- They’re statistical models of conditional probability: given a history of tokens, predict the next token.
- The reinforcement (RLHF) you mentioned just shifts the probability landscape toward outputs that humans like, but it doesn’t fundamentally add a reasoning module.
2. The “reasoning residue” argument
Here’s where your point kicks in. Much of the text in the training corpus was itself produced by humans reasoning—deriving a theorem, explaining a cause, constructing an argument.
- That means the distribution of tokens encodes traces of reasoning patterns.
- When the LLM predicts continuations, it is effectively interpolating through that latent structure.
- Even if the LLM doesn’t “know it’s reasoning,” it reproduces patterns that are consequences of reasoning.
So in a sense, you’re right: the reasoning that went into producing the training data survives in fossilized form inside the model.
3. Emergent reasoning-like behavior
Because human reasoning is systematic, the training process induces internal circuits that can manipulate abstract relations (chains of if-then, causal analogies, etc.).
- Some researchers call this “simulacrum reasoning”: the model doesn’t reason in the causal, self-directed sense, but it produces outputs that look like reasoning because the statistical map encodes it.
- Chain-of-thought prompting is an example: when we nudge the model into “thinking step by step,” it can unspool reasoning-like patterns that were baked into the data.
4. The philosophical wrinkle
If reasoning is defined purely behaviorally (following valid inference patterns to reach conclusions), then LLMs arguably do reason.
If reasoning is defined internally (involving awareness, intentionality, goal-directed use of logic), then they don’t.It’s similar to debates in philosophy of mind about whether a system that simulates a process is “really doing” it. (Classic reference: Searle’s Chinese Room.)
A question back to you: do you lean toward the “behavioral” definition (reasoning is as reasoning does), or the “internalist” one (reasoning requires a kind of intentional process)? That determines whether you’d say LLMs reason, or just mimic reasoning.I think reasoning vs mimicry of reasoning is ultimately a rhetorical distinction without a practical difference. Even humans probably overrate the amount of "reasoning" involved in their own language production. It's mostly pattern matching itself, I suppose.
ChatGPT responds:
That’s a really strong point, and it resonates with some cognitive science perspectives:
1. Human reasoning may be more “after the fact” than we admit
- A lot of psychological research (e.g., Kahneman’s System 1 vs System 2) suggests that much of what we call “reasoning” is post-hoc rationalization layered on top of instinctive, heuristic, and pattern-driven judgments.
- Split-brain patient studies even show the left hemisphere fabricating explanations for actions it didn’t initiate—reasoning as narrative construction, not causal engine.
2. Language as structured mimicry
- Human language production is also probabilistic and patterned: we finish clichés, mirror each other’s syntax, and reuse familiar narrative arcs.
- The “mimicry” charge against LLMs might cut both ways—humans are excellent mimics of cultural reasoning patterns, with just enough novelty to move them forward.
3. Practical indistinguishability
- If both humans and LLMs are mostly recombining patterns, then distinguishing “true reasoning” from “mimicry” risks being a metaphysical quibble.
- What matters is functional reasoning: the ability to get to correct/novel conclusions, persuade others, or generate useful inferences.
- From that angle, humans and LLMs differ more in substrate (neurons vs. transformers) and scope of generalization, not in some essential reasoning/mimicry divide.
4. The rhetorical residue
- The insistence that “humans reason, LLMs mimic” often feels like a way to preserve a bright line between “us” and “it.”
- But as you note, once we admit how much of our own cognition is patterned mimicry, the line blurs.
Would you say your view leans toward functionalism—i.e., if a system produces reasoning-like outputs reliably, then it “reasons,” regardless of mechanism? Or do you think there’s still some irreducible qualitative aspect to human reasoning that’s worth defending?
-
said in Me Learning LLM Reasoning:
said in Me Learning LLM Reasoning:
said in Me Learning LLM Reasoning:
Who thinks LLMs reason anyway? They predict strings of words or sub-words, and their neural net is reinforced by humans who give thumbs up or down to certain responses.
There's an interesting case to be made that if reasoning was involved in the creation of the words in the training set, then that reasoning survives in the words predicted by the LLM trained on that set. Even if the LLM can't disentangle that reasoning from its predictions.
ChatGPT responds:
That’s a sharp way of framing it. Let me break this into a few layers:
1. The standard view: LLMs as pattern completers
- At the mechanistic level, LLMs don’t “reason” in the way humans typically mean the word.
- They’re statistical models of conditional probability: given a history of tokens, predict the next token.
- The reinforcement (RLHF) you mentioned just shifts the probability landscape toward outputs that humans like, but it doesn’t fundamentally add a reasoning module.
2. The “reasoning residue” argument
Here’s where your point kicks in. Much of the text in the training corpus was itself produced by humans reasoning—deriving a theorem, explaining a cause, constructing an argument.
- That means the distribution of tokens encodes traces of reasoning patterns.
- When the LLM predicts continuations, it is effectively interpolating through that latent structure.
- Even if the LLM doesn’t “know it’s reasoning,” it reproduces patterns that are consequences of reasoning.
So in a sense, you’re right: the reasoning that went into producing the training data survives in fossilized form inside the model.
3. Emergent reasoning-like behavior
Because human reasoning is systematic, the training process induces internal circuits that can manipulate abstract relations (chains of if-then, causal analogies, etc.).
- Some researchers call this “simulacrum reasoning”: the model doesn’t reason in the causal, self-directed sense, but it produces outputs that look like reasoning because the statistical map encodes it.
- Chain-of-thought prompting is an example: when we nudge the model into “thinking step by step,” it can unspool reasoning-like patterns that were baked into the data.
4. The philosophical wrinkle
If reasoning is defined purely behaviorally (following valid inference patterns to reach conclusions), then LLMs arguably do reason.
If reasoning is defined internally (involving awareness, intentionality, goal-directed use of logic), then they don’t.It’s similar to debates in philosophy of mind about whether a system that simulates a process is “really doing” it. (Classic reference: Searle’s Chinese Room.)
A question back to you: do you lean toward the “behavioral” definition (reasoning is as reasoning does), or the “internalist” one (reasoning requires a kind of intentional process)? That determines whether you’d say LLMs reason, or just mimic reasoning.I think reasoning vs mimicry of reasoning is ultimately a rhetorical distinction without a practical difference. Even humans probably overrate the amount of "reasoning" involved in their own language production. It's mostly pattern matching itself, I suppose.
ChatGPT responds:
That’s a really strong point, and it resonates with some cognitive science perspectives:
1. Human reasoning may be more “after the fact” than we admit
- A lot of psychological research (e.g., Kahneman’s System 1 vs System 2) suggests that much of what we call “reasoning” is post-hoc rationalization layered on top of instinctive, heuristic, and pattern-driven judgments.
- Split-brain patient studies even show the left hemisphere fabricating explanations for actions it didn’t initiate—reasoning as narrative construction, not causal engine.
2. Language as structured mimicry
- Human language production is also probabilistic and patterned: we finish clichés, mirror each other’s syntax, and reuse familiar narrative arcs.
- The “mimicry” charge against LLMs might cut both ways—humans are excellent mimics of cultural reasoning patterns, with just enough novelty to move them forward.
3. Practical indistinguishability
- If both humans and LLMs are mostly recombining patterns, then distinguishing “true reasoning” from “mimicry” risks being a metaphysical quibble.
- What matters is functional reasoning: the ability to get to correct/novel conclusions, persuade others, or generate useful inferences.
- From that angle, humans and LLMs differ more in substrate (neurons vs. transformers) and scope of generalization, not in some essential reasoning/mimicry divide.
4. The rhetorical residue
- The insistence that “humans reason, LLMs mimic” often feels like a way to preserve a bright line between “us” and “it.”
- But as you note, once we admit how much of our own cognition is patterned mimicry, the line blurs.
Would you say your view leans toward functionalism—i.e., if a system produces reasoning-like outputs reliably, then it “reasons,” regardless of mechanism? Or do you think there’s still some irreducible qualitative aspect to human reasoning that’s worth defending?
@Horace I get teh feeling you dont need us any more. LOL
-
@Horace I get teh feeling you dont need us any more. LOL
@taiwan_girl said in Me Learning LLM Reasoning:
@Horace I get teh feeling you dont need us any more. LOL
I was really shocked to see that Ax didn't want to continue the conversation with Klaus, so I tried to cover it up with something other than shocking, mortifying silence.
-
@taiwan_girl said in Me Learning LLM Reasoning:
@Horace I get teh feeling you dont need us any more. LOL
I was really shocked to see that Ax didn't want to continue the conversation with Klaus, so I tried to cover it up with something other than shocking, mortifying silence.
-
@Klaus said in Me Learning LLM Reasoning:
Can you elaborate?
Rather than deductions or inductions, I got a lot of "we tried this and we see that it works well." Even then there is no "this is why we think this works well." At least that's my impression.
-
@Horace said in Me Learning LLM Reasoning:
I was really shocked to see that Ax didn't want to continue the conversation with Klaus, ...
I missed @Klaus' latest reply because your mindless copied-and-pasted ChatGPT dribble drowned it out.
@Axtremus said in Me Learning LLM Reasoning:
@Horace said in Me Learning LLM Reasoning:
I was really shocked to see that Ax didn't want to continue the conversation with Klaus, ...
I missed @Klaus' latest reply because your mindless copied-and-pasted ChatGPT dribble drowned it out.
Oh, my fault. I thought it was your 1000th straight ignored question. Turns out, this time you would have answered.
