Look up "mechanistic interpretability" in the context of LLMs. The next token prediction machinery is just a foundation for a higher order learned structure that appears to encode specific concepts, regardless of input language.
The analogy to humans is that the human brain is "just atoms bouncing around", but there's unquestionably something "more" going on that just that.
Until ais get better from feeding on their own output the way humans do there is nothing to question.
That will probably be the fundamental indicator that something other than repeating things some human previously created is going on.
So far, ais only get worse from feeding on their own output. Meanging/including the output of other ais not a single ai feeding on it's own output. Also bear in mind that so far even the output of ais is 100% the downstream of a human command. No ai has persued it's own curiosity that didn't result from a human asking a question or giving a command. That is input which is different from a human taking in their environment even though our limited language can call those both the same word input.
The fact that humans also repeat and remix things, and humans also produce essentially procedurally generated empty output like corporate-speak etc, is an irrelevant distraction in the same way that both a human and an electric motor can both perform the same simple mechanical task.
The analogy to humans is that the human brain is "just atoms bouncing around", but there's unquestionably something "more" going on that just that.