Regarding the debate about anthropomorphism of AI systems - complex systems can be described at more than one level of abstraction. An entity can be described by its molecular physics, its organic chemistry, or (at a much higher level of abstraction) its beliefs, intentions, and ability to reason.
Most arguments against anthropomorphization of AI systems seem to merely assert that a lower level of abstraction exists - and therefore higher levels of abstraction cannot exist. But that doesn't follow; multiple levels of abstraction can exist at the same time.
Whether an AI system really has attributes like "beliefs" depends ultimately on how such words are defined. Most words we use in everyday language are not rigorously defined; they're fuzzy round the edges. So when someone asserts that an AI system does or does not possess "beliefs" (for example), they're not really making a statement of fact - they're making a statement of preference about whether the word ought to be defined to encompass the AI system or not.
Work on AI interpretability shows that a model's hidden layers can contain data structures encoding things like current state of world (albeit imperfectly and incompletely) and target state of world (in the case of agentic systems). They also show ability to infer actions needed to get from current state to target state (for example). Describing these internal data structures and processes using terms like "beliefs", "goals" and "reasoning" seems justifiable, if the overall system's behaviour functionally aligns with how we ordinarily understand such terms.
Finally, you don't need to take any particular position on whether AI systems are capable of phenomenal consciousness to hold that attributes like "beliefs", "goals" and "reasoning" are suitable descriptions at a high-level of abstraction. An AI system could hold many such attributes without having phenomenal consciousness.
This is wrong. Chalmers does not claim that a process could have exactly the same [computed] thoughts with or without it being conscious.
What he claims is more like this: 1/ there's a gap in our understanding in that we can't explain why physical processes alone give rise to phenomenal consciousness; 2/ to prove that this "explanatory gap" exists, he says let's consider the scenario you describe ("p-zombies") as a thought experiment; 3/ since you cannot give any definitive reason why p-zombies are impossible, that proves that the gap in our understanding must be real.
This argument does *not* require that p-zombies are actually possible in real life. It only requires us to be unable to explain why they're impossible. Indeed, Chalmers himself said "zombies are probably not naturally possible: they probably cannot exist in our world, with its laws of nature."
At some point in the future, someone might eventually figure out a convincing explanation for why phenomenal consciousness occurs (e.g. that it happens inevitably as an "emergent property" of being functionally conscious). If so, that would be perfectly consistent with Chalmers' argument. He doesn't claim that there will *always* be a gap in our understanding. He's only saying that there is *currently* a gap in our understanding.
Without a convincing explanation for phenomenal consciousness, ideas like it being "an emergent property of being functionally conscious" are essentially just hunches (albeit appealing ones). They don't explain how or why it occurs as an emergent property; they merely conjecture that it does. Chalmers is effectively calling them out as hunches, and saying that's why the hard problem is hard.
