Work

Intention modeling mediates cooperative incompatibility via character separability

Year: 2026
Type: preprint
Abstract: Humans continually infer what others intend. This inference matters equally for AI, where it is studied as partner modeling. Partner modeling arises in two forms: an implicit form, in which an agent dynamically encodes partner behavior in its hidden state as a byproduct of return maximization, and an explicit form, in which a dedicated module learns a separate representation of partner intent. While explicit methods report gains, it remains unclear when, and why, they improve over the implicit representation approach. We address this question by establishing a novel connection to the character and mental-state decomposition proposed in the nascent work, machine theory of mind. With an explicit module, a division of labor arises: the dedicated intention module maintains a stable, separable character of a partner, while the agent's internal latent specializes in its mental state, which is continually updated. Under certain settings, the explicit approach reduces partner incompatibility, i.e., the failure to coordinate with certain partners. Specifically, we find that, in settings that require distinct best responses per partner type, the reduction of partner incompatibility is associated with separable character modeling, which the explicit module supplies. In other settings that we examined, explicit and implicit agents produce comparable return distributions.
Language: English
Open Access status: green
OpenAlex ID: W7204221902
Cites: 25
Cited by:
Related to: 10