Please register for the CV & Online Identity Workshop below.

Almost all of the recent advances in AI (large language models, image/video diffusion models, recommendation systems, and visuomotor policy in robotics) are based around a neural architecture known as transformer. We identify the process of propagation of representations through layers of transformer with a particular kind of an interacting particle system, whose dynamics exhibit several interesting properties. In the initial (short) phase, the particles coalesce to form medium-sized clusters. In the second (long) phase, these clusters slowly spin around and occasionally merge until they form a single lump. We hypothesize that the first phase's (meta-stable) clustering may explain transformers' ability for robust long-horizon logical deductions. The second phase corresponds to synchronization behavior discovered in certain dynamics, such as the Kuramoto model, which turns out to be a special case of the transformer.