Year: 2024
Type: dissertation
Abstract: We study diffusion models and causal transformers under the same lens by treating both architectures as discrete approximations of continuous stochastic processes. To do so, we introduce Continuous Causal Transformers (CCTs), a time- and space-continuous generalization of causal transformers, and provide qualitative evidence showing that vanilla causal transformers implicitly approximate CCTs. We then introduce Structured Autoregressivity, a collection of five properties that are shared by diffusion models and causal transformers, and show how they emerge naturally from our analysis. Finally, we describe the implications of our framework, identifying research directions for the design of both generative and non-generative models.
Author: Marro, Samuele
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W6998591826