Perceiver IO: A General Architecture for Structured Inputs & Outputs

The authors address the challenge of building a truly general-purpose machine learning architecture. Traditional models are often "bespoke"—engineered specifically for one modality (e.g., vision or language) or a single task. This specialization makes it difficult to handle diverse, large-scale, or multi-modal data without complex, task-specific pipelines. The researchers ask: Can a single, domain-agnostic architecture handle arbitrary input and output structures while scaling efficiently?

Approach: Perceiver IO

Perceiver IO is an evolution of the original Perceiver architecture. It uses a "read-process-write" design that decouples the core computation from the specific structure of the input and output:

• Encoding (Read): The model uses cross-attention to map arbitrary input data (e.g., pixels, bytes, or audio) into a fixed-size "latent array." This latent space acts as a bottleneck, allowing the model to handle large inputs without the quadratic memory costs associated with standard Transformers.
• Processing: The latent array is refined through a series of deep, fully attentional layers. Because this happens in the latent space, the computational cost is independent of the input size.
• Decoding (Write): The authors introduce a flexible querying mechanism. To produce an output, the model uses "output queries" that specify the desired semantics (e.g., spatial coordinates for pixels or task-specific embeddings for classification). This allows the model to generate outputs of any shape or structure, such as optical flow fields or symbolic game actions.

Perceiver IO demonstrates that a single architecture can compete with or outperform specialized models across widely different domains: