The Transformer architecture revolutionized sequence modeling through attention mechanisms, prioritizing relevant input data by looking back at earlier inputs. However, the computational cost escalates dramatically with sequence length, severely limiting Transformer-based models for tasks like full-document understanding or genomic analysis.
Researchers explored efficient linear recurrent neural networks (RNNs) and state space models (SSMs) such as Mamba-2. These models provide fast, linear scaling by compressing context into a fixed size, but this compression inadequately captures rich information from extremely long sequences.
Introducing two new papers, Titans and MIRAS, we present an architecture and theoretical blueprint that merge RNN speed with Transformer accuracy. Titans is the specific architecture, while MIRAS is the theoretical framework for generalizing these approaches. Together, they advance test-time memorization—the AI model's ability to maintain long-term memory by incorporating powerful “surprise” metrics (unexpected information) during active operation, without requiring offline retraining.
The MIRAS framework, exemplified by Titans, signifies a substantial shift toward real-time adaptation. Instead of compressing information into a static state, this architecture actively learns and updates its parameters as data streams in, enabling the model to instantly incorporate new, specific details into its core knowledge.
Titans: Master New Context Dynamically
An effective learning system demands distinct yet interconnected memory modules, mirroring the human brain's separation of short-term and long-term memory. While attention excels at precise, short-term recall, Titans introduces a novel neural long-term memory module. Unlike the fixed-size vector or matrix memory in traditional RNNs, this module functions as a deep neural network (a multi-layer perceptron), offering significantly higher expressive power. This allows the model to synthesize vast amounts of information without losing critical context—it understands and synthesizes the entire narrative, not just takes notes.
Crucially, Titans actively learns to identify and retain important relationships and conceptual themes across the entire input. A core component of this capability is the “surprise metric.” In human psychology, we readily recall surprising or emotionally charged events while forgetting routine occurrences. Similarly, Titans detects significant differences between its current memory and new input.
Overview of the Titans (MAC) architecture. It employs a long-term memory to compress past data, then integrates this summary into the context passed to attention, allowing attention to decide whether to reference the past summary.
In Titans, the "surprise metric" quantifies the model's detection of a significant divergence between its current memory and incoming data. Low surprise indicates that new input aligns with the model's expectations (e.g., encountering "cat" when expecting an animal), allowing it to bypass permanent long-term storage. High surprise signals anomalous or critical information (e.g., a banana peel in a financial report summary), prioritizing it for permanent storage in the long-term memory module. This internal error signal acts as a powerful indicator of importance, enabling the Titans architecture to selectively update long-term memory with novel, context-breaking information efficiently.
Titans refines this mechanism with two key elements: Momentum, which considers both momentary and past surprise to capture relevant subsequent information; and Forgetting (weight decay), an adaptive mechanism that discards less critical information to manage finite memory capacity in extremely long sequences.
MIRAS: Unifying Sequence Modeling Perspectives
Every significant breakthrough in sequence modeling, from advanced transformers to rapid linear RNNs, fundamentally involves a sophisticated associative memory module. MIRAS offers a unique and practical perspective by viewing diverse architectures as different methods for solving the same core problem: efficiently combining new information with existing memories without losing essential concepts.
MIRAS defines a sequence model through four critical design choices: Memory architecture (the data storage structure), Attentional bias (the model's internal learning objective for prioritization), Retention gate (a memory regularizer that balances new learning with past knowledge retention), and Memory algorithm (the optimization method for memory updates).
The MIRAS framework overview. MIRAS aims to learn an associative memory, mapping keys to values. For each token, the memory module internally optimizes its attentional bias while using its retention gate to maintain its past state. Optimization occurs via a gradient-based optimizer.
Innovating Beyond Mean Squared Error
Most successful sequence models rely on mean squared error (MSE) or dot-product similarity for bias and retention, leading to sensitivity to outliers and limited expressive power. MIRAS transcends this by offering a generative framework that explores a richer design space informed by optimization and statistics, enabling novel architectures with non-Euclidean objectives and regularization.
Leveraging MIRAS, we developed three attention-free models: YAAD, which uses a gentler penalty (Huber loss) for reduced sensitivity to outliers, enhancing robustness with messy data; MONETA, which explores complex, strict penalties (generalized norms) for a potentially more powerful and stable long-term memory system; and MEMORA, which enforces strict probability mapping for controlled, balanced memory updates, ensuring stable information integration.
Experiments and Empirical Validation
We rigorously evaluated Titans and MIRAS variants (YAAD, MONETA, MEMORA) against leading architectures like Transformer++, Mamba-2, and Gated DeltaNet. Testing on genomic modeling (DNA) and time-series forecasting confirmed Titans' versatility beyond text. Across standard language modeling datasets (C4, WikiText) and zero-shot reasoning tasks (HellaSwag, PIQA), our models consistently achieved superior accuracy and perplexity.
The Impact of Deep Memory Architectures
Ablation studies definitively demonstrate the critical role of memory architecture depth. Deeper memory modules consistently yield lower perplexity in language modeling and exhibit superior scaling properties, maintaining performance as sequence length expands significantly. This highlights the power of deep memory structures for enhanced AI comprehension.
The effect of memory depth on perplexity across 360M and 760M parameter scales.
Language Modeling and System Efficiency
Titans architectures surpass state-of-the-art linear recurrent models (Mamba-2, Gated DeltaNet) and Transformer++ baselines in language modeling and commonsense reasoning tasks, achieving comparable sizes. The novel MIRAS variants (MONETA, YAAD, MEMORA) also demonstrate improved performance, validating the benefits of exploring robust, non-MSE optimization mechanisms. Crucially, these models retain efficient, parallelizable training and swift linear inference speeds.
Unprecedented Long-Context Recall Capabilities
The most significant advantage of these novel architectures lies in their exceptional handling of extremely long contexts. The BABILong benchmark, which requires reasoning across facts in vast documents, showcases Titans outperforming all baselines, including large models like GPT-4, despite possessing fewer parameters. Titans also effectively scales to context window sizes exceeding 2 million tokens.
Performance of Titans on extreme long-context reasoning tasks.
Conclusion: The Future of Sequence Modeling
The introduction of Titans and the MIRAS framework represents a significant leap forward in sequence modeling. By utilizing deep neural networks as memory modules that learn dynamically as data arrives, these approaches effectively overcome the limitations of fixed-size recurrent states. MIRAS provides a powerful theoretical unification, revealing the intrinsic connection between online optimization, associative memory, and architectural design. Moving beyond the conventional Euclidean paradigm, this research pioneers a new generation of sequence models that seamlessly combine RNN efficiency with the expressive power demanded by the era of long-context AI.