Titans: A Novel Neural Architecture for Learning to Memorize at Test Time
2025-01-16
Researchers introduce Titans, a novel neural architecture that combines a neural memory module with an attention mechanism to effectively memorize long-term historical context. Unlike traditional recurrent models and attention mechanisms, Titans demonstrates superior efficiency and accuracy in handling long sequences, particularly excelling in "needle-in-a-haystack" tasks. It outperforms Transformers and recent linear recurrent models across various tasks including language modeling, common-sense reasoning, genomics, and time series, and scales to context windows exceeding 2 million tokens.