arXivLabs is a framework for collaborators to develop and share new arXiv features directly on the website. Individuals and organizations involved embrace arXiv's values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only partners with those who share them. Have an idea to improve the arXiv community? Learn more about arXivLabs.
arXivLabs is a framework enabling collaborators to develop and share new arXiv features directly on the website. Individuals and organizations involved share arXiv's values of openness, community, excellence, and user data privacy. arXiv only partners with those who adhere to these principles. Got an idea to enhance the arXiv community? Learn more about arXivLabs.
arXivLabs is a framework enabling collaborators to develop and share new arXiv features directly on the website. Individuals and organizations involved uphold arXiv's values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only partners with those who share them. Have an idea to enhance the arXiv community? Learn more about arXivLabs.
arXivLabs is a framework enabling collaborators to develop and share new arXiv features directly on the website. Individuals and organizations involved uphold arXiv's values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only partners with those who share them. Have an idea to enhance the arXiv community? Learn more about arXivLabs.
arXivLabs is a framework that enables developers to collaborate with the arXiv community to build and share new features directly on the arXiv website. Participants must adhere to arXiv's core values of openness, community, excellence, and user data privacy. If you have an idea for a project that will add value to the arXiv community, learn more about arXivLabs.
arXivLabs is a framework for collaborators to build and share new arXiv features directly on the website. Individuals and organizations involved share arXiv's commitment to openness, community, excellence, and user data privacy. arXiv only partners with those who uphold these values. Got an idea to improve the arXiv community? Learn more about arXivLabs.
arXivLabs is a framework enabling collaborators to develop and share new arXiv features directly on the website. Individuals and organizations involved share arXiv's values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only partners with those who adhere to them. Have an idea to enhance the arXiv community? Learn more about arXivLabs.
arXivLabs is a framework enabling collaborators to develop and share new arXiv features directly on the website. Participants embrace arXiv's values of openness, community, excellence, and user data privacy. arXiv only works with partners adhering to these principles. Got an idea to enhance the arXiv community? Learn more about arXivLabs.
arXivLabs is a framework enabling collaborators to develop and share new arXiv features directly on the arXiv website. Individuals and organizations involved share arXiv's values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners who uphold them. Got an idea for a project that will benefit the arXiv community? Learn more about arXivLabs.
arXivLabs is a framework enabling collaborators to develop and share new arXiv features directly on the website. Individuals and organizations involved uphold arXiv's values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners who share them. Have an idea to improve the arXiv community? Learn more about arXivLabs.
arXivLabs is a framework for collaborators to develop and share new arXiv features directly on the website. Individuals and organizations working with arXivLabs embrace our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only partners with those who share them. Got an idea for a project that will benefit the arXiv community? Learn more about arXivLabs.
arXivLabs is a framework enabling collaborators to develop and share new arXiv features directly on the website. Individuals and organizations working with arXivLabs embrace arXiv's values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only partners with those who adhere to them. Have an idea for a project that will add value to arXiv's community? Learn more about arXivLabs.
arXivLabs is a framework enabling collaborators to develop and share new arXiv features directly on the arXiv website. Individuals and organizations involved with arXivLabs embrace our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners who adhere to them. Have an idea for a project that will benefit the arXiv community? Learn more about arXivLabs.
arXivLabs is an experimental platform enabling collaborators to develop and share new arXiv features directly on the website. Participants share arXiv's values of openness, community, excellence, and user data privacy. Got an idea to improve the arXiv community? Learn more about arXivLabs.
arXivLabs is a framework enabling collaborators to build and share new arXiv features directly on the website. Participants, both individuals and organizations, embrace arXiv's values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only partners with those who share them. Got an idea for a project that will benefit the arXiv community? Learn more about arXivLabs.
arXivLabs is an experimental framework enabling collaborators to develop and share new arXiv features directly on the website. Participants must adhere to arXiv's values of openness, community, excellence, and user data privacy. Got an idea to improve the arXiv community? Learn more about arXivLabs.
arXivLabs is a framework enabling collaborators to develop and share new arXiv features directly on the website. Individuals and organizations involved share arXiv's values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only partners with those who adhere to them. Got an idea for a project that will benefit the arXiv community? Learn more about arXivLabs.
arXivLabs is a framework enabling collaborators to develop and share new arXiv features directly on the website. Participants, individuals and organizations alike, embrace arXiv's values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only partners with those who share them. Have an idea to enhance the arXiv community? Learn more about arXivLabs.
arXivLabs is a framework enabling collaborators to develop and share new arXiv features directly on the website. Individuals and organizations involved embrace arXiv's values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only partners with those who share them. Have an idea to enhance the arXiv community? Learn more about arXivLabs.
arXivLabs is a framework for collaborators to develop and share new features directly on the arXiv website. Individuals and organizations involved share arXiv's values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only partners with those who adhere to them. Have an idea to improve the arXiv community? Learn more about arXivLabs.
arXivLabs is a framework for collaborators to develop and share new arXiv features directly on the website. Individuals and organizations involved embrace arXiv's values of openness, community, excellence, and user data privacy. arXiv only partners with those adhering to these values. Got an idea to enhance the arXiv community? Learn more about arXivLabs.
Ronit Shah presents an improved algorithm for the Quantum Fourier Transform (QFT). Traditionally, approximate QFT requires Θ(n log n) gates, and exact QFT requires Θ(n²) gates. The new algorithm, leveraging a novel recursive partitioning of qubits, reduces the cost of approximate QFT to Θ(n(log log n)²) gates and exact QFT to Θ(n(log n)²) gates. This breakthrough promises significant efficiency gains in quantum computation.
DeepSeek-AI unveils DeepSeek-R1, its first-generation reasoning model trained via large-scale reinforcement learning (RL) without supervised fine-tuning. Its precursor, DeepSeek-R1-Zero, surprisingly demonstrated strong reasoning capabilities, but suffered from readability and language mixing issues. DeepSeek-R1 addresses these flaws with multi-stage training and cold-start data, achieving performance comparable to OpenAI's models. To foster research, DeepSeek-AI open-sources DeepSeek-R1-Zero, DeepSeek-R1, and six distilled models of varying sizes, built upon Qwen and Llama.
A new book, "Foundations of Large Language Models," has been released. Instead of aiming for comprehensive coverage of cutting-edge technologies, it delves into the core foundational concepts of large language models. Structured into four chapters covering pre-training, generative models, prompting techniques, and alignment methods, the book is geared towards college students, professionals, and practitioners in natural language processing and related fields. It serves as a valuable reference for anyone interested in LLMs.
Researchers introduce a lossless compression scheme for vector IDs to address the high storage cost of indexes in approximate nearest neighbor search. Leveraging the fact that the order of IDs is irrelevant in many index structures, and utilizing asymmetric numeral systems or wavelet trees, the method achieves up to 7x compression of vector IDs without impacting accuracy or search runtime. This translates to a 30% reduction in index size for billion-scale datasets. Furthermore, the approach can also losslessly compress quantized vector codes by exploiting sub-optimalities in the original quantization algorithm.
Large language models are expensive to train and deploy for assisting with Excel formula authoring. This paper introduces FLAME, a transformer-based model trained exclusively on Excel formulas. With only 60 million parameters and a fraction of the training data used by larger models, FLAME achieves competitive or even superior performance on formula repair, completion, and retrieval tasks compared to models like Codex and CodeT5. This is attributed to its novel pre-training objectives and Excel-specific tokenizer.
Scaling language models to handle longer input sequences typically requires large key-value (KV) caches, resulting in substantial memory overhead during inference. This paper proposes Tensor Product Attention (TPA), a novel attention mechanism that uses tensor decompositions to represent queries, keys, and values compactly, significantly reducing KV cache size during inference. By factorizing these representations into contextual low-rank components (contextual factorization) and seamlessly integrating with RoPE, TPA improves model quality while maintaining memory efficiency. Based on TPA, the authors introduce the Tensor ProducT ATTenTion Transformer (T6), a new model architecture for sequence modeling. Extensive empirical evaluation on language modeling tasks demonstrates that T6 surpasses standard Transformer baselines including MHA, MQA, GQA, and MLA across various metrics, including perplexity and a range of well-known evaluation benchmarks. Notably, TPA's memory efficiency enables the processing of significantly longer sequences under fixed resource constraints, addressing a critical scalability challenge in modern language models. Code is available.
Researchers have successfully resurrected ELIZA, widely considered the world's first chatbot, on a restored CTSS—the world's first time-sharing system (emulated on an IBM 7094). Using original printouts, MAD-SLIP code, and supporting documents found in Prof. Weizenbaum's archives at MIT, they recreated ELIZA and its famous DOCTOR script. The entire project is open-source, allowing anyone with a Unix-like OS to run the groundbreaking chatbot.
A paper on arXiv explores the mathematics behind the New York Times' daily word game, Waffle. Author S.P. Glasby delves into the combinatorial properties of the game, explaining why some puzzles are easy while others are exceptionally difficult. The research reveals that a perfect solution requires precisely 11 orbits among the 21 squares, with at least one orbit of length 1. This provides a mathematical framework for understanding and potentially improving similar word puzzles.
Researchers introduce Titans, a novel neural architecture that combines a neural memory module with an attention mechanism to effectively memorize long-term historical context. Unlike traditional recurrent models and attention mechanisms, Titans demonstrates superior efficiency and accuracy in handling long sequences, particularly excelling in "needle-in-a-haystack" tasks. It outperforms Transformers and recent linear recurrent models across various tasks including language modeling, common-sense reasoning, genomics, and time series, and scales to context windows exceeding 2 million tokens.
