TokenDagger: A Blazing Fast TikToken Implementation

TokenDagger offers a high-performance alternative to OpenAI's TikToken, optimized for large-scale text processing. Benchmarks show TokenDagger achieving over 4x speedup on code tokenization and a 2x throughput increase compared to TikToken. Leveraging an optimized PCRE2 regex engine and a simplified BPE algorithm to mitigate the performance impact of large special token vocabularies, TokenDagger provides a drop-in replacement. Installation and performance testing are straightforward with a few simple commands.