Yek: A Fast Tool for Chunking Repository Files for LLMs

Yek is a fast Rust-based tool designed to read text-based files from a repository or directory, chunk them, and serialize them for Large Language Model (LLM) consumption. It intelligently skips unwanted files (using .gitignore rules and Git history), splits content into chunks based on token count or byte size, and offers extensive configuration options such as custom ignore patterns and file priority rules. Yek aims to streamline the data preparation process for LLMs, boosting efficiency.