Adaptive Hashing in SBCL: Making Hash Tables Faster and More Robust
A talk at the 2024 ELS focused on adaptive hashing, aiming to make general-purpose hash tables faster and more robust. Traditional hash table theory primarily concerns itself with asymptotic worst-case costs, neglecting the impact of constant factors on real-world performance. This research introduces an online adaptive approach, adjusting the hash function based on the actual key distribution to reduce collisions and improve cache utilization. Experiments demonstrate significant improvements in reducing expected comparisons and speeding up PUT operations, particularly with specific key distributions. SBCL's built-in hash tables now employ this technique, dynamically switching hash functions (including linear search, bit-shifting hash, and MurmurHash) based on collision counts and hash table size. For composite keys like strings and lists, a truncation strategy is used, dynamically adjusting the truncation length when too many collisions occur. This improvement enhances SBCL hash table speed in common cases and robustness in others.
Read more