xxHash

xxHash - Extremely fast hash algorithm Website · API documentation · xxhsum manual · Format specification xxHash is an extremely fast non-cryptographic hash algorithm, working at RAM speed limits. It is highly portable and produces identical hashes on all platforms, including little- and big-endian systems. Once finalized, algorithm outputs remain stable across xxHash releases. The library includes the following algorithms: XXH32 generates 32-bit hashes, using 32-bit arithmetic. XXH64 generates 64-bit hashes, using 64-bit arithmetic. XXH3, available since v0.8.0, generates 64-bit or 128-bit hashes using vectorized arithmetic. The 128-bit variant is called XXH128. For new applications, XXH3_64bits() is the recommended default. Use XXH3_128bits() when a 128-bit hash is required. Benchmarks The benchmarked reference system uses an Intel i7-9700K CPU running Ubuntu x64 20.04. The open source benchmark program is compiled with Clang v10.0 using -O3. Hash Name Width Bandwidth Small Data Velocity Comment XXH3 (AVX2) 64 59.4 GB/s 133.1 XXH128 (AVX2) 128 57.9 GB/s 118.1 XXH3 (SSE2) 64 31.5 GB/s 133.1 XXH128 (SSE2) 128 29.6 GB/s 118.1 memcpy, from RAM N/A 28.0 GB/s N/A for reference City64 64 22.0 GB/s 76.6 T1ha2 64 22.0 GB/s 99.0 Slightly worse collisions City128 128 21.7 GB/s 57.7 XXH64 64 19.4 GB/s 71.0 SpookyHash 64 19.3 GB/s 53.2 Mum 64 18.0 GB/s 67.0 Slightly worse collisions XXH32 32 9.7 GB/s 71.9 City32 32 9.1 GB/s 66.0 Murmur3 32 3.9 GB/s 56.1 SipHash 64 3.0 GB/s 43.2 FNV64 64 1.2 GB/s 62.7 Poor avalanche properties Blake2 256 1.1 GB/s 5.1 Cryptographic SHA1 160 0.8 GB/s 5.6 Cryptographic but broken MD5 128 0.6 GB/s 7.8 Cryptographic but broken For current measurements on additional platforms, see the benchmarks on xxhash.com. Note 1: Small data velocity is a rough evaluation of an algorithm's efficiency on small data. For more detailed analysis, see the next section. Note 2: Some algorithms feature faster than RAM speed. They can only reach their full speed potential when input is already in CPU cache (L3 or better). Otherwise, they are limited by RAM speed. Small data Performance on large data is only one part of the picture. Hashing is also very useful in constructions like hash tables and bloom filters. In these use cases, it is common to hash many small inputs, sometimes only a few bytes long. An algorithm's performance can be very different in such scenarios, since initialization and finalization become fixed costs. Branch misprediction also has a much greater impact. XXH3 has been designed for excellent performance on both long and small inputs, which can be observed in the following graph: For a more detailed analysis, see the performance comparison on the wiki. xxHash is not a cryptographic hash function. Do not use it for signatures, password storage, or any other purpose that requires resistance to attacks. Branch Status release dev Getting started The default make target builds both the library and the xxhsum command line utility: make ./xxhsum -H3 README.md The library can then be linked as libxxhash, or compiled directly from xxhash.c. For a header-only integration, define XXH_INLINE_ALL before including xxhash.h: #define XXH_INLINE_ALL #include "xxhash.h" For CMake integration, see the CMake guide. The xxhsum manual documents checksum generation, verification, benchmarking and advanced command line options. xxhash.h is generated from the source units in lib/. It only keeps the documentation of public declarations, which IDEs display. Smaller variants, limited to some hash families, can be generated with make variants, or downloaded from each release (see limited variants). Contributors should modify these units: make regenerates xxhash.h (see AGENTS.md). Example The simplest API hashes a contiguous block of memory in a single call: #include <stddef.h> #include "xxhash.h" XXH64_hash_t hash_buffer(const void* buffer, size_t size) { return XXH3_64bits(buffer, size); } The API also supports incremental hashing of streams of unknown size. Complete single-shot and streaming examples are provided in the API documentation. Quality Speed is not the only property that matters. For non-adversarial inputs, xxHash aims to produce a uniform distribution so that any subset of the output bits can spread entries evenly in a table or index. Like any fixed-width hash, it is still subject to collisions and the birthday paradox. All variants successfully complete Austin Appleby's SMHasher test suite, providing a baseline measure of statistical quality. Additional tests that evaluate speed and collision properties more thoroughly are also provided. Finally, xxHash provides its own massive collision tester, able to generate and compare billions of hashes to test the limits of 64-bit hash algorithms. On this front too, xxHash features good results, in line with the birthday paradox. A more detailed analysis is documented in the wiki. Packages xxHash is available from many package managers. With vcpkg, install the library with: vcpkg install xxhash Add the xxhsum feature to install the command line utility as well: vcpkg install "xxhash[xxhsum]" The current package versions available across distributions are tracked by Repology. Advanced build options Library macros The following macros can be set at compilation time to modify libxxhash's behavior. They are generally disabled by default. XXH_INLINE_ALL: Make all functions inline, implementation is directly included within xxhash.h. Inlining functions is beneficial for speed, notably for small keys. It's extremely effective when key's length is expressed as a compile time constant, with performance improvements observed in the +200% range . See this article for details. XXH_PRIVATE_API: same outcome as XXH_INLINE_ALL. Still available for legacy support. The name underlines that XXH_* symbol names will not be exported. XXH_STATIC_LINKING_ONLY: gives access to internal s