forkrun

le — and where forkrun is designed to perform. A few of the techniques that make this possible: - Born-local NUMA: stdin is splice()'d into a shared memfd, then pages are placed on the target NUMA node via set_mempolicy(MPOL_BIND) before any worker touches them, making the memfd NUMA-spliced. - SIMD scanning: per-node indexers use AVX2/NEON to find line boundaries at memory bandwidth and publish byte-offsets and line-counts into per-node lock-free rings. - Lock-free claiming: workers claim batches with a single atomic_fetch_add — no locks, no CAS retry loops; contention is reduced to a single atomic on one cache line. - Memory management: a background thread uses fallocate(PUNCH_HOLE) to reclaim space without breaking the logical offset system. …and that’s just the surface. The implementation uses many additional systems-level techniques (phase-aware tail handling, adaptive batching, early-flush detection, etc.) to eliminate overhead at every stage. In its fastest (-b) mode (fixed-size batches, minimal processing), it can exceed 1B lines/sec. In typical streaming workloads it's often 50×–400× faster than GNU Parallel. forkrun ships as a single bash file with an embedded, self-extracting C extension — no Perl, no Python, no install, full native support for parallelizing arbitrary shell functions. The binary is built in public GitHub Actions so you can trace it back to CI (see the GitHub "Blame" on the line containing the base64 embeddings). - Benchmarking scripts and raw results: https://github.com/jkool702/forkrun/blob/main/BENCHMARKS - Architecture deep-dive: https://github.com/jkool702/forkrun/blob/main/DOCS - Repo: https://github.com/jkool702/forkrun Trying it is literally two commands: . frun.bash # OR `. <(curl https://raw.githubusercontent.com/jkool702/forkrun/main/frun.bash)` frun shell_func_or_cmd < inputs Happy to answer questions.

ncy low-latency workloads (vs GNU Parallel) A few of the techniques that make this possible: * Born-local NUMA: stdin is splice()'d into a shared memfd, then pages are placed on the target NUMA node via set_mempolicy(MPOL_BIND) before any worker touches them, making the memfd NUMA-spliced. Each numa node only claims work that is already born-local on its node. Stealing from other nodes is permitted under some conditions when no local work exists. * SIMD scanning: per-node indexers/scanners use AVX2/NEON to find line boundaries (delimiters) at speeds approaching memory bandwidth, and publish byte-offsets and line-counts into per-node lock-free rings. * Lock-free claiming: workers claim batches with a single atomic_fetch_add — no locks, no CAS retry loops; contention is reduced to a single atomic on one cache line. * Memory management: a background thread uses fallocate(PUNCH_HOLE) to reclaim space without breaking the logical offset system. …and that’s just the surface. The implementation uses many additional systems-level techniques (phase-aware tail handling, adaptive batching, early-flush detection, etc.) to eliminate overhead, increase throughput and reduce latency at every stage. In its fastest (-b) mode (fixed-size batches, minimal processing), it can exceed 1B lines/sec. forkrun ships as a single bash file with an embedded, self-extracting C extension — no Perl, no Python, no install, full native support for parallelizing arbitrary shell functions. The binary is built in public GitHub Actions so you can trace it back to CI (see the GitHub "Blame" on the line containing the base64 embeddings). Trying it is literally two commands: . frun.bash frun shell_func_or_cmd < inputs For benchmarking scripts and results, see the BENCHMARKS dir in the GitHub repo For an architecture deep-dive, see the DOCS dir in the GitHub repo Happy to answer questions.