HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
lakehq
sail
Rust2024–2026
lakehq / sail

sail

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

Data & DatabasesSystems & InfrastructureScience & EngineeringAI & Machine LearningLearning & Reference
Explore on GitHubRead Hacker News discussions
4Distinct people
6Mentions
0Explicit recommenders
3.5KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 5:04:57 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by month6 dated mentions

Observed mentions: 2024-09-10 00:52:12 UTC to 2026-08-03 10:26:00 UTC.

0122024-09-01 00:00:00 UTC to 2024-10-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 commentsSep 20242024-10-01 00:00:00 UTC to 2024-11-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-11-01 00:00:00 UTC to 2024-12-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2024-12-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-01-01 00:00:00 UTC to 2025-02-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-02-01 00:00:00 UTC to 2025-03-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-03-01 00:00:00 UTC to 2025-04-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2025-04-01 00:00:00 UTC to 2025-05-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-05-01 00:00:00 UTC to 2025-06-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-06-01 00:00:00 UTC to 2025-07-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-07-01 00:00:00 UTC to 2025-08-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2025-08-01 00:00:00 UTC to 2025-09-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 commentsAug 20252025-09-01 00:00:00 UTC to 2025-10-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-10-01 00:00:00 UTC to 2025-11-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-11-01 00:00:00 UTC to 2025-12-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-12-01 00:00:00 UTC to 2026-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2026-01-01 00:00:00 UTC to 2026-02-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2026-02-01 00:00:00 UTC to 2026-03-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2026-03-01 00:00:00 UTC to 2026-04-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2026-04-01 00:00:00 UTC to 2026-05-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2026-05-01 00:00:00 UTC to 2026-06-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2026-06-01 00:00:00 UTC to 2026-07-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2026-07-01 00:00:00 UTC to 2026-08-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2026-08-01 00:00:00 UTC to 2026-09-01 00:00:00 UTC (exclusive): 2 mentions; 2 submissions, 0 commentsAug 2026

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
Sep 20242024-09-01 00:00:00 UTC2024-10-01 00:00:00 UTC110
Oct 20242024-10-01 00:00:00 UTC2024-11-01 00:00:00 UTC000
Nov 20242024-11-01 00:00:00 UTC2024-12-01 00:00:00 UTC101
Dec 20242024-12-01 00:00:00 UTC2025-01-01 00:00:00 UTC000
Jan 20252025-01-01 00:00:00 UTC2025-02-01 00:00:00 UTC000
Feb 20252025-02-01 00:00:00 UTC2025-03-01 00:00:00 UTC000
Mar 20252025-03-01 00:00:00 UTC2025-04-01 00:00:00 UTC101
Apr 20252025-04-01 00:00:00 UTC2025-05-01 00:00:00 UTC000
May 20252025-05-01 00:00:00 UTC2025-06-01 00:00:00 UTC000
Jun 20252025-06-01 00:00:00 UTC2025-07-01 00:00:00 UTC000
Jul 20252025-07-01 00:00:00 UTC2025-08-01 00:00:00 UTC101
Aug 20252025-08-01 00:00:00 UTC2025-09-01 00:00:00 UTC000
Sep 20252025-09-01 00:00:00 UTC2025-10-01 00:00:00 UTC000
Oct 20252025-10-01 00:00:00 UTC2025-11-01 00:00:00 UTC000
Nov 20252025-11-01 00:00:00 UTC2025-12-01 00:00:00 UTC000
Dec 20252025-12-01 00:00:00 UTC2026-01-01 00:00:00 UTC000
Jan 20262026-01-01 00:00:00 UTC2026-02-01 00:00:00 UTC000
Feb 20262026-02-01 00:00:00 UTC2026-03-01 00:00:00 UTC000
Mar 20262026-03-01 00:00:00 UTC2026-04-01 00:00:00 UTC000
Apr 20262026-04-01 00:00:00 UTC2026-05-01 00:00:00 UTC000
May 20262026-05-01 00:00:00 UTC2026-06-01 00:00:00 UTC000
Jun 20262026-06-01 00:00:00 UTC2026-07-01 00:00:00 UTC000
Jul 20262026-07-01 00:00:00 UTC2026-08-01 00:00:00 UTC000
Aug 20262026-08-01 00:00:00 UTC2026-09-01 00:00:00 UTC220

What Hacker News says

6 sources
chenxi9649·Jul 8, 2025
Hey HN! We're excited to share Sail 0.3, our open-source distributed computing framework that serves as a drop-in replacement for Apache Spark. Sail is a Rust-native execution engine that speaks the Spark Connect protocol. Your existing Spark SQL and DataFrame code runs unchanged, but executes on average 4x faster while using 94% less infrastructure spend. Here's how we are achieving that performance: 1. No JVM overhead: Rust's zero-cost abstractions and deterministic memory management eliminate GC pauses 2. Columnar processing: Apache Arrow format enables SIMD instructions to process multiple records per CPU cycle 3. Zero-copy data transfer: Python UDFs run in-process with shared memory buffers (no serialization) 4. Lightweight workers: Processes start in seconds What's new in v0.3: - This release is a major milestone - we now support bo
Read the rest
th Spark 3.5 and 4.0, including the new lightweight pyspark-client. The framework automatically detects your Spark version and adjusts its behavior accordingly, so one Sail binary works across versions. Why this matters: - Spark revolutionized big data 15 years ago, but its JVM foundation struggles with modern workloads. As teams process more real-time data and AI workloads, they're hitting walls with latency, cloud costs, and operational complexity. Sail is trying to solve all of these problems while not requiring you to rewrite everything that you already did with Spark. - We're working toward unifying batch, streaming, and AI workloads in a single framework. Imagine running your ETL, real-time analytics, and model training on the same infrastructure with predictable performance. The project is open source (Apache 2.0) and we'd love your feedback! We have a growing community on Slack where early adopters are already running production workloads. GitHub: https://github.com/lakehq/sail Docs: https://lakesail.com Our internal benchmarks(for the 4x and 94% number): https://docs.lakesail.com/sail/latest/introduction/benchmark... Slack: https://lakesail.com/slack Happy to answer any questions about the architecture, benchmarks, or migration path!
In Sail: a Rust-Based Spark Replacement
chenxi9649·Nov 22, 2024
Previous discussion in September when they didn't have distributed processing: https://news.ycombinator.com/item?id=41496033 Github Repo: https://github.com/lakehq/sail Few interesting notes: - Benchmarks show 4x faster than Spark on TPC-H with 94% cost reduction. - Currently at 65.7% PySpark test compatibility(they talk about this in more detail in the post) - Built in Rust using Tokio runtime and Arrow IPC for high performance - Already supports 79/99 TPC-DS queries
In Sail 0.2: Spark replacement in Rust, runs 4x faster, drop-in PySpark compatible

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.