HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
dremio
gandiva
OPEN SOURCE2018–2020
dremio / gandiva · Archived

gandiva

Vectorized processing for Apache Arrow

Explore on GitHubRead Hacker News discussions
3Distinct people
3Mentions
0Explicit recommenders
484GitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Vectorized processing for Apache Arrow

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 5:04:57 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by month3 dated mentions

Observed mentions: 2018-06-24 19:33:42 UTC to 2020-02-24 19:32:49 UTC.

012018-06-01 00:00:00 UTC to 2018-07-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 commentsJun 20182018-07-01 00:00:00 UTC to 2018-08-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2018-08-01 00:00:00 UTC to 2018-09-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments2018-09-01 00:00:00 UTC to 2018-10-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2018-10-01 00:00:00 UTC to 2018-11-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2018-11-01 00:00:00 UTC to 2018-12-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2018-12-01 00:00:00 UTC to 2019-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2019-01-01 00:00:00 UTC to 2019-02-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2019-02-01 00:00:00 UTC to 2019-03-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2019-03-01 00:00:00 UTC to 2019-04-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2019-04-01 00:00:00 UTC to 2019-05-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 commentsApr 20192019-05-01 00:00:00 UTC to 2019-06-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2019-06-01 00:00:00 UTC to 2019-07-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2019-07-01 00:00:00 UTC to 2019-08-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2019-08-01 00:00:00 UTC to 2019-09-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2019-09-01 00:00:00 UTC to 2019-10-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2019-10-01 00:00:00 UTC to 2019-11-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2019-11-01 00:00:00 UTC to 2019-12-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2019-12-01 00:00:00 UTC to 2020-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2020-01-01 00:00:00 UTC to 2020-02-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2020-02-01 00:00:00 UTC to 2020-03-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 commentsFeb 2020

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
Jun 20182018-06-01 00:00:00 UTC2018-07-01 00:00:00 UTC101
Jul 20182018-07-01 00:00:00 UTC2018-08-01 00:00:00 UTC000
Aug 20182018-08-01 00:00:00 UTC2018-09-01 00:00:00 UTC110
Sep 20182018-09-01 00:00:00 UTC2018-10-01 00:00:00 UTC000
Oct 20182018-10-01 00:00:00 UTC2018-11-01 00:00:00 UTC000
Nov 20182018-11-01 00:00:00 UTC2018-12-01 00:00:00 UTC000
Dec 20182018-12-01 00:00:00 UTC2019-01-01 00:00:00 UTC000
Jan 20192019-01-01 00:00:00 UTC2019-02-01 00:00:00 UTC000
Feb 20192019-02-01 00:00:00 UTC2019-03-01 00:00:00 UTC000
Mar 20192019-03-01 00:00:00 UTC2019-04-01 00:00:00 UTC000
Apr 20192019-04-01 00:00:00 UTC2019-05-01 00:00:00 UTC000
May 20192019-05-01 00:00:00 UTC2019-06-01 00:00:00 UTC000
Jun 20192019-06-01 00:00:00 UTC2019-07-01 00:00:00 UTC000
Jul 20192019-07-01 00:00:00 UTC2019-08-01 00:00:00 UTC000
Aug 20192019-08-01 00:00:00 UTC2019-09-01 00:00:00 UTC000
Sep 20192019-09-01 00:00:00 UTC2019-10-01 00:00:00 UTC000
Oct 20192019-10-01 00:00:00 UTC2019-11-01 00:00:00 UTC000
Nov 20192019-11-01 00:00:00 UTC2019-12-01 00:00:00 UTC000
Dec 20192019-12-01 00:00:00 UTC2020-01-01 00:00:00 UTC000
Jan 20202020-01-01 00:00:00 UTC2020-02-01 00:00:00 UTC000
Feb 20202020-02-01 00:00:00 UTC2020-03-01 00:00:00 UTC101

What Hacker News says

3 sources
gopalv·Feb 24, 2020
> I don't understand why SQL on LLVM? SQL is mostly "do the same thing a billion times", which aligns very closely with generating code for the inner loop, particularly when the inner loop can be generated without branches. SQL codegen is basically an industry standard for large-scale sql query processing on top of a large-scale column store. I work one of the few engines at this scale which does not do go query stage codegen & instead uses source codegen vectorized primitives instead of query time (Apache Hive, specifically & the only other one in the same style is Google's supersonic engine). Apache Impala - SQL to LLVM IR (C++) https://llvm.org/devmtg/2013-11/slides/Wanderman-Milne-Cloud... [PDF] Gandiva - SQL to LLVM IR (Java) https://github.com/dremio/gandiva MemSQL - SQL to MPL to LLVM IR http://highscalability.com/blog/2016/9/7/cod
Read the rest
e-generation-the... Greenplum - SQL to LLVM IR http://engineering.pivotal.io/post/codegen-gpdb-qx/ Postgres 9.4 Vitesse - SQL to LLVM IR <https://www.postgresql.org/message-id/CAJNt7%3DZ6w5%2BwyeTKK... Postgres 11 LLVM - SQL to LLVM IR https://www.postgresql.org/docs/11/jit-reason.html SparkSQL - SQL to Java source (Janino + javac) https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark... And as far as I know Redshift's horrible query compilation performance is derived from using C++ codegen which forks gcc inside (not sure why this is so slow though, almost feels like untuned gcc flags or something). https://docs.aws.amazon.com/redshift/latest/dg/c-query-plann...
In Snel: SQL Native Execution for LLVM
chrisaycock·Jun 24, 2018
This blog post is the first mention I'd seen of Gandiva: https://github.com/dremio/gandiva As I began reading about that, I thought it sounded a lot like Weld, so I was happy to see it listed in the prior art: https://www.weld.rs These projects present a compiler IR that can be emitted by an analytics framework. For example, Spark and pandas currently run functions directly to perform filters and aggregations. These can be fast for simple user cases, but complex user code might result in a lot of branching and temporary values. So instead, the libraries could emit a mid-level IR that is then JITed via LLVM. That's exactly what Weld is for and it looks like Gandiva is similar (on top of Arrow data). I've seen a lot of interest in LLVM-based array programming lately, including TensorFlow XLA and Intel's nGraph. So I'm glad to see these tech
Read the rest
niques applied to tabular data as well.
In The Origin and History of Apache Arrow

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.