apache

hive

Java2014–2021

apache / hive

hive

Apache Hive

Data & Databases

Explore on GitHub Read Hacker News discussions

2Distinct people

6Mentions

0Explicit recommenders

6KGitHub stars

REPOSITORY AT A GLANCE

What to know before exploring

What it is

Apache Hive

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 1:49:46 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year6 dated mentions

Observed mentions: 2014-04-04 07:58:12 UTC to 2021-10-04 04:47:40 UTC.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges

UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
Period	Start (UTC)	End (exclusive, UTC)	Mentions	Comments
2014	2014-01-01 00:00:00 UTC	2015-01-01 00:00:00 UTC	1	1
2015	2015-01-01 00:00:00 UTC	2016-01-01 00:00:00 UTC	0	0
2016	2016-01-01 00:00:00 UTC	2017-01-01 00:00:00 UTC	0	0
2017	2017-01-01 00:00:00 UTC	2018-01-01 00:00:00 UTC	0	0
2018	2018-01-01 00:00:00 UTC	2019-01-01 00:00:00 UTC	0	0
2019	2019-01-01 00:00:00 UTC	2020-01-01 00:00:00 UTC	0	0
2020	2020-01-01 00:00:00 UTC	2021-01-01 00:00:00 UTC	0	0
2021	2021-01-01 00:00:00 UTC	2022-01-01 00:00:00 UTC	5	5

What Hacker News says

6 sources

gopalv·Oct 4, 2021

> whether this would really work out in most workloads > just because it keeps the cache-lines hotter and less likely to be evicted. Okay, so keeping cache for a bloom filter problem is real - but the real force evicting memory out of the cache line is the next row-group you read + all the other stuff you have to do when you implement this in a database product. Memory bandwidth is contested heavily and the caches will get knocked out every 1024 rows. So the two things I work with, Apache Hive and Apache Impala switched to a blocked bloom filter at different points in time. Hive BloomKFilter - https://github.com/apache/hive/blob/master/storage-api/src/j... Impala/Kudu one - https://github.com/apache/impala/blob/master/be/src/kudu/uti... The C++ one also has an AVX specialization, while the Java one relies on the JVM to do it (not always)

Read the rest

- https://github.com/apache/impala/blob/master/be/src/kudu/uti... We ran a lot of trivial benchmarks and several benchmarks where the shuffle-join+network (not sort-merge, this is just a partitioned hash join) generates a bloom filter (a semijoin) before sending rows out and the 1-cache line version won out when the bloom filter went slightly over the 1 Million + 5% rate [1]. The regular bloom filter went from (38ns -> 108ns for 1k -> 1m items), while the BloomK stuck at (27ns) despite making room for a million times more items in the bloom. The bloom-1 (which is the 64bit version) underperformed on accuracy (was ~2x faster at 16ns per op, but worse at filtering out items). [1] - https://github.com/prasanthj/bloomfilter/tree/master/benchma...

In Word-Aligned Bloom Filters

gopalv·Aug 24, 2021

> though given how fast multipliers are these days using a prime modulus They are really fast on a single thread, but at least on Xeons I use the mul and modulo seems to have a weak spot with thread parallelism though. To fix a set of threading scaling issue with LIKE "%pattern%" SQL queries (something like TPC-H Query13), I spent a week digging through the algorithms which work better when all the threads are doing the same thing. I narrowed down on BoyerMooreHorspool[1] as the algorithm which doesn't seem to be hit by other threads running the same algorithm (of course, everything utf-8 matching turns unicode char patterns into exact byte lookups). [1] - https://github.com/apache/hive/blob/master/storage-api/src/j...

In Rolling hash; Rabin-Karp string search

gopalv·Jun 22, 2021

> the abstraction is specifying the order of operations unnecessarily That needs an "in SQL", the standard imperative language operator ordering has short-cut operations in there (a is null or a.value == true) etc. In the code I work with, this actually sorts the conditions based on estimated selectivity[1] and type (long compares to constant are cheaper on a columnar data-set due to the SIMD, but string isn't etc). > Is there any reason to choose binary trees for this? The parse-tree does come off as binary because inserting logical parentheses makes it easier to tackle, because there are association rules which neatly go into a BinaryOp structure when dealing with operator precedence in parsing. So it is easier to handle the parsing when you treat (a + (b + c) ) and (a / (b / c)) in similar fashion. I won't make the same mistake again i

Read the rest

f I have to build a SQL engine, but this actually made the logical expression match the parse tree very closely and was a good enough generalization until the traversal time bugs[2] started to pop up. [1] - https://github.com/apache/hive/blob/master/common/src/java/o... [2] - https://issues.apache.org/jira/browse/HIVE-9166

In Accidentally exponential behavior in Spark

gopalv·Jun 21, 2021

I've hit almost the exact same issue with Hive, with a somewhat temporary workaround (like this post) to build a balanced tree out of this by reading it into a list [1] and rebuilding a binary balanced tree out of it. But we ended up implementing a single level Multi-AND [2] so that this no longer a tree for just AND expressions & can be vectorized neater than the nested structure with a function call for each (this looks more like a tail-call rather than a recursive function). The ORC CNF conversion has a similar massively exponential item inside which is protected by a check for 256 items or less[3]. [1] - https://github.com/t3rmin4t0r/captain-hook/blob/master/src/m... [2] - https://issues.apache.org/jira/browse/HIVE-11398 [3] - https://github.com/apache/hive/blob/master/storage-api/src/j...

In Accidentally exponential behavior in Spark

gopalv·Mar 22, 2021

Reading this code almost makes me want to think about how differently I review Java than C or C++. ch = ++index < strlen ? str.charAt(index) : 0; If I saw that code scattered across in some java code, I would immediately be like - "rewrite please". But the same code fragment in C would be like - yeah, we need a bounds check, good job, can you make it a macro please. The java default Float parser was somewhat of a complete perf headache when multi-threading, more than about single core consumption (till jdk8b96 - JDK-7032154), so in the past Hive has shipped with its own strod impl[1] to avoid the central lock in parseDouble. Also it worked off a raw utf8 byte[] stream, which is more common coming out of a disk file or network serialization rather than a CharSequence. [1] - https://github.com/apache/hive/blob/4446414f4478091db1eb20bc...

In FastDoubleParser: Java port of Daniel Lemires fast_double_parser

rusanu·Apr 4, 2014

Good work, and I'm impressed by the PostgreSQL foreign data wrapped API power to allow for such ease of implementation. The .proto defined for cstore_fwd differs from the ORC as defined for Hive. At a quick glance I can't find references to dictionary encoding nor statistics, and the datatypes used are apparently the native PostgreSQL data types. From what I can tell this implementation leverages reduced IO (fetch only columns of interest from disk), segment elimination (use min/max info to skip over entire row groups) and pgzl compression for the data stream. I couldn't find references to run-length encoding (RLE) or dictionary encoding. I'm sure the shortcoming will be improved in future iterations, specially better encoding schemes which would result in better compression. But I'm a bit disappointed that the ORC format used is not the

Read the rest

same as the one originally used in Hive, Pig and the rest of Java/Hadoop ecosystem. Had it shared the actual file format it would had enabled a number of very interesting scenarios. Think about Hive/PIG serving as ETL pipelines to produce ORC file that are attached directly to PostgreSQL (add today's partition as FOREIGN DATA WRAPPER ... OPTIONS (filename '.../4_4_2014.cstore') and then do interactive analytic driven by PostgreSQL. It would reduce the cost of ingress significantly (fast attach, no need for expensive INSERT and re-compression). I realize data type translation would had been thorny, to say the least (datetime and decimal as primitives, probably many of the structured types too). - github.com/citusdata/cstore_fdw/blob/master/cstore.proto - github.com/apache/hive/blob/trunk/ql/src/protobuf/org/apache/hadoop/hive/ql/io/orc/orc_proto.proto

In PostgreSQL Columnar Store for Analytic Workloads