Hey I just got back into work after the long weekend!
I thought a colleague of mine had filed an issue but I didn’t find it. I filed it myself just now: https://github.com/apache/arrow/issues/49310
In Apache Arrow is 10 years oldapache
arrow
C++2016–2026
apache / arrow
arrow
Apache Arrow is the universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics
29Distinct people
33Mentions
0Explicit recommenders
17.3KGitHub stars
What to know before exploring
What it is
Apache Arrow is the universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics
GitHub
Where it is used
No documented product use was verified in the last source check.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 11:56:56 AM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by year33 dated mentions
Observed mentions: 2016-02-17 16:17:28 UTC to 2026-02-17 15:43:16 UTC.
* An incomplete calendar period is marked with an outlined bar. Its count is not directly comparable with a full period.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2016 | 2016-01-01 00:00:00 UTC | 2017-01-01 00:00:00 UTC | 3 | 0 | 3 |
| 2017 | 2017-01-01 00:00:00 UTC | 2018-01-01 00:00:00 UTC | 2 | 0 | 2 |
| 2018 | 2018-01-01 00:00:00 UTC | 2019-01-01 00:00:00 UTC | 2 | 0 | 2 |
| 2019 | 2019-01-01 00:00:00 UTC | 2020-01-01 00:00:00 UTC | 3 | 0 | 3 |
| 2020 | 2020-01-01 00:00:00 UTC | 2021-01-01 00:00:00 UTC | 3 | 0 | 3 |
| 2021 | 2021-01-01 00:00:00 UTC | 2022-01-01 00:00:00 UTC | 7 | 1 | 6 |
| 2022 | 2022-01-01 00:00:00 UTC | 2023-01-01 00:00:00 UTC | 2 | 0 | 2 |
| 2023 | 2023-01-01 00:00:00 UTC | 2024-01-01 00:00:00 UTC | 5 | 0 | 5 |
| 2024 | 2024-01-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 2 | 0 | 2 |
| 2025 | 2025-01-01 00:00:00 UTC | 2026-01-01 00:00:00 UTC | 3 | 1 | 2 |
| 2026 (partial) | 2026-01-01 00:00:00 UTC | 2027-01-01 00:00:00 UTC | 1 | 0 | 1 |
From the project README
An excerpt from the repository, fetched Nov 13, 2026.
Read the project’s introduction
Apache Arrow
Powering In-Memory Analytics
Apache Arrow is a universal columnar format and multi-language toolbox for fast
data interchange and in-memory analytics. It contains a set of technologies that
enable data systems to efficiently store, process, and move data.
Major components of the project include:
The Arrow Columnar Format:
a standard and efficient in-memory representation of various datatypes, plain or nested
The Arrow IPC Format:
an efficient serialization of the Arrow format and associated metadata,
for communication between processes and heterogeneous environments
ADBC (Arrow Database Connectivity) ↗: Arrow-powered API,
drivers, and libraries for access to databases and query engines
The Arrow Flight RPC protocol:
based on the Arrow IPC format, a building block for remote services exchanging
Arrow data with application-defined semantics (for example a storage server or a database)
C++ libraries
C bindings using GLib
.NET libraries ↗
Gandiva:
an LLVM-based Arrow expression compiler, part of the C++ codebase
Go libraries ↗
Java libraries ↗
JavaScript libraries ↗
Julia implementation ↗
Python libraries
R libraries
Ruby libraries
Rust libraries ↗
Swift libraries ↗
The ↗ icon denotes that this component of the project is maintained in a separate
repository.
Arrow is an Apache Software Foundation project. Learn more at
arrow.apache.org.
What's in the Arrow libraries?
The reference Arrow libraries contain many distinct software components:
Columnar vector and table-like containers (similar to data frames) supporting
flat or nested types
Fast, language agnostic metadata messaging layer (using Google's FlatBuffers
library)
Reference-counted off-heap buffer memory management, for zero-copy memory
sharing and handling memory-mapped files
IO interfaces to local and remote filesystems
Self-describing binary wire formats (streaming and batch/file-like) for
remote procedure calls (RPC) and interprocess communication (IPC)
Integration tests for verifying binary compatibility between the
implementations (e.g. sending data from Java to C++)
Conversions to and from other in-memory data structures
Readers and writers for various widely-used file formats (such as Parquet, CSV)
Implementation status
The official Arrow libraries in this repository are in different stages of
implementing the Arrow format and related features. See our current
feature matrix
on git main.
How to Contribute
Please read our latest project contribution guide.
If you are using AI coding tools, please review our
AI-generated code guidance.
Getting involved
Even if you do not plan to contribute to Apache Arrow itself or Arrow
integrations in other projects, we'd be happy to have you involved:
Join the mailing list: send an email to
dev-subscribe@arrow.apache.org. Share your ideas and use cases for the
project.
Follow our activity on GitHub issues
Learn the format
Contribute code to one of the reference implementations
Continuous Integration Sponsors
We use runs-on for managing the project self-hosted runners.
We use AWS for some of the required infrastructure for the project.
Read the full README on GitHubWhat Hacker News says
33 sourcesNew Parquet writer allows easy insert/delete/edit
2 points · Open discussionFair benchmarks would justify merging aiopandas into pandas. Benchmark grid axes: aiopandas, dtype_backend="pyarrow", dask-cudf
pandas pyarrow docs:
https://pandas.pydata.org/docs/dev/user_guide/pyarrow.html
/? async pyarrow:
https://www.google.com/search?q=async+pyarrow
/? repo:apache/arrow async language:Python :
https://github.com/search?q=repo%3Aapache%2Farrow+async+lang... :
test_flight_async.py
https://github.com/apache/arrow/blob/main/python/pyarrow/tes...
pyarrow/src/arrow/python/async.h:
https://github.com/apache/arrow/blob/main/python/pyarrow/src... : "Bind a Python callback to an arrow::Future."
--
dask-cudf:
https://docs.rapids.ai/api/dask-cudf/stable/ :
> Neither Dask cuDF nor Dask DataFrame provide support for multi-GPU or multi-node execution on their own. You must also deploy a dask.distributed cluster to leverage multi
Read the rest
ple GPUs. We strongly recommend using Dask-CUDA to simplify the setup of the cluster, taking advantage of all features of the GPU and networking hardware.
cudf.pandas > FAQ > "When should I use cudf.pandas vs using the cuDF library directly?" https://docs.rapids.ai/api/cudf/stable/cudf_pandas/faq/#when... :
> cuDF implements a subset of the pandas API, while cudf.pandas will fall back automatically to pandas as needed.
> Can I use cudf.pandas with Dask or PySpark?
> [Not at this time, though you can change the dask df to e.g. cudf, which does not implement the full pandas dataframe API]
--
dask.distributed docs > Asynchronous Operation; re Tornado or asyncio:
https://distributed.dask.org/en/latest/asynchronous.html#asy...
--
tqdm.dask, tqdm.notebook: https://github.com/tqdm/tqdm#ipythonjupyter-integration
from tqdm.notebook import trange, tqdm
for n in trange(10):
time.sleep(1)
--
But then TPUs instead of or in addition to async GPUs;
TensorFlow TPU docs: https://www.tensorflow.org/guide/tpu
@1egg0myegg0 that's great to hear. I'll check to see if it applies to Arrow.
Another performance issue with DuckDB/Arrow integration that we've been working to solve is that Arrow lacked a canonical way to pass statistics along with a stream of data. So for example if you're reading Parquet files and passing them to DuckDB, you would lose the ability to pass the Parquet column statistics to DuckDB for things like join order optimization. We recently added an API to Arrow to enable passing statistics, and the DuckDB devs are working to implement this. Discussion at https://github.com/apache/arrow/issues/38837.
In Adding concurrent read/write to DuckDB with Arrow Flighthttps://github.com/apache/arrow/issues/34738
https://lists.apache.org/thread/lk277x3b9gjol42sjg27bst2ggm5...
In Show HN: Iceoryx2 – Fast IPC Library for Rust, C++, and CI learned yesterday about GoLang's assembler https://go.dev/doc/asm - after browsing how arrow is implemented for different languages (my experience is mainly C/C++) - https://github.com/apache/arrow/tree/main/go/arrow/math - there are bunch of .S ("asm" files) and I'm still not able to comprehend how these work exactly (I guess it'll take more reading) - it seems very peculiar.
The last time I've used inlined assembly was back in Turbo/Borland Pascal, then bit in Visual Studio (32-bit), until they got disabled. Then did very little gcc with their more strict specification (while the former you had to know how the ABI worked, the latter too - but it was specced out).
Anyway - I wasn't expecting to find this in "Go" :) But I guess you can always start with .go code then produce assembly (-S) then optimize it, or find/hire someone to do it.
In From slow to SIMD: A Go optimization storyOpenWRT's LuCI WebUI, Torch ML, and embedded interpreters in game engines are also written in Lua.
Apache Arrow's C GLib implementation works with Lua. From https://news.ycombinator.com/item?id=38103326 :
> Apache Arrow already supports C, C++, Python, Rust, Go and has C GLib support Lua: https://github.com/apache/arrow/tree/main/c_glib/example/lua
LearnXinYminutes Lua: https://learnxinyminutes.com/docs/lua/
OpenWRT is a Make-based Linux distro for embedded devices with limited RAM and flash ROM, x86, and docker. OpenWRT is built on `uci` (and procd and ubusd instead of systemd and dbus). UCI is an /etc/config/* dotted.key=value configuration system which procd sys-v /etc/init.d/* scripts read values in from when regenerating their configuration when $1 is e.g. 'start', 'restart', or 'reload' per sys-v. LuCI is the OpenWRT web UI which
Read the rest
is written in Lua; which is now implemented mostly as a JSON-RPC API instead of with server-side HTML templates for usability and performance on embedded devices.
Notes on how to write a LuCI app in Lua:
https://github.com/x-wrt/luci/commit/73cda4f4a0115bb05bbd3d1...
applications/luci-app-example:
https://github.com/openwrt/luci/tree/master/applications/luc...
openwrt/luci//docs:
https://github.com/openwrt/luci/tree/master/docs
https://openwrt.org/supported_devices
It's probably impossible to build OpenWRT (and opkg packages) for an RP2040W.
https://www.reddit.com/r/O3DE/comments/rdvxhx/why_python/ :
> Python is used for scripting the editor only, not in-game behaviors.
> For implementing entity behaviors the only out of box ways are C++, ScriptCanvas (visual scripting) or Lua. Python is currently not available for implementing game logic.
C++, Lua, and Python all implement CFFI (C Foreign Function Interface) for remote function and method calls.
"Using CFFI for embedding" https://cffi.readthedocs.io/en/latest/embedding.html :
> You can use CFFI to generate C code which exports the API of your choice to any C application that wants to link with this C code. This API, which you define yourself, ends up as the API of a .so/.dll/.dylib library—or you can statically link it within a larger application.
Apache Arrow already supports C, C++, Python, Rust, Go and has C GLib support
Read the rest
Lua:
https://github.com/apache/arrow/tree/main/c_glib/example/lua :
> Arrow Lua example: All example codes use LGI to use Arrow GLib based bindings
pyarrow.from_numpy_dtype:
https://arrow.apache.org/docs/python/generated/pyarrow.from_...
https://github.com/scikit-learn-contrib/sklearn-pandas :
> Sklearn-pandas: This module provides a bridge between Scikit-Learn's machine learning methods and pandas-style Data Frames. In particular, it provides a way to map DataFrame columns to transformations, which are later recombined into features.
Pandas docs > PyArrow > I/O https://pandas.pydata.org/docs/user_guide/pyarrow.html#i-o-r... :
> By default, these functions and all other IO reader functions return NumPy-backed data. These readers can return PyArrow-backed data by specifying the parameter dtype_backend="pyarrow"
> [...] Several non-IO reader functions can also use the dtype_backend argument to return PyArrow-backed data including: to_numeric() , DataFrame.convert_dtypes() , Series.convert_dtypes()
df = pandas.read_csv(".csv", dtype_backend="pyarrow")
sx = pandas.Series([1.0,2.0], dtype="float32[pyarrow]")
The CSV parser for [Perspective](https://github.com/finos/perspective) uses the [Apache Arrow](https://github.com/apache/arrow) C++ CSV parser compiled to WASM. It's not currently multi-threaded but this is possible as well to my understanding.
In Show HN: uDSV.js – A faster CSV parserWorker should really adopt Apache Arrow, which has a much bigger ecosystem.
https://github.com/apache/arrow
In Cap'n Proto 1.0`pyarrow`'s `read_csv` function[0] has just four default arguments (defaulted to None): 3 option objets and one Memory Pool option.
```
pyarrow.csv.read_csv(input_file, read_options=None, parse_options=None, convert_options=None, MemoryPool memory_pool=None)
```
You can then pass a `ReadOptions`[1] object if needed.
For example:
```
read_options = csv.ReadOptions(
column_names=["animals", "n_legs", "entry"],
skip_rows=1)
csv.read_csv(io.BytesIO(s.encode()), read_options=read_options)
```
You can see how ReadOptions is written on this link [2]. It's interesting they use a `cdef class` from `Cython` for this.
This doesn't solve all issues (the ReadOptions object and the others will inevitably have a bunch of default arguments) but I do think it's safer and it's easier to have a mental map of the things you need
Read the rest
to decide and what's decided for you.
[0] https://arrow.apache.org/docs/python/generated/pyarrow.csv.r...
[1] https://arrow.apache.org/docs/python/generated/pyarrow.csv.R...
[2] https://github.com/apache/arrow/blob/master/python/pyarrow/_...
There are several community members actively working on making things more accessible in general. (Flight SQL definitely needs some polish here.)
For Arrow itself, you could start with the Cookbook [1] (and feel free to file issues if there are things that are unclear, or that you would like to learn more about [2])
The example there is very long, but much of it is repetitive (that said, it could certainly be improved). But to your specific question, what you would eventually do is write a plugin or proxy for Postgres (or another database) implementing these RPC methods. Then you could submit SQL queries using the Flight SQL client (which calls those RPC methods internally) to get back columnar data. You could wrap this all in a JDBC driver as well (which is being actively worked on [3]), in which case you'd be mostly ignorant of what's g
Read the rest
oing on underneath; the benefit there would (hopefully) be faster data access due to the lower layers being columnar (and zero-copy where possible), and enabling parallelism due to Flight (Postgres might not support this for instance, but other databases could, if they could expose internal parallelism or partitioning). Hope that helps.
[1]: https://arrow.apache.org/cookbook/
[2]: https://github.com/apache/arrow-cookbook/
[3]: https://github.com/apache/arrow/pull/12254
I'm not tuned into Arrow all that much. I've read some of the about and stuff, but the code examples (to my eye) look really complex and complicated. [1]
Could someone point me to a more glossy "arrow flight sql for dummies" examples? What I'm gleaning from this (or am I wrong?) is you could use a JDBC driver + arrow jdbc client and write... SQL? Or is it something a lot different?
Is this the sort of thing where you could just add a plugin to postgres and be arrowified or something?
[1] https://github.com/apache/arrow/blob/release-7.0.0/java/flig...
In Apache Arrow Flight SQL: Accelerating Database AccessLots of confusion derives from the best-of-breed parquet readers for Python and R residing in the Arrow packages, mostly because Arrow is (and does) a lot of things.
There's:
* arrow, a in-memory format for dataframe-shaped things supporting fast computation, zero-copy sharing, etc.
* arrow Feather V1, an on-disk format for dataframe-shaped things
* arrow IPC, a (de)serialization for arrow buffers and a protocol for sending/receiving to other processes.
* arrow Feather V2, an on-disk format that's basically the IPC serialization written to file[1]
* arrow Flight, a protocol for requesting/sending/receiving data to remote machines that's basically gRPC layered over the IPC format
* arrow DataFusion/Ballista, nascent system(s) for local/distributed query execution over arrow buffers
* other subprojects I'm surely forgetting
* a (very good)
Read the rest
C++ Parquet reader [2]/[3] developed under the auspices of the project
* libraries in / bindings to many languages, including R and Python, supporting interaction with (subsets of) the above.
It's only the last piece that's exposed to most data-science-y users, and thus identified with the 'arrow' name. Since those libraries are also very good, and hiding their abstractions well, those users are free to use the functionality relevant to them, be it dealing with parquet, feather, etc. without needing to understand how they work.
Not that this is a criticism of the project, or those users! Arrow encompasses lots of different functionality, which enables it to provide different things to different people. As a result, though, 'Arrow' connotes lots of different things (and different _sorts_ of things) to different users, which can cause some confusion if terms aren't fully specified, or even a bit misunderstood
[1] https://stackoverflow.com/a/67911190/881025
[2] https://github.com/apache/parquet-cpp
[3] https://github.com/apache/arrow/tree/master/cpp
At this point, just don't design file format. Use flatbuffers.
1. It has good forward / backward compatibility stories;
2. Schema is your file format documentation;
3. Don't need to decode every parts;
4. Can skip for unknowns (if you use union type);
5. Many language bindings.
The only downside, and this is a big one: validate a flatbuffers payload method doesn't have nearly as much language coverage as the encode / decode methods. (i.e. C++ has it, but a lot of other languages don't have validate method implemented).
Case study - Apache Arrow uses flatbuffers to persist its metadata: https://github.com/apache/arrow/tree/master/format
In Zip – How not to design a file formatShowing the 15 most recent of 33 sources. Enable JavaScript to search and browse every indexed mention.