SHshrikant·Mar 21, 2022
I see that you're primarily looking into Python work, so I'd recommend `smart_open` as a nice, compact way to get started.
https://github.com/RaRe-Technologies/smart_open
In Ask HN: Codebases with great, easy to read code?piskvorky
smart_open
Python2015–2022
piskvorky / smart_open
smart_open
Utils for streaming large files (S3, HDFS, gzip, bz2...)
3Distinct people
4Mentions
0Explicit recommenders
3.5KGitHub stars
What to know before exploring
What it is
Utils for streaming large files (S3, HDFS, gzip, bz2...)
GitHub
Where it is used
No documented product use has been verified yet.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:56:07 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by year4 dated mentions
Observed mentions: 2015-01-25 22:58:45 UTC to 2022-03-21 14:35:00 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2015 | 2015-01-01 00:00:00 UTC | 2016-01-01 00:00:00 UTC | 2 | 2 | 0 |
| 2016 | 2016-01-01 00:00:00 UTC | 2017-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2017 | 2017-01-01 00:00:00 UTC | 2018-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2018 | 2018-01-01 00:00:00 UTC | 2019-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2019 | 2019-01-01 00:00:00 UTC | 2020-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2020 | 2020-01-01 00:00:00 UTC | 2021-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2021 | 2021-01-01 00:00:00 UTC | 2022-01-01 00:00:00 UTC | 1 | 0 | 1 |
| 2022 | 2022-01-01 00:00:00 UTC | 2023-01-01 00:00:00 UTC | 1 | 0 | 1 |
What Hacker News says
4 sourcesSee also smart_open: https://github.com/RaRe-Technologies/smart_open which might be more user-friendly? Never used it myself but it was on HN before. Discussion on their bugtracker: https://github.com/RaRe-Technologies/smart_open/issues/579
Personally I wasn't too impressed with fsspec, though I am using it. I had to wrap files with Python's io wrappers to get acceptable performance, and found that different fsspec implementations still have significant differences that you can't ignore. They don't seem interested in supporting or documenting use cases outside of Pandas and Dask.
In Fsspec: Filesystem Interfaces for PythonSmart_open: stream S3 files from Python
1 points · Open discussionStreaming large files from/to S3 [Python]
1 points · Open discussion