VIvitaliyf·Dec 1, 2024
The company I work for open-sourced a straightforward library that does exactly that: https://github.com/embrace-io/s3-batch-object-store
In Building a distributed log using S3 (under 150 lines of Go)embrace-io
s3-batch-object-store
Go2024–2024
embrace-io / s3-batch-object-store
s3-batch-object-store
Go module that allows for batch uploading of objects to a single S3 file and retrieving each object separately using the AWS S3 API, fetching only the bytes for that specific object
2Distinct people
2Mentions
0Explicit recommenders
38GitHub stars
What to know before exploring
What it is
Go module that allows for batch uploading of objects to a single S3 file and retrieving each object separately using the AWS S3 API, fetching only the bytes for that specific object
GitHub
Where it is used
No documented product use has been verified yet.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:56:07 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by week2 dated mentions
Observed mentions: 2024-07-22 16:27:43 UTC to 2024-12-01 17:22:48 UTC. Weeks start Monday, UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| Jul 22 | 2024-07-22 00:00:00 UTC | 2024-07-29 00:00:00 UTC | 1 | 0 | 1 |
| Jul 29 | 2024-07-29 00:00:00 UTC | 2024-08-05 00:00:00 UTC | 0 | 0 | 0 |
| Aug 5 | 2024-08-05 00:00:00 UTC | 2024-08-12 00:00:00 UTC | 0 | 0 | 0 |
| Aug 12 | 2024-08-12 00:00:00 UTC | 2024-08-19 00:00:00 UTC | 0 | 0 | 0 |
| Aug 19 | 2024-08-19 00:00:00 UTC | 2024-08-26 00:00:00 UTC | 0 | 0 | 0 |
| Aug 26 | 2024-08-26 00:00:00 UTC | 2024-09-02 00:00:00 UTC | 0 | 0 | 0 |
| Sep 2 | 2024-09-02 00:00:00 UTC | 2024-09-09 00:00:00 UTC | 0 | 0 | 0 |
| Sep 9 | 2024-09-09 00:00:00 UTC | 2024-09-16 00:00:00 UTC | 0 | 0 | 0 |
| Sep 16 | 2024-09-16 00:00:00 UTC | 2024-09-23 00:00:00 UTC | 0 | 0 | 0 |
| Sep 23 | 2024-09-23 00:00:00 UTC | 2024-09-30 00:00:00 UTC | 0 | 0 | 0 |
| Sep 30 | 2024-09-30 00:00:00 UTC | 2024-10-07 00:00:00 UTC | 0 | 0 | 0 |
| Oct 7 | 2024-10-07 00:00:00 UTC | 2024-10-14 00:00:00 UTC | 0 | 0 | 0 |
| Oct 14 | 2024-10-14 00:00:00 UTC | 2024-10-21 00:00:00 UTC | 0 | 0 | 0 |
| Oct 21 | 2024-10-21 00:00:00 UTC | 2024-10-28 00:00:00 UTC | 0 | 0 | 0 |
| Oct 28 | 2024-10-28 00:00:00 UTC | 2024-11-04 00:00:00 UTC | 0 | 0 | 0 |
| Nov 4 | 2024-11-04 00:00:00 UTC | 2024-11-11 00:00:00 UTC | 0 | 0 | 0 |
| Nov 11 | 2024-11-11 00:00:00 UTC | 2024-11-18 00:00:00 UTC | 0 | 0 | 0 |
| Nov 18 | 2024-11-18 00:00:00 UTC | 2024-11-25 00:00:00 UTC | 0 | 0 | 0 |
| Nov 25 | 2024-11-25 00:00:00 UTC | 2024-12-02 00:00:00 UTC | 1 | 0 | 1 |
What Hacker News says
2 sourcesTL;DR: s3-batch-object-store is a Go module that allows for batch uploading of objects to a single S3 file and retrieving each object separately using the AWS S3 API, fetching only the bytes for that specific object.
The method basically consists of appending multiple objects to a single file, keep the information of where each object is placed in the file, and then upload one single file to s3 with many objects in it, reducing drastically the number of PUT operations needed to store a large number of objets.
The developer company educed storage costs by 70% with our new approach. You can head to their s3-batch-object-store GitHub repository (https://github.com/embrace-io/s3-batch-object-store) to check out the module and try it out.
In Enhancing storage efficiency and reduce storage costs with S3-batch-object-store