HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
awslabs
soci-snapshotter
Go2023–2026
awslabs / soci-snapshotter

soci-snapshotter

A containerd snapshotter plugin which enables standard OCI images to be lazily loaded without requiring a build-time conversion step.

3Distinct people
3Mentions
0Explicit recommenders
794GitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

A containerd snapshotter plugin which enables standard OCI images to be lazily loaded without requiring a build-time conversion step.

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 5:04:57 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by month3 dated mentions

Observed mentions: 2023-07-11 14:45:14 UTC to 2026-02-07 04:41:03 UTC.

012023-07-01 00:00:00 UTC to 2023-08-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 commentsJul 20232023-08-01 00:00:00 UTC to 2023-09-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-09-01 00:00:00 UTC to 2023-10-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-10-01 00:00:00 UTC to 2023-11-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-11-01 00:00:00 UTC to 2023-12-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-12-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-01-01 00:00:00 UTC to 2024-02-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-02-01 00:00:00 UTC to 2024-03-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-03-01 00:00:00 UTC to 2024-04-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-04-01 00:00:00 UTC to 2024-05-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-05-01 00:00:00 UTC to 2024-06-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-06-01 00:00:00 UTC to 2024-07-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-07-01 00:00:00 UTC to 2024-08-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-08-01 00:00:00 UTC to 2024-09-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-09-01 00:00:00 UTC to 2024-10-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-10-01 00:00:00 UTC to 2024-11-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 commentsOct 20242024-11-01 00:00:00 UTC to 2024-12-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-12-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-01-01 00:00:00 UTC to 2025-02-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-02-01 00:00:00 UTC to 2025-03-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-03-01 00:00:00 UTC to 2025-04-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-04-01 00:00:00 UTC to 2025-05-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-05-01 00:00:00 UTC to 2025-06-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-06-01 00:00:00 UTC to 2025-07-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-07-01 00:00:00 UTC to 2025-08-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-08-01 00:00:00 UTC to 2025-09-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-09-01 00:00:00 UTC to 2025-10-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2025-10-01 00:00:00 UTC to 2025-11-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-11-01 00:00:00 UTC to 2025-12-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-12-01 00:00:00 UTC to 2026-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2026-01-01 00:00:00 UTC to 2026-02-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2026-02-01 00:00:00 UTC to 2026-03-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 commentsFeb 2026

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
Jul 20232023-07-01 00:00:00 UTC2023-08-01 00:00:00 UTC101
Aug 20232023-08-01 00:00:00 UTC2023-09-01 00:00:00 UTC000
Sep 20232023-09-01 00:00:00 UTC2023-10-01 00:00:00 UTC000
Oct 20232023-10-01 00:00:00 UTC2023-11-01 00:00:00 UTC000
Nov 20232023-11-01 00:00:00 UTC2023-12-01 00:00:00 UTC000
Dec 20232023-12-01 00:00:00 UTC2024-01-01 00:00:00 UTC000
Jan 20242024-01-01 00:00:00 UTC2024-02-01 00:00:00 UTC000
Feb 20242024-02-01 00:00:00 UTC2024-03-01 00:00:00 UTC000
Mar 20242024-03-01 00:00:00 UTC2024-04-01 00:00:00 UTC000
Apr 20242024-04-01 00:00:00 UTC2024-05-01 00:00:00 UTC000
May 20242024-05-01 00:00:00 UTC2024-06-01 00:00:00 UTC000
Jun 20242024-06-01 00:00:00 UTC2024-07-01 00:00:00 UTC000
Jul 20242024-07-01 00:00:00 UTC2024-08-01 00:00:00 UTC000
Aug 20242024-08-01 00:00:00 UTC2024-09-01 00:00:00 UTC000
Sep 20242024-09-01 00:00:00 UTC2024-10-01 00:00:00 UTC000
Oct 20242024-10-01 00:00:00 UTC2024-11-01 00:00:00 UTC000
Nov 20242024-11-01 00:00:00 UTC2024-12-01 00:00:00 UTC000
Dec 20242024-12-01 00:00:00 UTC2025-01-01 00:00:00 UTC000
Jan 20252025-01-01 00:00:00 UTC2025-02-01 00:00:00 UTC000
Feb 20252025-02-01 00:00:00 UTC2025-03-01 00:00:00 UTC000
Mar 20252025-03-01 00:00:00 UTC2025-04-01 00:00:00 UTC000
Apr 20252025-04-01 00:00:00 UTC2025-05-01 00:00:00 UTC000
May 20252025-05-01 00:00:00 UTC2025-06-01 00:00:00 UTC000
Jun 20252025-06-01 00:00:00 UTC2025-07-01 00:00:00 UTC000
Jul 20252025-07-01 00:00:00 UTC2025-08-01 00:00:00 UTC000
Aug 20252025-08-01 00:00:00 UTC2025-09-01 00:00:00 UTC000
Sep 20252025-09-01 00:00:00 UTC2025-10-01 00:00:00 UTC101
Oct 20252025-10-01 00:00:00 UTC2025-11-01 00:00:00 UTC000
Nov 20252025-11-01 00:00:00 UTC2025-12-01 00:00:00 UTC000
Dec 20252025-12-01 00:00:00 UTC2026-01-01 00:00:00 UTC000
Jan 20262026-01-01 00:00:00 UTC2026-02-01 00:00:00 UTC000
Feb 20262026-02-01 00:00:00 UTC2026-03-01 00:00:00 UTC101

What Hacker News says

3 sources
There's three main things we do to solve this, all of which relate to the fact that we have our own (OCI-compatible) container runtime under the hood instead of using Docker. 1. We don't gzip layers like Docker does. Gzip is really slow, and it's much slower than the network. Storage is cheap. So it's much faster to transmit uncompressed layers than to transmit compressed layers and decompress them. 2. We've heavily tuned our agents for pulling layers fast. Disk throughput and IOPS are really important so we provision those higher than you typically would for running workloads in the cloud. When pulling layers we modify kernel parameters like the dirty_ratio to values that we've empirically found with layer pulls. We make sure we completely exhaust our network bandwidth and throughput when pulling layers. And so on. 3. This third one is e
Read the rest
xperimental and something we're actively working on improving, but we have our own underlying filesystem which lazily loads the files from a layer instead of pulling tons of (potentially unneeded) files up front. This is similar to AWS's [Seekable OCI](https://github.com/awslabs/soci-snapshotter) but tuned for our particular needs. I've been slowly working on improving our documentation to explain these kinds of differentiators that our architecture and container runtime provide, but most of it is unpublished so far. We definitely need to do a much better job of explaining _how_ we are faster and better rather than just stating it :). The other side of this is that we also made _building_ those layers much much faster. We blogged a little bit about it at https://www.rwx.com/blog/we-deleted-our-dockerfiles but just to hit some quick notes: in RWX you can vary the compute by task, and it turns out throwing a big machine at (e.g.) `npm install` is quite effective. Plus we make using an incremental cache very easy, and layers generated from an incremental cache are only the incremental parts, so they tend to be smaller. And we're a DAG, so you can parallelize your setup in a way that is very painful to do with Docker, even when using multi-stage builds. And our cache registry is global and very hard to mess up, whereas a lot of people misconfigure their Docker caches and have cache misses all over their docker builds. And we have miss-then-hit semantics for caching. Okay, I'm rambling now! But happy to go into more depth on any of this!
In GitHub Actions is slowly killing engineering teams
Containerd/nerdctl supports a number of snapshotter plugins: Nydus, e Stargz, SOCI: Seekable OCI, fuse-overlayfs; containerd/stargz-snapshotter: https://github.com/containerd/stargz-snapshotter containerd/nerdctl//docs/nydus.md: https://github.com/containerd/nerdctl/blob/main/docs/nydus.m... : nydusify and Check Nydus image: https://github.com/dragonflyoss/nydus/blob/master/docs/nydus... : > Nydusify provides a checker to validate Nydus image, the checklist includes image manifest, Nydus bootstrap, file metadata, and data consistency in rootfs with the original OCI image. Meanwhile, the checker dumps OCI & Nydus image information to output (default) directory. nydus: https://github.com/dragonflyoss/nydus awslabs/soci-snapshotter: https://github.com/awslabs/soci-snapshotter ; lazy start standard OCI images /? lxc copy on write: https://
Read the rest
www.google.com/search?q=lxc+copy+on+write : lxc-copy supports btrfs, zfs, lvm, overlayfs lxc/incus: "Add OCI image support" https://github.com/lxc/incus/issues/908 opencontainers/image-spec; OCI Image spec: https://github.com/opencontainers/image-spec opencontainers/distribution-spec; OCI Image distribution spec: https://github.com/opencontainers/distribution-spec But then in the opencontainers/runtime-spec//config.md OCI runtime spec TODO bundle config.json there is an example of a config.json https://github.com/opencontainers/runtime-spec/blob/main/con... The LXC approach is to run systemd in the container. The quadlet approach is to not run systemd /sbin/init in the container; instead create .container files in /etc/containers/systemd/ (rootful) or ~/.config/containers/systemd/*.container (for rootless) so that the host systemd manages and logs the container processes. Then realized you said QEMU not LXC. LXD: https://canonical.com/lxd : > LXD provides both [QEMU,] KVM-based VMs and system containers based on LXC – that can run a full Linux OS – in a single open source virtualisation platform. LXD has numerous built-in management features, including live migration, snapshots, resource restrictions, projects and profiles, and governs the interaction with various storage and networking options. From https://documentation.ubuntu.com/lxd/latest/reference/storag... : > LXD supports the following storage drivers for storing images, instances and custom volumes: > Btrfs, CephFS, Ceph Object, Ceph RBD, Dell PowerFlex, Pure Storage, HPE Alletra, Directory, LVM, ZFS You can run Podman or Docker within an LXD host; with or without a backing storage pool. FWIU it's possible for containers in an LXD VM to use BTRFS, ZFS, or KVM storage drivers to create e.g. BTRFS subvolumes instead of running overlayfs within the VM by editing storage.conf.
In How Container Filesystem Works: Building a Docker-Like Container from Scratch
So I can give some behind the scenes insight on that. I don't think image caching will be a thing in the way people are explicitly asking, but we are exploring some alternative approaches to speeding up container launch that we think will actually be even more effective than what people are asking for. First of all we want to leverage some of the learnings from AWS Lambda, in specific some of the research we've done that shows that about 75% of container images only contain 5% unique bytes (https://brooker.co.za/blog/2023/05/23/snapshot-loading.html). This makes deduplication incredibly effective, and allows the deployment of a smart cache that holds the 95% of popular recurring files and file chunks from container images, while letting the unique 5% be loaded over the network. There will be outliers of course, but if you base your image
Read the rest
off a well used base image then it will already be in the cache. This is partially implemented. You will notice that if you use certain base images your Fargate tasks seems to start a bit faster. (Unfortunately we do not really publish this list or commit to what base images are in the cache at this time). In another step along this path we are working on SOCI Snapshotter (https://github.com/awslabs/soci-snapshotter) forked off of Stargz Snapshotter. This allows a container image to have an attached index file that actually allows it to start up before all the contents are downloaded, and lazy load in remaining chunks of the image as needed. This takes advantage of another aspect of container images which is that many of them don't actually use all of the bytes in the image anyway. Over time we want to make these two pieces (deduplication and lazy loading) completely behind the scenes so you just upload your image to Elastic Container Registry and AWS Fargate seems to magically start your image dramatically faster than you could locally if downloading the image from scratch.
In Kubernetes SidecarContainers feature is merged

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.