nix-snapshotter

Definitely don't recommend going down this path if you're not already familiar with Nix, but if you are, a strategy that I find works really well is to package your software with Nix, then you can run it easily via systemd but also create super lightweight containers using nix-snapshotter[0] so you don't have to "build" container images if you still want the flexibility of containers. You can then run the containers on Docker or Kubernetes without having to build heavy images. [0] https://github.com/pdtpartners/nix-snapshotter

seems similar to this https://github.com/pdtpartners/nix-snapshotter so kind of allowing pull images from nix store, mounting shared host nix store per node into each container, incremental fast rebuilds, generating basic pod configs are good things. and local, ci and remote runs same flows and envs.

Why does the "Comparison" table show "Nix" as "No" for "Containerized"? One can [run Nix in containerized environments](https://nix.dev/manual/nix/2.24/installation/installing-dock...), [build containerized environments from Nix](https://nix.dev/tutorials/nixos/building-and-running-docker-...), and [even run Nix-generated environments directly in containerd through a plugin](https://github.com/pdtpartners/nix-snapshotter/blob/main/doc...). I believe the former two apply to Guix as well which is also marked as "No" for "Containerized".

> so (from my perspective), they punted the entire concept of making what I would refer to as a "layer aware linux distribution" Would you consider there to be any 'layer-aware Linux distributions' today, e.g., NixOS, GuixSD, rpm-ostree-based distros like Fedora CoreOS, or distri? > so much duplicate layers out in the world Have you seen this, which lets existing container systems understand a Linux package manager's packages as individual layers? https://github.com/pdtpartners/nix-snapshotter

Not super recent, but nix-snapshotter is one that I'd call awesome(but I'm also a k8s fanboi): https://github.com/pdtpartners/nix-snapshotter https://news.ycombinator.com/item?id=37407758

Excellent info! I started head-deving a project similar to nix-snapshotter[0] and I was thinking "ok, I can probably just build CRI impl that builds a rootfs dir with nix and just shell out to bubblewrap to make a "container". But once I went through that mental exercise I started reading code in containerd and cri-o. Wow, these are _not_ simple projects; containerd itself having a full GRPC-based service registry for driving dynamic logic via config. One thing I was pretty disappointed about is how deeply ingrained OSI images are in the whole ecosystem. While you can replace almost all functional parts of runtime, but not really the concept of images. I think images are a poor solution to the problem they solve, and a big downside of this is a bunch of complexity in the runtimes trying to work around how images work (like remote snapshot

Does anyone here have any experience using https://github.com/pdtpartners/nix-snapshotter ? I build a lot of Docker images using Nix, and while yes it’s generally more pleasant than using Dockerfiles, the 128 layer limit is really annoying and easy to hit when you start building images with Nix. The workaround of grouping store paths makes poor use of storage and bandwidth.

> What is the difference between "native understanding of nix packages" versus DIY putting your project derivation into dockerTools.buildImage? This is addressed in the readme FAQ. https://github.com/pdtpartners/nix-snapshotter#faq If you're running Nix-based images on Kubernetes, seems like this tool has reduced overhead / storage compared to if you build an OCI/Docker image.

> I assume that if the nix daemon is being started with each pod, then you're downloading a fresh copy of everything every time I got the impression that it uses the node's nix daemon. From the project readme's FAQ https://github.com/pdtpartners/nix-snapshotter#faq > What's the difference between this and a nix-in-docker? > If you run nix inside a container (e.g. nixos/nix or nixpkgs/nix-flake) then you are indeed fetching packages using the Nix store. However, each container will have its own Nix store instead of de-duplicating at the host level. > nix-snapshotter is intended to live on the host system (sibling to containerd and/or kubelet) so that multiple containers running different images can share the underlying packages from the same Nix store.

If you are pushing nix-snapshotter images to a Docker Registry, then it can only either use what's in your host nix store, or fetch from a binary cache. The same is true if you are using the special image reference like `nix:0/nix/store/f8b1hia3hcqwa5d46anzy3cszi3s6ybk-nix-image-redis.tar`, it just means that instead of fetching the image manifest from a Registry, you are using the Nix protocols. You will only build from source dynamically (and a mix of caching) if you are doing `kubectl apply -f ${podSpec}` as a full deployment Nix expression. See the README's asciinema and this file as an example: https://github.com/pdtpartners/nix-snapshotter/blob/v0.1.0/e...

Hello! This is Edgar and Robbie and we built nix-snapshotter. nix-snapshotter brings native understanding of Nix packages to containerd. We built this because Nix is a great fit for making efficient containers. They don't need an OS because Nix captures all dependencies exactly. However, the current process of creating Nix images is subpar because one needs to transform Nix packages into a format that container runtimes understand. Using nix-snapshotter, instead of downloading image layers, packages come directly from the Nix store. Packages can be fetched from a binary cache or built on the fly if necessary. All existing non-Nix images continue to be supported, and Nix layers can be interleaved with normal layers. nix-snapshotter also provides a CRI image service, which allows Kubernetes to resolve image manifests from Nix directly too.