latent-scope

Particularly noteworthy for me (as someone doing Elixir & ML at the moment) is the 100k$ attributed to Elixir Nx: > Building a cutting-edge distributed ML framework with Elixir from Nx – Creating a scalable, distributed machine learning framework that outperforms current solutions, the Nx project leverages Elixir’s unique strengths in multitasking and fault-tolerance. By automatically spreading tasks across multiple devices, Nx aims to deliver a more scalable and efficient solution. And lots of other interesting projects, including https://github.com/enjalot/latent-scope, https://github.com/Intelligent-Instruments-Lab/tolvera, https://ente.io/, https://github.com/VikParuchuri/marker ...

Here's a great tool that does almost exactly the same thing for any dataset: https://github.com/enjalot/latent-scope Obviously the scale of OP's project adds a lot of interesting complexity, this tool cannot handle that, but it's great for medium-sized datasets.