CLAP

Not exactly what you asked for, but are you aware of CLAP? https://github.com/LAION-AI/CLAP This generates audio embeddings - much like CLIP does for visual inputs.

I've tried self-hosting with navidrome [0] / plex / jellyfin but the thing I miss most is music discovery via radios / discover weekly. I've tried replicating it a few times with embedding vectors + vector search but at best it finds songs in the (sub)-genre with the tempo / mood being pretty different. Maybe I just need better data, been meaning to try again when that spotify crawl by annas-archive gets released. I've just been using musicbrainz [1] and youtube. Model-wise I've tried off-the-shelf ones like [2] and [3] and training auto-encoders like VAEs / MAEs [4] [0] - https://www.navidrome.org/ [1] - https://musicbrainz.org/ [2] - https://github.com/LAION-AI/CLAP [3] - https://github.com/SonyCSLParis/music2latent [4] - https://arxiv.org/pdf/2207.06405