audiocraft

There are! Audio models are actually quite similar to image models, but there are a few key differences. First, is the autoencoder needs to be designed much more carefully as human hearing is insanely good and music requires orders of magnitude more spatial compression (image AEs do 8X8 downsampling, audio AEs need to do thousands of times downsampling). Second the model itself needs to be really good at placing lyrics/beats (similar to placing text in image diffusion): a sixth finger in an image model is fine, but a missed beat can ruin a song. That's why language model approaches (which have a stronger sequential inductive bias than diffusion models which is good for rhythm and lyric placement) have been really popular in audio. If you're interested in papers (IMO not good for new people as they make everything seem more complicated tha

Meta says to imagine you can: "Imagine a professional musician being able to explore new compositions without having to play a single note on an instrument. Or an indie game developer populating virtual worlds with realistic sound effects and ambient noise on a shoestring budget. Or a small business owner adding a soundtrack to their latest Instagram post with ease." In reality, you can't, as they licensed the weights for noncommercial use only: https://github.com/facebookresearch/audiocraft#license

The source code is, as it's MIT, but the weights are not, as they're CC-BY-NC: https://github.com/facebookresearch/audiocraft#license

Sadly looks unlikely if the base model wasn't trained on vocals. > Mitigations: Vocals have been removed from the data source using corresponding tags, and then using a state-of-the-art music source separation method, namely using the open source Hybrid Transformer for Music Source Separation (HT-Demucs). > Limitations: The model is not able to generate realistic vocals. (https://github.com/facebookresearch/audiocraft/blob/main/mod...) I suspect this was a combination of playing it safe and that the model isn't well architected to reproduce meaningful vocals.

The model cards from the repo[0] link to Colab and HF spaces. [0]: https://github.com/facebookresearch/audiocraft#models

This is purely a function of everyone remembering the RIAA's decade-long campaign to prevent people from taking the music they had rightfully stolen. As far as I'm aware LLaMA was trained on "publicly available data"[0], not "licensed data". Furthermore, MusicGen's weights are licensed CC-BY-NC, which is effectively a nonlicense as there is no noncommercial use you could make of an art generator[1]. This is not only a 'weights-available' license, but it's significantly more restrictive than the morality clause bearing OpenRAIL license that Stability likes to use[2]. [0] https://github.com/facebookresearch/llama/blob/main/MODEL_CA... [1] https://github.com/facebookresearch/audiocraft/blob/main/LIC... [2] These are also very much Not Open Source™ but the morality clauses in OpenRAIL are at least non-onerous enough to collaborate over.

And here's the code: https://github.com/facebookresearch/audiocraft just so you know for the weights specifically: "The weights in this repository are released under the CC-BY-NC 4.0 license as found in the LICENSE_weights file." Combine it with AI voices and voice cloning and so begins the further devaluation of musicians and artists. Might as well accelerate it and see what happens. What could possibly go wrong?

The submitted URL is broken currently https://github.com/facebookresearch/audiocraft works