bark

To my knowledge, the model being used for this is "chirp" which is 'based on' bark[1], an AI text to speech model. The github page for bark links to a page about chirp, which returns a 404 page for me [2]. My guess is that the model used for suno.ai's song generator isn't too much different than the text to speech model. I also have a hunch is that it was more like a coincidence than intentional that the bark model was capable of producing music, and that was spun off into this product. Unfortunately, there seems to still be issues with bark when generating long (like book length) spoken audio. Which is too bad, as someone who's worked jobs that require lots of driving, it would be awesome to be able to have any text read to me in a natural sounding voice. [1]https://github.com/suno-ai/bark [2] https://www.suno.ai/examples/chirp-v1

Stable-audio and MusicGen sounds better than Jukebox. But the best so far is Suno.ai ( https://app.suno.ai ) especially with their V3 model they have very impressive results, the fidelity is not studio quality but they're getting very close. It's very likely based on their TTS model they have released before (Bark), but trained on more data and with higher resolution. https://github.com/suno-ai/bark

I don't have any special insight into how it works, but I suspect it is largely synthesizing audio from scratch. The more I've thought about it, the task of generating music feels very similar to the task of text-to-speech with realistic intonation. So feels like the same techniques would be applicable. Suno do have an open source repo here that presumably uses similar tech: https://github.com/suno-ai/bark > Bark was developed for research purposes. It is not a conventional text-to-speech model but instead a fully generative text-to-audio model, which can deviate in unexpected ways from provided prompts. Suno does not take responsibility for any output generated. Use at your own risk, and please act responsibly. I've generated probably >200 songs now with Suno, of which perhaps 10 have been any good, and I can't detect any pattern in

hahah wow! cool :-) PS: OT, I am reading this Bark thing(https://github.com/suno-ai/bark). Can I run it locally on a Macbook 2015 with 8GB RAM?

Bark seems pretty good https://github.com/suno-ai/bark Demo at https://huggingface.co/spaces/suno/bark In the couple samples I tried it was substantially better at picking up meaning compared to VALL-E-X

Previously TortoiseTTS was associated with PlayHT in some way, although the exact connection is a bit vague [0]. From the descriptions here it sounds a lot like AudioLM / SPEAR TTS / some of Meta's recent multilingual TTS approaches, although those models are not open source, sounds like PlayHT's approach is in a similar spirit. The discussion of "mel tokens" is closer to what I would call the classic TTS pipeline in many ways... PlayHT has generally been kind of closed about what they used, would be interesting to know more. If you are interested in some recent open to sample-from work pushing on this kind of random expressiveness (sometimes at the expense of typical "quality" in terms of TTS), Bark is pretty interesting [1]. Though the audio quality suffers a bit from how they realize sequences -> waveforms, the prosody and timing is re

With some tinkering you can create really interesting stuff with Bark. I managed to generate a couple of song snippets / intros using free form text [1] Haven't tested it personally yet but if you are interested in voice cloning, you might wanna check this fork of Bark [2] [1] https://github.com/suno-ai/bark/discussions/249 [2] https://github.com/serp-ai/bark-with-voice-clone

Tortoise looks really nice! The output is very "polished" and audiobook-like. It's a contrast to Bark[0] which is far more expressive but unpredictable. [0]: https://github.com/suno-ai/bark

Is prompt engineering a snake oil? The engineering, as the word stands so far in history is about observing clear limitations and capabilities of something and then "engineering" things around it such as "engineering a compiler" is all about knowing the underlying processor, memory and other characteristics and then engineering a solution that converts a text notation to a stream of instructions that try to strike a balanced tradeoff. But with LLMs, no one knows their workings once they are trained. The other day, Bark model[0] for text to speech, the team itself has following to say on details: > Below is a list of some known non-speech sounds, but we are finding more every day. Please let us know if you find patterns that work particularly well on Discord! [laughter], [laughs], [sighs].... So that's the team themselves not knowing what