ControlNet

re: parlor trick Are you referring to: https://github.com/lllyasviel/controlnet ?

Midjourney provides much better images by default. It's really impressive. Stable Diffusion's advantage is in the huge amount of open source activity around it. Most recently that resulted in ControlNet, which is far more powerful than anything Midjourney can currently do - if you know how to use it. https://github.com/lllyasviel/ControlNet

https://github.com/lllyasviel/ControlNet Controlnet is a neural network added to an already trained model so they can be conditioned on new stuff like canny edge, depth map, segmentation map. Controlnet let you train this model on the new condition "easily", without catastrophic forgetting and without a huge dataset. In the repo linked by OP, they have trained a controlnet model on the segmentation map generated by SAM: https://segment-anything.com/

ControlNet maybe? https://github.com/lllyasviel/ControlNet At huggingface: https://huggingface.co/spaces/hysts/ControlNet Play around with the different models, you might get better results with some vs others.

I mentioned it in another response as well, but here's the original implementation repo for anyone interested: https://github.com/lllyasviel/ControlNet

Github link to actual source project for anyone curious: >https://github.com/lllyasviel/ControlNet

If you're creating the normal maps yourself, perhaps you would benefit from generating the image from that using https://github.com/lllyasviel/ControlNet#controlnet-with-nor...

Stable diffusion + ControlNet is fire! Nothing compares to it. ControlNet allows you to have tight control over the output. https://github.com/lllyasviel/ControlNet

Hey, Scribble Diffusion author here. Thanks for your kind words and feedback. I love seeing all these links to your scribbles. I'm an engineer at Replicate, which is a place to run ML models in the cloud. [0] We built Scribble Diffusion as an open-source app [1] to demonstrate how to use Replicate. This is all built on ControlNet [2], a brilliant technique by Lvmin Zhang and Maneesh Agrawala [3] for conditioning diffusion models on additional sources, in this case human scribbles. It also allows for controlling Stable Diffusion using other inputs like pose estimation, edge detection, and depth maps. ControlNet has only existed for three weeks, but people are already doing all kinds of cool stuff with it, like an app [4] that lets you pose a stick figure and generate DreamBooth images that match the pose. There are already a bunch of model

The whole thing with SD is extremely interesting and difficult to keep up with as there are so many new things coming in almost daily. Just last week I saw ControlNet[1] which ads a lot more control. Today I saw what corridor crew[2] did to stabilize the randomness when you want to make videos. Very exciting. [1] https://github.com/lllyasviel/ControlNet [2] https://www.youtube.com/watch?v=_9LX9HSQkWo

Hi HN! We're working on an AI studio product, and have been experimenting with the recently-released ControlNet (https://github.com/lllyasviel/ControlNet) After wasting hours with it, we decided to put together a quick fun project. "Sketch" allows you to draw anything (works w/ touch inputs on mobile too), enter a prompt (it can be as simple as "drawing" and it'll try to guess) and you get something out fast. Keeping it all on a public feed as well, to serve as inspiration. Hope anyone here finds it fun :-)

Note that this article is from four months ago. Their Patreon and Kickstarter have been suspended but they collected money on their website and they will first focus on making yet another anime model. The big news in the Stable Diffusion land has been the release of ControlNet: https://github.com/lllyasviel/ControlNet. It hasn't gotten much traction on HN: https://news.ycombinator.com/item?id=34761780. It allows you to maintain the shape of existing images (or sketch new ones) and use Stable Diffusion to fill them out, my example: https://twitter.com/LechMazur/status/1626668677473918981.