Discrete-Distribution-Networks.github.io

Please check this new promising GenAI architecture namely Discrete Distribution Networks or DDN that's recently posted here at HN [1]. The proposed DDN can even work as LLM based on binary strings instead of text [3]. [1] Show HN: I invented a new generative model and got accepted to ICLR (91 comments): https://news.ycombinator.com/item?id=45536694 [2] DDN+GPT for LLM: https://github.com/Discrete-Distribution-Networks/Discrete-D...

Exactly what i think! - The DDN single-shot generator architecture is more efficient than diffusion. - DDN is fully end-to-end differentiable, allowing for more efficient optimization when integrated with discriminative models or reinforcement learning. - Moreover, DDN inherently avoids mode collapse. These points are all mentioned in the blog: https://github.com/Discrete-Distribution-Networks/Discrete-D...

Thank you very much for your interest. 1. The comparison with GANs and the issue of mode collapse are addressed in Q2 at the end of the blog: https://github.com/Discrete-Distribution-Networks/Discrete-D... 2. Regarding scalability, please see “Future Research Directions” in the same blog: https://github.com/Discrete-Distribution-Networks/Discrete-D... 3. Answers or relevant explanations to any other questions can be found directly in the original paper (https://arxiv.org/abs/2401.00036), so I won’t restate them here.

The first version of DDN was developed in less than three months, almost entirely by one person. Consequently, the experiments were preliminary and the results far from SoTA. The current goal in research is scaling up. Here are some thoughts in blog about future directions: https://github.com/Discrete-Distribution-Networks/Discrete-D...

Yes, it's absolutely possible—just like how diffusion LLMs work, we can do the same with DDN LLMs. I made an initial attempt to combine [DDN with GPT](https://github.com/Discrete-Distribution-Networks/Discrete-D...), aiming to remove tokenizers and let LLMs directly model binary strings. In each forward pass, the model adaptively adjusts the byte length of generated content based on generation difficulty (naturally supporting speculative sampling).

The posts mentions that: https://github.com/Discrete-Distribution-Networks/Discrete-D...