unilm

I found a fairly large improvement in my toy transformer model where I added a "global" token akin to the CLS token in ViT. Another approach I've seen is the "Diff transformer" from MS Research (https://github.com/microsoft/unilm/tree/master/Diff-Transfor...).

There is a downside, every attention layer has to effectively compute attention twice (run scaled_dot_product_attention). As scaled_dot_product_attention is usually one of the most expensive operations in training and inference of a model, it seems like networks using this may be substantially slow and perhaps should considered against larger networks with more attention layers. https://github.com/microsoft/unilm/blob/master/Diff-Transfor...

Has anyone tried Kosmos [0] ? I came across it the other day and it looked shiny and interesting, but I haven't had a chance to put it to the test much yet. [0] - https://github.com/microsoft/unilm/tree/master/kosmos-2.5

BitNet Implementations: * https://huggingface.co/1bitLLM/bitnet_b1_58-large * https://github.com/Oxen-AI/BitNet-1.58-Instruct * https://github.com/nkotak/1.58BitNet See some followups that has some training advice: https://github.com/microsoft/unilm/tree/master/bitnet

> Then there's a trend towards much shorter numbers. 16 bit floating point? 8 bit? 2 bit? 1 bit? There was that recent paper titled "The Era of 1-bit LLMs" [0] which was actually suggeting a 1.58 bit LLM (2 bits in practice). > Someone reading this is probably writing it in VHDL right now, or will be soon. Yeah, I think I'm in the "will be soon" camp - FPGA board has been ordered. Especially with the 2-bit data types outlined in that paper [0] and more details in [1]. There's really a need for custom hardware to do that 2-bit math efficiently. Customizing one of the simpler open source RISC-V integer implementations seems like something to try here adding in the tiled matrix registers and custom instructions for dealing with them (with the 2 bit data types). [0] https://arxiv.org/abs/2402.17764 [1] https://github.com/microsoft/unilm/blob

+1 On this, the real proof would have been testing both models side-by-side. It seems that it may be published on GitHub [1] according to HuggingFace [2]. [1] https://github.com/microsoft/unilm/tree/master/bitnet [2] https://huggingface.co/papers/2402.17764

I think it's definitely coming, with more recent advances in unsupervised learning and multimodal learning (along the lines of https://github.com/microsoft/unilm or https://github.com/Alpha-VLLM/LLaMA2-Accessory)

e5-mistral is essentially a distillation from gpt-4 to a smaller model. You can see here https://github.com/microsoft/unilm/blob/16da2f193b9c1dab0a69... they actually have custom prompts for each dataset being tested. Question would be, if you haven't seen the task before, what is a good prompt to prepend for your task? IMO e5-mistral is overfit to MTEB

They link to this repo at the top of the paper https://github.com/microsoft/unilm although I don't see anything that actually says "Kosmos". Slightly interestingly, the last commit changes a heading from "AI" to "AGI" https://github.com/microsoft/unilm/commit/bbbb5b4b06c2dd501d... The model is relatively small, 1.6B. I am guessing the goal is to be able to run on a user's home PC. But it would be interesting to see how much better it gets if you scale it up by a factor of 10 or 100.

This seems amazing. Has anyone here tried to actually use this stuff? I am earnestly trying to create a website that can code simple applications for users. Or at least highly intelligent technical users or programmers. In particular it would be great to have an alternative to relying 100% on OpenAI for code. Or if there is for example some model that can "see" web page layout and also output markup.. I may have to experiment with some of these like this one maybe https://github.com/microsoft/unilm/tree/master/markuplm