unilm

There is a downside, every attention layer has to effectively compute attention twice (run scaled_dot_product_attention). As scaled_dot_product_attention is usually one of the most expensive operations in training and inference of a model, it seems like networks using this may be substantially slow and perhaps should considered against larger networks with more attention layers. https://github.com/microsoft/unilm/blob/master/Diff-Transfor...

Has anyone tried Kosmos [0] ? I came across it the other day and it looked shiny and interesting, but I haven't had a chance to put it to the test much yet. [0] - https://github.com/microsoft/unilm/tree/master/kosmos-2.5

BitNet Implementations: * https://huggingface.co/1bitLLM/bitnet_b1_58-large * https://github.com/Oxen-AI/BitNet-1.58-Instruct * https://github.com/nkotak/1.58BitNet See some followups that has some training advice: https://github.com/microsoft/unilm/tree/master/bitnet

> Then there's a trend towards much shorter numbers. 16 bit floating point? 8 bit? 2 bit? 1 bit? There was that recent paper titled "The Era of 1-bit LLMs" [0] which was actually suggeting a 1.58 bit LLM (2 bits in practice). > Someone reading this is probably writing it in VHDL right now, or will be soon. Yeah, I think I'm in the "will be soon" camp - FPGA board has been ordered. Especially with the 2-bit data types outlined in that paper [0] and more details in [1]. There's really a need for custom hardware to do that 2-bit math efficiently. Customizing one of the simpler open source RISC-V integer implementations seems like something to try here adding in the tiled matrix registers and custom instructions for dealing with them (with the 2 bit data types). [0] https://arxiv.org/abs/2402.17764 [1] https://github.com/microsoft/unilm/blob

+1 On this, the real proof would have been testing both models side-by-side. It seems that it may be published on GitHub [1] according to HuggingFace [2]. [1] https://github.com/microsoft/unilm/tree/master/bitnet [2] https://huggingface.co/papers/2402.17764

I think it's definitely coming, with more recent advances in unsupervised learning and multimodal learning (along the lines of https://github.com/microsoft/unilm or https://github.com/Alpha-VLLM/LLaMA2-Accessory)

e5-mistral is essentially a distillation from gpt-4 to a smaller model. You can see here https://github.com/microsoft/unilm/blob/16da2f193b9c1dab0a69... they actually have custom prompts for each dataset being tested. Question would be, if you haven't seen the task before, what is a good prompt to prepend for your task? IMO e5-mistral is overfit to MTEB