GLM-130B

The ones I saw mentioned so far were Flan, Cerebras, GPT-J, and RWKV. Not yet mentioned: * Pythia https://github.com/EleutherAI/pythia * GLM-130B https://github.com/THUDM/GLM-130B - see also ChatGLM-6B https://github.com/THUDM/ChatGLM-6B * GPT-NeoX-20B https://huggingface.co/EleutherAI/gpt-neox-20b * GeoV-9B https://github.com/geov-ai/geov * BLOOM https://huggingface.co/bigscience/bloom and BLOOMZ https://huggingface.co/bigscience/bloomz

> GLM-130B, a model comparable with GPT-3, has 130 billion parameters in FP16 precision, a total of 260G of GPU memory is required to store model weights. The DGX-A100 server has 8 A100s and provides an amount of 320G of GPU memory (640G for 80G A100 version) so it suits GLM-130B well. https://github.com/THUDM/GLM-130B/blob/main/docs/low-resourc...

GLM-130B[1] is something comparable to GPT-3. It's a 130billion parameter model vs GPT-3's 175 billion, and it can comfortably run on current-gen high end consumer level hardware. A system with 4 RTX 3090s (< $4k) gets results in about 16 seconds per query. The proverbial 'some guy on Twitter'[2] got it setup, and broke down the costs, demonstrated some prompts, and what not. The output's pretty terrible, but it's unclear to me whether that's inherent or a result of priority. I expect OpenAI spent a lot of manpower on supervised training, whereas this system probably had minimal, especially in English (it's from a Chinese university). If these technologies end up as anything more than a 'novelty of the year' type event, then I expect to see them able to be run locally on phones within a decade. There will be a convergence between hardware

GLM-130B[1] (a 130 billion parameter model vs GPT-3's 175 billion parameter model) is able to run optimally on consumer level high-end hardware, 4xRTX 3090 in particular. That's < $4k at current prices, and as hardware prices go one can only imagine what it'll be in a year or two. It also enables running with degraded performance on lesser systems. It's a whole lot cheaper to run neural net style systems than to train them. "Somebody on Twitter"[2] got it setup, and broke down the costs, demonstrated some prompts, and what not. Cliff notes being a fraction of a penny per query, with each taking about 16s to generate. The output's pretty terrible, but it's unclear to me whether that's inherent or a result of priority. I expect OpenAI spent a lot of manpower on supervised training, whereas this system probably had minimal, especially in Eng

Can you think of any non-weapons examples where centralization/gatekeeping of a tech meaningfully and causally benefited society or a technology itself? Actually, thinking about my own question I'm even inclined to remove the non-weapons qualifier. The most knee jerk response, nuclear weapons, is perhaps the best example of unexpected benefit. The 'decentralization' of nuclear weapons is undoubtedly why the Cold War was the Cold War, and not World War 3. And similarly why we haven't* seen an open war between nations with nuclear weapons. One power to rule over all suddenly turned into "war with this country no longer has a win scenario" effectively ending open warfare between nuclear nations. There's also the inevitability/optics argument. There are already viable open source alternatives [1], and should this tech ultimately prove viable/

https://github.com/THUDM/GLM-130B might be a useful place to look

There are a handful of "open source LLM" initiatives out there, although I don't think any of them are quite up to the level of ChatGPT. Possibly one of the more interesting ones is GLM-130B. https://github.com/THUDM/GLM-130B Released by some folks at Tsinghua University in China, back in August. The model itself is licensed under some janky "free to use, but not open source" license, but it looks like most of the code for training, evaluation, etc. is available and licensed under either the Apache License or a BSD-like license. You might also find this of interest: https://arxiv.org/pdf/2103.08894 - "Distributed Deep Learning Using Volunteer Computing-Like Paradigm" FWIW, I tend to agree with your overall sentiment. As AI becomes progressively more capable, it represents an ever increasing possibility of consolidating more and more power