mergekit

You don't have to teach an transformer model using a language corpus even if that was the pretraining. You can e.g. write algorithms directly and merge them into the model. https://github.com/yashbonde/rasp https://github.com/arcee-ai/mergekit

> Can I actually add "features" to the model? Yes. You can use a number of libraries to add, mix, merge, etc. layers [1] > Not with weights. I need it to learn case histories to extend its "feature-set". Again, yes. You can add attention heads, other features, heck you can even add classification if you want [2]. Because you are working with an open architecture! What you think of weights are not binary blobs. That is a common missconception. [1] - https://github.com/arcee-ai/mergekit [2] - https://github.com/center-for-humans-and-machines/transforme...

For others like me who’d not heard of merging before, this seems to be one tool[0] (there may be others) [0] https://github.com/arcee-ai/mergekit

mergekit is the tool you need to do this https://github.com/cg123/mergekit you can slice off layers and blend models with different strategies.

Ah, actually reviewing that more closely I found a link to it in the acknowledgements. https://github.com/cg123/mergekit