transformer-arithmetic

Check out my project https://github.com/Thopliterce/transformer-arithmetic. This is a concrete implementation based on GPT-2 model that does multiplication accurately, digit by digit. It does so by generating a dataset that tells the model how to do multiplication step by step. Doing arithmetic actually works with just GPT-2, without an oracle.

> The accuracy largely comes from the fact that addition rarely requires carrying more than a single digit. So it's easy to pattern match from single digit problems that it was previously trained on. Again, not at all true due to BPEs. [12] [65] [25] [42] [185] + [580] [22] [75] [80] [93] --------------------------- [706] [75] [300] [278] (note that GPT-3 is never told that the BPE [580] is composed of the digits [5], [8], and [0]. It has to guess this from the contexts [580] occurs in.) > With multiplication, which requires much more extensive cross-column interaction, accuracy falls off a cliff with anything more than a few digits. You couldn't learn long multiplication if you had to use BPEs, were never told how BPEs worked or corresponded to sane encodings, were basically never shown how to do multiplication, and