minbpe

Hehe. https://github.com/karpathy/minbpe/blob/master/minbpe/base.p... This code will incorrectly count pairs in [1,1,1] as two pairs of (1,1), despite there is only one, if you consider what merge function does. ;) This can be seen in real world scenarios, for example, when there are many spaces. My attempt to implement BPE has the same flaw and the fix appear to be very complex. But, as I looked at stats, it just makes such pairs to appear as top ones a couple of runs earlier, so it appears as a benign flaw. But, it's interesting that this is not mentioned in the code.