RSrspeer·Dec 8, 2015
What's hard, however, is telling multi-byte CJK encodings apart, and telling them apart from UTF-8.
I've tried it, as I was working on extending my mojibake-fixing library [1] to the language that gave us the word "mojibake". The ambiguous cases actually happen.
[1] http://github.com/LuminosoInsight/python-ftfy
Suppose you found this string (which someone actually tweeted) encoded in Shift-JIS:
(|| * m *)ｳ､ｳｯﾌﾟ･･
Those half-width katakana with a comma in the middle look a lot like typical Japanese mojibake, so the classifier would be entirely justified in deciding that Shift-JIS is wrong. Next let's try decoding those bytes as EUC-JP instead:
(|| * m *)海劾餅ゥ
Hey, maybe that's a name or something! No other encoding works, so the classifier happily decides it's EUC-JP. Except this string is complete nonsense and the person actually
Read the rest
meant the first string.
I do agree with you that it's a problem worth trying to solve (clearly). We can't just throw up our hands and say "this is a problem that shouldn't happen, so let's not solve it". Microsoft Office exists, and its mojibake-causing "features" will never be removed due to backward compatibility, so this happens and will continue to happen.
But, any ideas where the corpus for telling apart encodings should actually come from? I've been using Twitter, but this limits the domain. Only thoroughly defective Twitter clients send anything but UTF-8 to Twitter. There are lots of defective Twitter clients, it turns out, but the creative mistakes that Excel users make every day are one in a billion on Twitter. I can apply artificial mojibake, but then I'm not correlating the text with the encoding it's likely to be in.