BIG-bench

The actual valid counterargument is maybe Anthropic does't care about bigbench and nor should they have to, google can make up any benchmark it wants, anthropic does't have to use or care about it, or ignore it, or anything else. However, let's assume they should care because it's a major benchmark from an industry leader. The entire point of the canary string is that LLMs are supposed to discard / ignore it and data found on the same document. After all, the documentation literally says ""Do not edit the canary comments. These are to prevent BIG-bench tasks from leaking into web-scraped training data."" Anthropic did not do that (they obviously HAVE scraped data containing the GUID), therefore it is demonstrably a gotcha. e.g. it should ignore both https://github.com/google/BIG-bench/blob/main/bigbench/bench... AND https://github.com/goo