Data provenance

We look for thirteen consecutive words in our training data that also appear literally in a test question. For that we first put both texts in lower case and normalise every character, so that a difference in formatting cannot hide a hit. Every hit is then checked against a longer yardstick: sixteen words, then twenty, then twenty-five. Twenty decides. A hit that disappears there was a fixed expression; what remains counts as a leak. The threshold is zero and it applies per dataset.

A test question shorter than that long yardstick gets no exemption. Such a question cannot meet that yardstick by definition. The multiple-choice questions are precisely the short ones. An exemption would therefore cover exactly the questions that matter.

For the measurements of July that is a fixed set of 35 files from EuroEval 17.6.0, together 29,273 records, each with its own checksum. Of those, 18,145 come from the test split, 8,674 from the train split and 2,454 from the validation split. Without that freeze a second round measures something else and there is nothing left to compare.

On 30 July we measured the two source files of our own source that preserves the formatting of text in the corpus. At thirteen words the outcome there comes to zero. On 12 September, after training, we measured the instruction data and the post-training data the model was actually trained on. There two questions from squad-nl were touched at thirteen words; at sixteen words they were gone and at twenty nothing was touched. That is a fixed phrase and the outcome is clean. The continued-pretraining corpus, in the build the model was trained on, was not measured against the test sets again as a whole. The pass of 22 July covered an earlier build.

We also measure at eight words. That measurement does not count towards the result and is there only as a check. On 30 July it touched 723 and 591 of the 29,273 records, per source. On 12 September it touched 659 items in the instruction data and 33 in the post-training data. At such a short fragment ordinary language comes along, such as treaty titles and licence headers. Those figures are here all the same, because a reader who finds them elsewhere would otherwise wonder why they were missing.

One hit did meet the long yardstick and was therefore a real leak, the Lord's Prayer in test question 1941 of wiki-lingua-nl. That was in the first pass of 22 July, over an earlier build of the corpus with the instruction data of that moment. We name it, because a check that never finds anything says more about how the search was done. How we dealt with it was fixed beforehand and the number of that question is in the pre-registration.