How to check an ML benchmark for train/test copies

PowerCons is a small time series dataset: 180 training rows, 180 test rows, two classes. It is one of the 128 datasets in the UCR archive, which researchers use to compare classifiers. 67 of its 180 test rows are also in its training set, value for value. A 1-nearest-neighbour classifier gets all 67 right, because each one has its own answer sitting in the training data.

I found this while auditing the UCR and UEA archives row by row, and the audit became a preprint. This post leaves the statistics to the paper and covers the check itself. It takes about 20 lines of Python and a fraction of a second, and you can point it at your own benchmark today. I ran every code block below against the real archive files, and every output is pasted from those runs.

Why nothing warns you

A random split assumes the rows are independent. Copies break that assumption silently. Nothing throws, the shapes are right, the class counts look fine, and the score goes up. A higher score looks like a better model, so nobody goes looking for the cause.

Comparing every test row with every training row would find them, but that is n * m comparisons of long vectors. A hash set does it in one pass over each split. Give each row a fingerprint, put the training fingerprints in a set, and look up each test row.

Two details decide whether this is useful. Leave the label out of the hash, because a copy with a different label is still a copy, and it is one of the more interesting things you can find. And keep the row indices rather than a count, because every later step needs them.

Run it on a .ts file

UCR and UEA ship in the .ts format. audit_ts.py reads it with only the standard library. A row's identity is its text with the label cut off, so this is the whole idea:

def read_ts(path):
    keys, labels = [], []
    in_data = False
    with open(path, "rb") as f:
        for raw in f:
            line = raw.strip()
            if not line or line.startswith(b"#"):
                continue
            if in_data:
                body, _, label = line.rpartition(b":")
                keys.append(hashlib.md5(body).digest())
                labels.append(label.decode())
            elif line.lower().startswith(b"@data"):
                in_data = True
    if not keys:
        sys.exit(f"{path}: no @data rows found")
    return keys, labels

MD5 is fine here. Nobody is attacking this hash, and all it has to do is tell rows apart. Download the archive zip, then run it on a train and test pair:

git clone https://github.com/mahmadmanzoor/archive-audit
python archive-audit/audit_ts.py PowerCons_TRAIN.ts PowerCons_TEST.ts

n_train: 180
n_test: 180
duplicate_rows: 130
test_rows_with_train_twin: 67
test_twin_pct: 37.22
twin_rows_with_conflicting_label: 0
groups_with_conflicting_labels: 0

test_rows_with_train_twin is the number that matters. duplicate_rows counts across both splits: 360 rows in total, 230 distinct, so 130 are repeats. The two conflict lines count copies whose labels disagree, and here there are none.

It stays fast on bigger data. Crop has 24,000 rows, and the same command finished in 0.06 seconds and reported 2,753 of 16,800 test rows, or 16.39%. Add --json to get the row numbers.

The same check for arrays and dataframes

Your benchmark probably is not in .ts files. Once the data is numeric you can hash the values instead of the text, which also means 1.0 and 1.00 finally match.

import hashlib
import numpy as np

def fingerprints(X, decimals=6):
    """One hash per row, after rounding away float noise."""
    X = np.asarray(X, dtype=np.float64)
    X = np.round(X, decimals) + 0.0  # + 0.0 turns -0.0 into 0.0
    return [hashlib.blake2b(row.tobytes(), digest_size=16).hexdigest()
            for row in X]

def leak_report(X_train, X_test, decimals=6):
    train = set(fingerprints(X_train, decimals))
    test = fingerprints(X_test, decimals)
    twins = [i for i, h in enumerate(test) if h in train]
    pct = round(100 * len(twins) / len(test), 2)
    return {"test_rows": len(test), "twins": len(twins),
            "pct": pct, "indices": twins}

row.tobytes() does not care about shape, so this works on a batch of images as well as on a table. If you want to follow along with the archive files, this loader turns them into arrays:

import numpy as np

def load_ts(path):
    X, y = [], []
    in_data = False
    for line in open(path):
        line = line.strip()
        if not line or line.startswith("#"):
            continue
        if in_data:
            body, label = line.rsplit(":", 1)
            X.append([float(v) for v in body.split(",") if v != "?"])
            y.append(label)
        elif line.lower().startswith("@data"):
            in_data = True
    return np.array(X), np.array(y)

The decimals argument needs checking, because rounding is a trade-off. Too strict and float noise hides real copies. Too loose and different rows collapse into one. So run it at several values and see if the count holds:

X_train, y_train = load_ts("PowerCons_TRAIN.ts")
X_test, y_test = load_ts("PowerCons_TEST.ts")
for d in (2, 3, 4, 6, 10):
    print(d, leak_report(X_train, X_test, decimals=d)["twins"])

noisy = X_test + 1e-9
strict = leak_report(X_train, noisy, decimals=12)["twins"]
rounded = leak_report(X_train, noisy, decimals=6)["twins"]
narrow = leak_report(X_train, X_test.astype(np.float32))["twins"]
print("noise, decimals=12:", strict)
print("noise, decimals=6: ", rounded)
print("float32:           ", narrow)

2 67
3 67
4 67
6 67
10 67
noise, decimals=12: 0
noise, decimals=6:  67
float32:            67

The count is flat from 2 to 10 decimals, so these are real copies and not an artefact of rounding. The noise lines show why exact bytes are not enough: add 1e-9 to every value and a strict match finds nothing at all, while decimals=6 still finds all 67. If your own count climbs as you lower decimals, you are starting to merge rows that are merely similar.

Exact matching also misses a copy that has been rescaled. If your pipeline normalises each row, hash after normalising:

def znorm(X):
    X = np.asarray(X, dtype=np.float64)
    std = X.std(axis=1, keepdims=True)
    std[std == 0] = 1
    return (X - X.mean(axis=1, keepdims=True)) / std

scaled = X_test * 2
raw = leak_report(X_train, scaled)
normed = leak_report(znorm(X_train), znorm(scaled), decimals=3)
print("raw:         ", raw["twins"])
print("z-normalised:", normed["twins"])

raw:          0
z-normalised: 67

With a dataframe, pandas has the hash built in. Drop the label column first:

import pandas as pd
from pandas.util import hash_pandas_object

columns = [f"x{i}" for i in range(X_train.shape[1])]
train_df = pd.DataFrame(X_train, columns=columns).assign(label=y_train)
test_df = pd.DataFrame(X_test, columns=columns).assign(label=y_test)

features = [c for c in train_df.columns if c != "label"]
def row_hashes(df):
    return hash_pandas_object(df[features].round(6), index=False)

is_copy = row_hashes(test_df).isin(set(row_hashes(train_df)))
print(is_copy.sum(), f"{is_copy.mean():.2%}")

67 37.22%

Those are the same 67 rows the array version found.

Measure what the copies do to your score

A leak rate tells you copies exist. It does not tell you what they cost. For that, split your accuracy into twin rows and the rest:

from sklearn.neighbors import KNeighborsClassifier

X_train, y_train = load_ts("PowerCons_TRAIN.ts")
X_test, y_test = load_ts("PowerCons_TEST.ts")

report = leak_report(X_train, X_test)
is_twin = np.zeros(len(X_test), dtype=bool)
is_twin[report["indices"]] = True

model = KNeighborsClassifier(n_neighbors=1).fit(X_train, y_train)
pred = model.predict(X_test)
correct = pred == y_test
print(f"all rows:  {correct.mean():.2%} ({len(correct)})")
print(f"twin rows: {correct[is_twin].mean():.2%} ({is_twin.sum()})")
print(f"the rest:  {correct[~is_twin].mean():.2%} ({(~is_twin).sum()})")

all rows:  97.78% (180)
twin rows: 100.00% (67)
the rest:  96.46% (113)

This is raw values on the one default split, so treat it as a demonstration. The twin rows are a free 100%, and removing them takes the score down by over a point. Worms on the same setup goes from 45.45% to 37.31%.

For a fairer estimate the paper averages 30 resamples with and without twins:

• 1-NN on z-normalised values: PowerCons 91.67% to 86.63%, Worms 47.75% to 35.52%, Crop 71.06% to 65.47%.
• MiniRocket (2,000 kernels): PowerCons 95.52% to 92.83%, Worms 70.48% to 66.48%, Crop 69.99% to 65.44%.

So this is not only a nearest-neighbour effect. A model that trains on a row and is tested on that same row benefits whatever its design. On the published fold-0 predictions, the mean over 17 classifiers on Crop goes from 74.7% to 69.8%.

Check both directions

Everything above asks how much of the test set is in train. InsectEPGSmallTrain shows why that is not enough:

X_a, _ = load_ts("InsectEPGSmallTrain_TRAIN.ts")
X_b, _ = load_ts("InsectEPGSmallTrain_TEST.ts")

in_train = leak_report(X_a, X_b)["twins"]
in_test = leak_report(X_b, X_a)["twins"]
print(in_train, "of", len(X_b), "test rows are in train")
print(in_test, "of", len(X_a), "train rows are in test")

15 of 249 test rows are in train
15 of 17 train rows are in test

Reported the usual way, this dataset leaks 6% and you move on. Flip the arguments and 15 of its 17 training rows are in the test set. Freezer's SmallTrain variant has 26 of 28, while MixedShapes' is clean. A dataset that exists to test learning from a few examples should not test on nearly all of those examples.

Read the copies before you call it a bug

Not every copy is a mistake. ElectricDevices has 208 test rows with a training twin, 2.7% of its test set. But 54 of its twin groups carry conflicting labels, and 101 test rows have no training twin with their own label. Over 30 resamples, removing the twins there raises 1-NN accuracy slightly, from 69.39% to 69.65%, because twin rows only score 61.5%. So copies do not always help.

The reason shows up when you count distinct values in each twin row:

def median_distinct(X_tr, X_te):
    rows = X_te[leak_report(X_tr, X_te)["indices"]]
    counts = [len(set(row)) for row in rows]
    return int(np.median(counts)), rows.shape[1]

X_e, _ = load_ts("ElectricDevices_TRAIN.ts")
X_f, _ = load_ts("ElectricDevices_TEST.ts")
print("ElectricDevices:", median_distinct(X_e, X_f))
print("PowerCons:      ", median_distinct(X_train, X_test))

ElectricDevices: (2, 96)
PowerCons:       (77, 144)

Each result is the median number of distinct values and the row length. ElectricDevices twins are mostly on/off patterns, so two different appliances can produce the same row by coincidence. PowerCons twins are far richer, which makes a chance match much less believable. A copy can also be legitimate in the sense that real deployments see it too. So I would report the score both ways, and I would not quietly delete rows.

Do not loosen the match

The tempting upgrade is near-duplicate detection, because exact matching obviously misses some copies. Here is what a strict-looking correlation threshold does on OliveOil:

X_o, _ = load_ts("OliveOil_TRAIN.ts")
X_q, _ = load_ts("OliveOil_TEST.ts")

best_r = (znorm(X_q) @ znorm(X_o).T / X_o.shape[1]).max(axis=1)
share = (best_r >= 0.9999).mean()
print("best r >= 0.9999:", f"{share:.0%}", "of test rows")
print("lowest best r:   ", round(float(best_r.min()), 6))
print("exact twins:     ", leak_report(X_o, X_q)["twins"])

best r >= 0.9999: 100% of test rows
lowest best r:    0.999963
exact twins:      0

Every test row correlates above 0.9999 with something in train, and none of them is a copy. These are smooth spectra, and smooth spectra correlate. The paper sees the same thing on Wine (92.6%) and Meat (83.3%). A near-match is fine as a lead to go and look at, but I only count exact matches as leakage, and I accept that the result is a lower bound.

Fix the split

If you control the split, make the fingerprint the group, so all copies of a row land on the same side:

from sklearn.model_selection import (GroupShuffleSplit,
                                     StratifiedGroupKFold)

X = np.vstack([X_train, X_test])
y = np.concatenate([y_train, y_test])
groups = np.array(fingerprints(X))

def show(tr, te):
    twins = leak_report(X[tr], X[te])["twins"]
    print(len(tr), "train,", len(te), "test,", twins, "twins")

splitter = GroupShuffleSplit(n_splits=1, test_size=0.5, random_state=0)
show(*next(splitter.split(X, y, groups=groups)))

folds = StratifiedGroupKFold(n_splits=2, shuffle=True, random_state=0)
show(*next(folds.split(X, y, groups=groups)))

180 train, 180 test, 0 twins
181 train, 179 test, 0 twins

GroupShuffleSplit does not balance classes, so use StratifiedGroupKFold when that matters. Both keep repeated rows together instead of deleting them.

If the split is fixed, because it is the official one, keep it and report a second number without the twins. You do not need to retrain anything. Drop the rows from your existing predictions. For UCR and UEA the indices are already in the release folder:

import json

path = "release/clean_default_test_ucr.json"
entry = json.load(open(path))["PowerCons"]
drop = entry["test_rows_to_drop"]
keep = np.setdiff1d(np.arange(entry["n_test"]), drop)

before = (pred == y_test).mean()
after = (pred[keep] == y_test[keep]).mean()
print(f"{before:.2%} -> {after:.2%}")

97.78% -> 96.46%

Make it fail in CI

A check you ran once protects one version of the data. Put it next to the code that builds the split, so a new data drop cannot bring copies back unnoticed:

def assert_no_copies(X_train, X_test, max_pct=0.0, decimals=6):
    report = leak_report(X_train, X_test, decimals)
    if report["pct"] > max_pct:
        raise AssertionError(
            f"{report['twins']} of {report['test_rows']} test rows "
            f"are in train ({report['pct']}%): {report['indices'][:5]}"
        )

assert_no_copies(X_train, X_test, max_pct=40.0)  # passes
assert_no_copies(X_train, X_test)                # fails

AssertionError: 67 of 180 test rows are in train (37.22%): [6, 8, 9, 14, 15]

max_pct is there for data like ElectricDevices, where you have read the copies, decided they are fine and want to catch only an increase.

What this does not catch

• Anything inexact. Shifted windows, augmented rows and resampled versions all hash differently. Every count here is a lower bound.
• Shared sources. Two recordings of one patient are not copies but they still leak. For that you need a subject or source column to group on, and a hash cannot replace it.
• Whether a ranking is wrong. More copies do not automatically mean a broken leaderboard, and in the archives the ranking barely moved.

On that last point, among the 15 classifiers I could rescore, 0 of 105 pairwise outcomes changed on twin-free fold 0. That is a weak test, since only 16 datasets have twin test rows, and the newest classifiers publish no per-row predictions, so I could not rescore them. I would still trust the broad order of the field. I would trust one score on one of these datasets a lot less. The paper is a preprint and has not been peer reviewed. I could not find an earlier report of these copies, though that does not prove there is none, and the claim is about what the scores measure, not about the people who built the archives.

If you only take one thing from this, take fingerprint.py and run it on whatever you are benchmarking next. The row indices for both archives and the audit script are in the repo.