The test set is part of the question

Before looking at a molecular model's score, ask what the test required it to do. Was it recognising close relatives of examples it had already seen, or dealing with chemistry farther away? DataSAIL, introduced in 2025, turns that question into a data-splitting problem rather than leaving it hidden behind a random seed.¹

Draw a meaningful boundary

The package groups data using similarity and solves a constrained allocation problem to reduce information shared across training and test partitions.¹ It supports both individual entities and paired data such as molecules with protein targets.¹ That second case matters because novelty can lie on either side of a pair.

Imagine assessing predictions for an unfamiliar molecule against a familiar protein, then assessing predictions when both are unfamiliar. We would describe these as separate editorial questions. A single score should not be allowed to answer both without an explanation of how the evaluation was constructed. The split is part of the experiment that produces the score.

Harder is not automatically more realistic

The authors explicitly caution that difficult similarity-separated tests can be too pessimistic if a model's intended use involves data close to its training examples.¹ The appropriate similarity definition depends on the deployment question.¹ That is a useful qualification to a paper whose title might otherwise sound like a promise to remove every form of leakage.

Our interpretation is that an evaluator should write down the intended next prediction first. Only then should they decide which relationships must be separated. We would also want to know which information stayed shared: a data split can be strict along one dimension while leaving another familiar. The reason for each choice should be accessible to someone outside the modelling team.

Read the later comparison

A February 2026 addendum compares DataSAIL with splits supplied by dataset creators.² Its PINDER comparison finds lower leakage for the supplied PINDER split than for the tested DataSAIL split.² The update also distinguishes reducing total leakage from constraining the largest individual similarity across partitions.²

Those distinctions prevent a simple winner narrative. We would ask which objective an evaluation needs, not assume that one scalar definition of leakage captures every useful separation. Including the addendum changes the story from a tool announcement into a more informative discussion of benchmark design.

Preserve the evaluation recipe

For a molecular prediction study, our preferred reporting package would contain the actual split membership, the similarity definition, the reason for that definition and the result under relevant alternative splits. This would allow another team to investigate whether an apparent model improvement survives a changed evaluation question.

We would keep any excluded records visible in that package too. The original paper notes that splitting paired data can lose examples when their two entities are assigned to different partitions.¹ A cleaner test boundary can therefore have a data cost that deserves disclosure.

AiChemEx has not rerun the package or compared fresh model scores. The conclusion we draw is procedural: an impressive number becomes scientifically interpretable only when readers can see the test that produced it, including its intended use and its limitations.