A better optimizer needs a trustworthy test

A laboratory can lose an afternoon choosing its next experiment, or several weeks following a poor sequence of choices. Software that selects conditions promises to make that sequence more productive. Before trusting its recommendations, however, a chemist needs to know what a good result in a virtual laboratory actually predicts about a physical one.

RECOB, a September preprint by Zikai Xie, Jiaming Wan and Linjiang Chen, makes that question unusually concrete. It builds optimization tests from recorded chemistry and materials experiments, then asks whether conclusions survive changes to the model supplying the answers. The striking result is a combination of stability and movement: the broad advantage of several adaptive methods persists, while the identity of the leading method depends on how historical measurements become a searchable experiment.¹

That is a useful contribution even without a universal winner. The study gives readers a way to distinguish three claims that are easily collapsed: an optimizer works well against a particular response model; its ranking survives retraining that model; and its advantage also appears when only recorded experimental outcomes are returned. RECOB provides evidence for all three at different levels of specificity. It does not report a new prospective laboratory competition.¹

Building a laboratory that can answer any proposal

The authors began with 19 experimental datasets from 17 source studies, comprising 5,291 records and 43 candidate combinations of target and search space. They reconstructed controllable variables, units, bounds, objective directions and physical constraints. The final single-objective suite contains fourteen tasks from nine independent publications; two multi-objective tasks extend the comparison to competing outcomes.¹

This curation matters because an experiment is more than a vector of numbers. A formulation can require components to sum to a fixed total; a dispensing system can permit only particular increments; a catalyst can be a category rather than a continuous quantity. RECOB stores these distinctions and validates proposed conditions. In the formal comparisons, a common decoding procedure maps each optimizer's proposals into the same feasible domain before the response model is consulted.¹

The shared mapping is a strength of the comparison. An algorithm should not receive a practical advantage simply because its interface silently repairs an impossible composition differently. It also defines the scope of the result: these are particular implementations operating through a common parameterization. The ranking is not a verdict on every possible configuration of the named software.

A historical table cannot answer an arbitrary new combination of temperature, concentration and catalyst. RECOB therefore fits a predictive model, called an oracle, to each admitted task. Its answer is a prediction trained on measurements, rather than a newly measured chemical result. The supplement makes the model choice explicit: the AutoGluon pipeline considers random forests and extra-trees models, then deploys a refitted weighted ensemble. The oracle returns deterministic point predictions without added observation noise.¹

Test the model before testing the optimizer

Before admitting a task, the authors assess predictive performance across five held-out splits. Repeated identical input configurations are kept together for the affected datasets, preventing replicates of the same conditions from appearing on opposite sides of the training–test boundary. Admission combines sample count, predictive fit, correlation and normalized error; it is not based on a single favourable split.¹

Ten tasks pass the stricter Core criteria and four enter a Conditional tier. Among other requirements, Core admission needs mean R-squared of at least 0.50, while Conditional admission permits at least 0.20. The task is screened before the final oracle is refitted on all available labelled rows. Optimization histories do not determine which model is admitted.¹

These are sensible safeguards. A method ranking is difficult to interpret if the virtual reaction barely reproduces the measurements used to construct it. Retaining a separate Conditional tier also makes differences in response-model fidelity visible rather than burying them in an aggregate score.

The selection has a scientific consequence, rather than an automatic defect: the resulting benchmark represents historical systems that these models can reproduce adequately. It does not represent all experimental chemistry. A new campaign with sparse measurements, different mechanisms or stronger noise may fall outside that population. The authors' conclusion appropriately retains prospective laboratory validation as a separate step.¹

This work follows an established line of virtual reaction testing. Felton, Rittig and Lapkin's Summit framework compared optimization strategies using chemically motivated benchmarks, including a kinetic model for nucleophilic aromatic substitution and a data-driven cross-coupling model. It supplied a reusable interface for testing strategies before committing to physical experiments.² RECOB's distinctive emphasis is to interrogate the reliability of rankings through retrained oracles and a separate track returning historical measurements.¹

Equal starts make a meaningful race

For each task, RECOB runs twenty trials. Within a trial, all methods receive the same ten feasible initial conditions in the same order. Each then receives one hundred further sequential evaluations, with one proposal at a time. Internal model fitting and acquisition optimization do not grant extra oracle queries. Ten single-objective algorithms are compared under this paired design.¹

Matching the start is especially valuable when budgets are small. A lucky initial point can make an optimizer appear efficient before it has made a single distinctive decision. Sharing those observations directs the comparison toward what each method does with the same information. The repeated trials then show how much the outcome depends on initialization and subsequent random choices.

The reported single-objective score measures the gap between the best response found so far and the best response in the pooled benchmark histories, scaled by the pooled range. Those endpoints are frozen after the histories are collected and are not given to the optimizers. A score of zero therefore means reaching the best value encountered in this comparison; it is not certification of a globally optimal physical condition.¹

HEBO has the strongest aggregate final rank. BoTorch LogEI performs best on the area under the regret curve, which rewards improvement earlier in the budget. The paper's pairwise tests do not distinguish HEBO decisively from BoFire LogEI or BoTorch LogEI at the reported conventional threshold. The useful result is a leading group with differing trajectories, alongside clear task dependence, rather than an unqualified claim that one package dominates every alternative.¹

Computational overhead makes that distinction practical. Mean suggestion times are approximately 3.50 seconds for HEBO, 16.91 for BoFire LogEI and 14.65 for BoTorch LogEI. The authors' own random-forest expected-improvement implementation takes about 0.17 seconds per step. These timings exclude oracle evaluation and scheduler queueing; the forest method is SMAC-inspired, not the official SMAC software.¹

For a reaction taking hours, several extra seconds of planning may be unimportant. For a rapidly cycling instrument, they may shape throughput. That is an operational inference, not a measured speed-up of a laboratory in this study. Separating suggestion time from evaluation efficiency lets readers make the comparison relevant to their own apparatus.

A stable answer can still depend on the question

The first reliability test retrains the oracles for six representative tasks with three different seeds. The ten-algorithm rankings remain closely aligned: pairwise Spearman correlations range from 0.964 to 1.000. Predictions on shared, response-independent candidates are also highly consistent. The leading methods remain a leading group across those retrainings.¹

This is stronger evidence than reporting one favourable frozen model. It shows that ordinary randomness in fitting that pipeline does not drive the aggregate ordering. But the experiment changes seeds within the same forest-based modeling protocol. It does not establish agreement across fundamentally different response-model families, nor does it reveal the behaviour of an unmeasured physical region. Those are different questions from the one tested.¹

The second test removes predicted outcomes from the returned answers. In measured-table replay, each proposal is matched to the nearest unused historical condition. The optimizer receives that matched condition and its recorded response, and the row leaves the pool. Numerical distances are scaled by declared variable ranges; category mismatches receive a separate distance contribution. Replicates at identical inputs are collapsed by averaging their responses, with replicate information retained.¹

Replay is therefore neither another smooth oracle nor a new experiment at the exact requested point. It asks how the optimizer proceeds when its suggestions must be realized through a finite collection of previously measured conditions. This preserves the connection to recorded chemistry while changing the search problem. Reporting that change explicitly is one of the paper's most useful methodological decisions.

At sixty total evaluations across six tasks, the learned-oracle and replay rankings have Spearman correlation 0.758. BoTorch UCB leads replay; HEBO moves to sixth place. BoFire LogEI and BoTorch LogEI remain in the top three under both protocols. The lower-performing group remains broadly similar. The result supports a persistent separation between groups, but it does not support treating the precise learned-oracle ranking as experimentally confirmed.¹

Read the supplement before explaining the reversal

The supplement supplies a particularly important qualification to a tempting story. At 110 evaluations, the reported rank correlation is lower, at 0.455. It would be easy to attribute the entire reduction to depletion of nearby measured candidates as the budget grows. Yet that comparison contains only two tasks, whereas the sixty-evaluation comparison contains six. Appendix D explicitly warns that the two rows are separate comparisons, not a controlled estimate of the effect of budget alone.¹

Candidate depletion is a plausible mechanism described by the authors: once neighbouring rows have been used, a tightly concentrated proposal sequence can be redirected to increasingly different historical conditions. But the published aggregate numbers change both budget and task set. To isolate the budget effect, one would need to compare matched task sets across budgets. That qualification preserves the authors' mechanistic explanation as an interpretation rather than presenting it as a uniquely demonstrated cause.¹

Matching-distance diagnostics add another useful check. In the sixty-evaluation amide task, 37.5% of post-initialization matches exceed a normalized distance of 0.25, compared with none in several other tasks. The distance averages scaled differences across variables; it is not a chemically calibrated measure of how much reaction behaviour changes. Still, it shows that replay is sometimes answering a noticeably displaced proposal. The discrepancy is part of the evaluation mechanism and should accompany any claim about transferred rankings.¹

These details do not invalidate replay. They explain why it is informative. A response model and a finite measured table offer different freedoms, and an optimizer can exploit those freedoms differently. Agreement between them is encouraging; disagreement identifies which conclusions require a more specific experimental test.

Choosing the next physical comparison

The multi-objective extension reaches a similarly bounded conclusion. qNEHVI leads the aggregate comparison across the amide and printing tasks, while MESMO has the highest final hypervolume on printing. With only two continuous, all-maximization tasks, the authors describe the cross-task ranking as descriptive. Hypervolume is compared within each task, not treated as a universal unit of chemical performance.¹

For a laboratory choosing software, RECOB is most useful as a disciplined screening instrument. It supports testing several credible adaptive methods under a common budget, inspecting their early and final performance, and checking whether conclusions survive a change in evaluation protocol. Its source-derived domains and preserved provenance make the virtual comparison more closely connected to experimental practice.¹

The most persuasive advance is the refusal to let experimental ancestry do all the evidential work. Historical measurements make a benchmark relevant, but the transformation from table to response model still needs scrutiny. RECOB supplies concrete tests of that transformation and reports where exact rankings move. That leaves chemists with a better justified shortlist for the next physical campaign, and a clearer account of what that campaign still has to decide.