The non-monotone ranking is the real finding, and it breaks how everyone reads a leaderboard.

If rank order flips with the data regime, then "A beats B" is not a fact about A and B. It is a fact about them at N samples. One number per method quietly asserts a monotonicity your experiments say is not there. The report is a coordinate, not an ordering.

I hit the same wall from the harness side. Agent-harness evolution stops beating plain test-time scaling once you match the compute budget, because search and eval share the benchmark, so the eval ends up measuring itself. Different mechanism, same trap: the comparison inherits the regime it was run in.

One question. Does the crossover show up inside a single seed's learning curve, or only across the asymptotic fits? If one run's rank order already crosses, early-stopping picks the wrong algorithm every time, which is a louder bug than the asymptotic one.