TL;DR
- A big gap between two editors vanishes once compute is matched, and the “weaker” one uses 33% fewer network calls.
- With a fixed budget, averaging beats more timesteps, and it cuts seed-to-seed variance by ~60%.
- CLIP-T barely agrees with humans ($\rho = 0.31$). Our best-scoring configuration barely edits at all.
- One of our own ideas failed its pre-registered check.
The problem
Training-free editors on rectified flow (FlowEdit, FlowAlign, RF-Solver, …) are ranked with tables of SD, CLIP-T and CLIP-I.
But every row hides three choices:
- Compute. Methods spend 2, 3 or $2 \cdot n_{\text{avg}}$ network calls per step.
- Seed. One lucky seed stands in for the whole method.
- Operating point. Each paper picks one favourable point on its own trade-off curve.
The seed alone changes a lot:
Same image, same prompt, four seeds. Bottom: how much outputs vary across seeds, per method (lower = more stable).
Our fix
Pin all three knobs down:
- Count forward calls, not steps.
- Use many seeds, and report variance next to edit strength.
- Show the whole curve, not one cell.
- Same statistics for everyone, including us.
Typical practice vs. this protocol.
Setup: FLUX.1-dev, PIE-Bench, plus a small check on SD3. “Prox” below is an editor we wrote ourselves, and we audit it under the same rules (some figures call it JKOEdit).
Finding 1: the gap disappears
At default settings, Prox looks much worse than FlowAlign. Match the averaging depth ($n_{\text{avg}} = 4$) and:
| SD ↓ | CLIP-T ↑ | CLIP-I ↑ | calls/step | |
|---|---|---|---|---|
| Prox | 0.145 | 0.256 | 0.952 | 8 |
| FlowAlign | 0.140 | 0.258 | 0.946 | 12 |
| p-value | 0.46 | 0.42 | 0.16 |
No significant difference, at one third less compute.
It holds across 10 seeds ($N = 500$) and along the whole curve, not just this point.
Finding 2: average, don’t add steps
Same budget, two ways to spend it:
- 4× more timesteps → SD changes by 0.001
- 4× more averaging → SD improves by 0.05
More averaging keeps helping until about n_avg = 6, then flattens.
Averaging also makes results repeatable: seed variance drops 59% (Prox) and 62% (FlowAlign).
But it isn’t free. It costs up to 5× the compute, and the editor starts editing a bit less.
Finding 3: the metrics can be fooled
We asked three people to blindly judge 50 edits.
| Metric | Agreement with humans (ρ) |
|---|---|
| CLIP-T | 0.31 |
| EditScore prompt-following | 0.81 |
Then look at our own best configuration, Hybrid:
| Human success | SD ↓ | CLIP-I ↑ | |
|---|---|---|---|
| FlowEdit | 0.83 | 0.386 | 0.835 |
| FlowAlign | 0.25 | 0.118 | 0.936 |
| Hybrid | 0.04 | 0.046 | 0.980 |
Best preservation numbers in the study, yet it almost never makes the edit. A preservation-only leaderboard rewards doing nothing.
Two smaller results
Two mechanisms, not one. FlowAlign’s terminal correction and Prox’s proximal term can’t be rescaled into each other (residual 0.84 vs. threshold 0.30, replicated on new seeds).
The residual stays far above the 0.30 threshold at every step.
Our own idea failed. An “average early, not late” schedule looked good at small scale. The pre-registered confirmation said it was worse ($p = 0.047$). We report it anyway.
The edits do look fine
Hand-picked examples. All three methods give usable edits at matched budget.
A checklist for editing papers
- Report forward calls, not step count.
- Report how many seeds.
- Report a preservation and an edit-strength metric. Never CLIP-T alone.
- Say which operating point you used and why.
Scope: one main backbone (FLUX), a small human study (3 raters, 50 edits), and Prox is our own method, audited rather than proposed.