Grading a disparity map (and why 98% wrong is sometimes fine)

The results section of my own report has a row that reads 98.7% wrong. Right next to it, on the scene the report spent the most words on, is a row that reads 23.5% wrong: for the same cost function, the same code, the same threshold. Read as “percentage of the image the algorithm got wrong”, the first number says the matcher is useless and the second says it’s pretty good. Neither reading is right, because that is not what the number measures. The last post built the matcher: SAD, SSD and ZNCC, swept along the scanline, threshold, argmin. This one is about the other half: the metric that turns a disparity map into a single percentage, three ways that percentage can be made to say almost anything, and a live widget where you get to watch it happen.

What bad-N actually measures

Section 4.3 of the report (docs/pngs/report-14.png) recommends the Middlebury benchmark’s own bad1/bad2/bad4 measures: the percentage of pixels whose disparity is more than 1, 2 or 4 pixels from the ground truth. The code generalises this to four tolerances, ERR1 through ERR5, and the metric behind all of them is the same one:

$D$ is the computed disparity, $D_{\text{gt}}$ the ground truth, and $\Omega$ the set of pixels the metric actually counts. That last part is doing more work than it looks. Every number in the report depends on three choices baked into $\Omega$ and into how $D$ is written before the sum ever runs, and all three are choices, not facts about the algorithm:

1. Which pixels are in $\Omega$ at all: how wide a border gets excluded.
2. What a rejected match counts as: a pixel the threshold turned down still enters the sum with a specific numeric value.
3. What “1 pixel” means in the ground-truth file: the disparity scale that turns its bytes into pixels.

Get any one of those wrong, silently or on purpose, and $\mathrm{ERR}_N$ moves by tens of percentage points on an unchanged disparity map. The rest of this post is those three things, in the code, and then a widget that lets you turn each one into a slider and watch the same map score anywhere from 7% wrong to 99% wrong.

The tables, transcribed and made sortable

Rather than crop Tables 3 and 4 as two page images, here they are as one table: every row from both, a delta column so SAD and ZNCC sit next to each other, and a click on any header to sort by it.

Sorted by ERR1 descending, the shape of the whole table jumps out: barn1, poster, bull, sawtooth and venus all sit at 95–99% on image 2 (the near baseline) and teddy, map, cones and tsukuba sit at 20–40%. That split has nothing to do with SAD versus ZNCC. Sort by the Δ column and almost every row is within two or three points, exactly the report’s own §4.5 conclusion (below). It has everything to do with how much of each scene actually has ground truth disparity above the tolerance being asked about, which depends on the scene’s texture and its baseline, not on which cost function found the match.

Trap one: the border, and a crop offset that only looks like a bug

calcSSD and calcZNCC cannot report a disparity within $W$ pixels of the image edge (the window would run off the image), so their output is smaller than the input by $2W$ pixels on every side (Modules/SSD2.py:8-9: the valid rows and columns are W + arange(shape - 2W) ). With the code’s default $W=4$, a $384\times288$ pair produces a $376\times280$ disparity map, and pixel $(0,0)$ of that map is pixel $(4,4)$ of the original image.

The evaluation then does this, at TestDisp.py:80:

diff = np.abs(disparityMap[30:-30, 30:-30].astype(np.float32) - ground_truth_disparity[34:-34, 34:-34])

Two different crop widths on two arrays that are supposed to line up (30 against 34), and it is not a typo. disparityMap is already shifted by $W=4$ relative to the original image, so its row 30 is the original image’s row 34. Cropping the ground truth (which is full size) by 34 lands on the same physical row as cropping the already-shrunk disparity map by 30. The asymmetry is the correction, not the bug; $34 = 30 + W$.

Post 29’s Rust port sidesteps the whole thing by keeping the output full-size and marking invalid pixels rather than shrinking the array, so the widget below only needs one border slider, not an offset pair, but it is worth seeing the arithmetic that made two numbers necessary in the original.

Trap two: a rejected match still gets a value

SSD2.py:28 and ZNCC2.py:32 are where a match either survives or doesn’t:

disparityMap[:, row] = np.where(SSD[np.arange(SSD.shape[0]), best_i] < thresh, best_i_scaled.flat, 0)

disparityMap[:, row] = np.where(NCC[np.arange(NCC.shape[0]), best_i]>thresh, best_i_scaled.flat, 0)

A pixel that fails the threshold is not left out of the disparity map: it is written as 0. The evaluation code at TestDisp.py:80 then diffs that 0 straight against the ground truth, with no special case for it. Unless the true disparity there happens to be within the tolerance of zero, a rejected pixel is always counted as wrong. There is no toggle for this in the original code; “count unmatched as wrong” is not a choice TestDisp.py makes, it is the only thing TestDisp.py does. The widget below adds the choice back, because a fairer question (“of the pixels the matcher was confident enough to report, how many are right?”) is a completely different number, and the gap between the two questions turns out to be enormous.

Trap three: the scale that turns bytes into pixels

Both Middlebury and this code store disparity as an 8-bit image, which means a value has to be scaled to become a pixel count. From TestDisp.py:43-47:

maxDisparities = [31,31,31,31,31,31,31,31,31,31,31]
maxDisparities2 = [91,31,31,31,31,31,31,31,91,31,31]
dispScales = [8,8,8,8,8,8,8,8,8,8,8]
dispScales2 = [2.8,8,8,8,8,8,8,8,2.8,8,8]

Most scenes get a search range of 31 disparities at a scale of 8, so the stored byte tops out at $31\times8=248$. Cones and Teddy have a genuinely wider disparity range (up to 91) so their scale drops to 2.8, keeping $91\times2.8=254.8$ inside a byte. A disparity map does not know its own scale. Nothing in the PNG says “divide me by 8”. That number lives only in this Python list, matched to the dataset by array index. Load Cones’ ground truth with the wrong scale, or forget that Teddy needs 2.8 instead of the default 8, and every pixel’s ground-truth value is wrong by that ratio before the comparison even starts. The disparity map does not change. The number describing it collapses.

Watching the number move

Click reproduce the report’s row and the widget sets itself to the report’s own evaluation: SSD (the code that really produced the “SAD” column), a 9×9 window, disparity 0–31, threshold 9.2 (the RMS-per-sample form of SSD2.py:7’s constant), a 30 px border, tolerance 1 px, and unmatched pixels counted as wrong. On the bundled crop of the Tsukuba pair it reads 47.2% wrong: the report’s own row for this exact scene (tsukuba, row3col3) is 39.3%. I recomputed rather than reproduced: the bundled image is a re-crop of the report’s own rasterised figure, one generation removed through a PDF render and a webp re-encode, exactly the softening post 29 already flags for its own numbers. A softer edge shifts more pixels past a 1-pixel tolerance, so a higher number here is the expected direction of the gap, not a contradiction of the report’s.

Now move exactly one knob. Flip count unmatched as wrong to exclude them, changing nothing else, and the number drops to 31.7%, which is not a new number I am inventing for this post, it is the exact SSD row of post 29’s own gain table (77.2% matched, 31.7% bad1), computed by a different post from the same pass. Now widen the tolerance to 8 px and the border stays at 30: 6.9%. Nothing about the matcher changed between 47.2% and 6.9%: same window, same disparities, same threshold, same pixels. Only what got counted changed. Push the other way: border down to 0, tolerance down to 0.5, unmatched counted as wrong, and the same disparity map reads 59.8%. That is the whole argument of this post in three clicks.

What the report’s own numbers were actually saying

Section 4.5 (docs/pngs/report-17.png, docs/pngs/report-18.png) reaches a fair conclusion, even if the headline numbers look damning first:

“Both SAD and ZNCC performed relatively well, especially for such simple approaches… Both ZNCC and SAD performed equally well, with SAD performing better in some images and ZNCC performing better in others. This makes sense as the dataset contains little to no noise or distortions. ZNCC would in theory be more resilient to these artefacts but not perform better than SAD in a low noise environment. SAD is far less computationally expensive than ZNCC, creating a trade-off between resiliency and speed.”

The sortable table backs that up directly. Sort by the Δ column and the two cost functions are within a couple of points on almost every row, which is exactly “performed equally well.” What the discussion does not say, and what Tables 3 and 4 alone cannot say, is why barn1 reads 98.7% and teddy reads 23.5% for the identical pipeline. It is not that one scene is harder to match. It is that the metric’s denominator, its zero-handling and its scale are shared across every row in the table, and a scene where those defaults happen to be forgiving reads as a success while a scene where they are not reads as a catastrophe, regardless of what the matcher actually did.

The next post stays in the same two files a little longer, for a different reason: the gather trick that lets SSD2.py run in NumPy at all, why the more elegant version in SSD.py never got past a single test scene, and why the browser version in this chapter had to be Rust rather than a faster NumPy.