Reprojecting a 3D semantic map back into the camera

Post 26 in this series scored SemanticFusion against NYUv2 with assets/NYUv2/seg2d_all_c.tex’s three rows: 37.2% classification accuracy, 21.5% “semantic segmentation”, and a lone number that looks like a typo: 1.4% instance-segmentation accuracy. It isn’t a typo, and it isn’t the metric being unfair. Post 26 already showed accuracy, mIoU and FIoU are all well-defined and all doing their job. It’s what happens upstream of the metric, in the step that gets no attention because it produces no headline number of its own: the algorithm outputs a labelled point cloud, ground truth lives in image space, and something has to reconcile the two before any of Post 26’s functions can run at all. This post is that something: segtester/dataloaders/scannet.py::get_labelled_reproj_seg3d and segtester/assessments/segmentation2d_reproj.py, and the two frames, est_000272.png and est_000623.png, that make the reprojection’s honesty visible without reading a line of code.

Two spaces, one ground truth

SemanticFusion’s output is a point cloud: every surfel has a position and a predicted class, built up over a whole video sequence. NYUv2’s ground truth is a set of individual frames, each with a dense per-pixel label image. To compare them you need one space. The platform’s choice is to go image-space: take each ground-truth frame’s camera pose and depth, push every one of its pixels out into the same 3D world the point cloud lives in, find the nearest surfel to each of those pushed-out points, and carry the surfel’s label back to the pixel it came from. Four steps, in order: unproject, gate by depth, match, scatter back.

Unprojecting a pixel into the room

segtester/dataloaders/scannet.py:112-166 is the whole thing, quoted in full because every line earns its place:

# segtester/dataloaders/scannet.py:112-166
def get_labelled_reproj_seg3d(self, depth_min=0.001, depth_max=50.0, predicted_pose=None):
    from zipfile import ZipFile
    import imageio

    d_image_dims = self.get_depth_size()
    rgb_img_dim = self.get_rgb_size()
    rgb_intrinsics = self.get_intrinsic_rgb()

    d_scale = self.get_depth_scale()
    inv_intr = np.linalg.inv(rgb_intrinsics)
    v, u, const_1, const_2 = np.meshgrid(
        np.arange(rgb_img_dim[1]),
        np.arange(rgb_img_dim[0]),
        np.array([1]),
        np.array([1]))
    inds = np.stack((v.ravel(), u.ravel(), const_1.ravel(), const_2.ravel()))
    rw_points_cam = inv_intr @ inds
    inds_depth = inds[:2]*[[d_image_dims[1]-1], [d_image_dims[0]-1]]/[[rgb_img_dim[1]-1], [rgb_img_dim[0]-1]]
    inds_depth = np.round(inds_depth).astype(np.int)
    with ZipFile(self.projected_label_file, 'r') as labeled_arch, ZipFile(self.projected_instance_archive, 'r') as instance_arch:
        n_list_lbl = labeled_arch.namelist()
        n_list_inst = instance_arch.namelist()

        for depth_image, camera_to_world, i in self.get_depth_position_it():
            lbl_filt_name, lbl_seg_name = f'label-filt/{i}.png', f'instance-filt/{i}.png'
            if lbl_filt_name not in n_list_lbl or lbl_seg_name not in n_list_inst:
                continue
            lbl_img = imageio.imread(labeled_arch.open(lbl_filt_name))
            lbl_seg_img = imageio.imread(instance_arch.open(lbl_seg_name))

            scaled_d_img = depth_image[inds_depth[1], inds_depth[0]]/d_scale
            d_img_mask = np.logical_and(scaled_d_img >= depth_min, scaled_d_img <= depth_max)
            proj_points_cam = rw_points_cam[:, d_img_mask]*scaled_d_img[d_img_mask]
            proj_points_cam[3] = 1.0
            proj_points_world = camera_to_world@proj_points_cam
            pt_lbls = lbl_img.flat[d_img_mask]
            pt_seg = lbl_seg_img.flat[d_img_mask]
            stacked_label_inst = np.stack([pt_lbls, pt_seg])
            unique_segs = np.unique(stacked_label_inst, axis=1)
            instance_masks = np.all(np.equal(stacked_label_inst[:, None], unique_segs[:, :, None]), axis=0)
            s3d = Seg3D(
                proj_points_world.T[:, :3], pt_lbls, instance_masks, unique_segs[0], np.ones(len(pt_lbls))
            )
            pt_lbls = lbl_img.flat
            pt_seg = lbl_seg_img.flat
            stacked_label_inst = np.stack([pt_lbls, pt_seg])
            unique_segs = np.unique(stacked_label_inst, axis=1)
            instance_masks = np.all(np.equal(stacked_label_inst[:, None], unique_segs[:, :, None]), axis=0)
            s2d = Seg2D(
                lbl_img, instance_masks, unique_segs[0], np.ones(len(pt_lbls))
            )
            yield s3d, s2d, inds[[1, 0]][:, d_img_mask], i

The first eleven lines build one thing per scene, not per frame: inv_intr, the inverse RGB intrinsics, and inds, every RGB pixel written as a homogeneous 4-vector $(u, v, 1, 1)$. rw_points_cam = inv_intr @ inds turns every pixel into a camera-space ray, direction only, not yet scaled by how far along it the surface actually is. In maths, for a pixel $(u, v)$ with RGB intrinsics $K$:

$$\mathbf{r} = K^{-1}\begin{pmatrix}u\\v\\1\end{pmatrix}$$

That’s the easy part. The line above it is the one worth pausing on:

inds_depth = inds[:2]*[[d_image_dims[1]-1], [d_image_dims[0]-1]]/[[rgb_img_dim[1]-1], [rgb_img_dim[0]-1]]
inds_depth = np.round(inds_depth).astype(np.int)

ScanNet’s depth and colour streams aren’t the same resolution, so “the depth at this RGB pixel” first needs a resolution remap: scale the RGB pixel coordinates into depth-image space and round to the nearest depth pixel:

$$\begin{pmatrix}u_d\\v_d\end{pmatrix} = \mathrm{round}\left(\begin{pmatrix}u\\v\end{pmatrix}\cdot\begin{pmatrix}\dfrac{W_d-1}{W_{rgb}-1}\\[0.4em]\dfrac{H_d-1}{H_{rgb}-1}\end{pmatrix}\right)$$

Every pixel gets unprojected at the RGB camera’s intrinsics and resolution (the depth image only ever supplies one scalar, how far along that ray to go), which is why the label ends up living at RGB resolution too.

Depth gating, and the index bookkeeping that makes the round trip possible

scaled_d_img = depth_image[inds_depth[1], inds_depth[0]]/d_scale
d_img_mask = np.logical_and(scaled_d_img >= depth_min, scaled_d_img <= depth_max)
proj_points_cam = rw_points_cam[:, d_img_mask]*scaled_d_img[d_img_mask]
proj_points_cam[3] = 1.0
proj_points_world = camera_to_world@proj_points_cam

d_img_mask drops any pixel whose depth reading is outside [depth_min, depth_max], the sensor’s blind spots and its far-plane noise, by default anything under 1 mm or over 50 m. proj_points_cam then only ever holds the surviving rays, scaled by their depth and re-homogenised, before camera_to_world places them in the same frame the point cloud lives in:

$$\mathbf{p}_{world} = T_{cam \to world}\begin{pmatrix}d\cdot\mathbf{r}\\1\end{pmatrix}$$

The bookkeeping detail is the function’s last line: yield s3d, s2d, inds[[1, 0]][:, d_img_mask], i. inds[[1, 0]] is the pixel coordinates (row, column) of every ray, and [:, d_img_mask] filters them by the exact same boolean mask that filtered the 3D points. The pixel indices and the 3D points that came from them stay in lockstep through the whole function. Nothing downstream has to remember which world point belonged to which pixel: the caller just gets both arrays, still aligned, and that alignment is the only thing that makes “scatter the matched label back into the image” possible later. It would have been easy to drop that array from the return value, since nothing about the matching step that follows needs it; keeping it is the whole reason a 2D reprojected score exists at all.

Matching against what the algorithm actually built

segtester/assessments/segmentation2d_reproj.py is the driver. For each ground-truth frame it gets back reproj_gt_seg (the unprojected, depth-gated GT points, in world space) and the pixel indices from above, then matches every one of those points to the estimated cloud with a KD-tree:

# segtester/assessments/segmentation2d_reproj.py:90-104
if self.conf.use_reprojection:
    mapped_est_seg, dists = seg_3d_est.get_mapped_seg(reproj_gt_seg)
    dist_mask = dists.flatten() < self.conf.point_dist_thresh
    seg_inds = seg_inds[:, dist_mask]
    mapped_labels = np.zeros(s2d_gt.image_shape, dtype=np.int)
    mapped_labels[seg_inds[0], seg_inds[1]] = mapped_est_seg.classes[dist_mask]
    mapped_segs = np.zeros((mapped_est_seg.instance_masks.shape[0],) + s2d_gt.image_shape,
                           dtype=np.bool)
    mapped_segs[:, seg_inds[0], seg_inds[1]] = mapped_est_seg.instance_masks[:, dist_mask]
    seg_mask = np.any(mapped_segs, axis=(1, 2))

    s2d_est = Seg2D(
        mapped_labels, mapped_segs[seg_mask], mapped_est_seg.instance_classes[seg_mask],
        np.ones(mapped_labels.size)
    )

get_mapped_seg (segtester/types/seg3d.py:121-132) builds a KD-tree over the estimated cloud and, for every reprojected ground-truth point, finds its nearest estimated neighbour and copies that neighbour’s class across. dist_mask = dists.flatten() < self.conf.point_dist_thresh is the second gate this pipeline applies: nearest is not good enough on its own; nearest has to be close, closer than point_dist_thresh (0.05 m by default, segtester/configs/assessments/seg2d_reproj.py:19). A ground-truth point with no estimated surfel within 5 cm of it is excluded from scoring entirely, which is the correct treatment of “the algorithm never reconstructed that part of the room”: it isn’t a miss to be punished as a wrong label, it’s a hole the algorithm never filled, and punishing it as a wrong prediction would conflate two different failures.

Scattering back into the image

The three lines that matter are the ones with seg_inds on the left of the equals sign:

mapped_labels = np.zeros(s2d_gt.image_shape, dtype=np.int)
mapped_labels[seg_inds[0], seg_inds[1]] = mapped_est_seg.classes[dist_mask]

mapped_labels starts as an all-zeros (background) image, the same shape as the ground truth. seg_inds, filtered by the exact same dist_mask that filtered mapped_est_seg.classes, is what lets mapped_labels[seg_inds[0], seg_inds[1]] = ... write each matched class back to the exact pixel it was unprojected from, the same index bookkeeping from get_labelled_reproj_seg3d two functions ago, carried through a KD-tree query and a distance gate without ever being recomputed. Every pixel that either failed the depth gate on the way out, or found no close-enough estimated point on the way back, is left at 0. That zero is not “the algorithm predicted background”, it’s “nothing survived to write a value here at all”, and it’s the entire story of the next section.

The two frames that make the point

I counted pixels rather than eyeballing it. In est_000272.png, 302,088 of 307,200 pixels (98.3%) are the fixed background grey (76, 76, 76), only 1.7% of the frame got a label at all. est_000623.png is the opposite: 151,613 of 307,200 (49.4%) are background, so roughly half the frame is covered, and what’s there is visibly noisier: more distinct small regions, more scattered single-pixel disagreement with the ground truth next to it. Neither ground-truth frame is anywhere near that empty (gt_000272 is 26.0% unlabelled, gt_000623 is 18.0%), this is not NYUv2 having a bad day, it’s specific to what the estimated cloud had to offer at those two camera positions.

The control: reading the CNN without any of this

Everything above answers “did the fusion lose information the CNN had.” There’s a second question hiding behind the first: is the CNN actually good, independent of whether the 3D fusion step preserved its answers? The same assessment class answers it with one config flag:

# segtester/assessments/segmentation2d_reproj.py:105-109
else:
    s2d_est = scene.get_seg_2d_from_labeled_frame_id(img_nr, s2d_gt.image_shape)
    est_label_map = self.label_map.get_label_map(scene.label_map_id_col,
                                                 self.conf.label_map_dest_col)
    s2d_est.map_own_classes(est_label_map)

use_reprojection: False skips the entire unproject-match-scatter pipeline above and reads a per-frame .npz of raw per-pixel CNN likelihoods instead:

# segtester/dataloaders/results.py:30-42
def get_seg_2d_from_labeled_frame_id(self, frame_id, output_shape):
    lk = np.load(f"{self.base_path}/frames/frame_{frame_id}.npz")["likelihoods"]
    classes = lk.argmax(axis=2).T
    probs = lk.max(axis=2).T
    v, u = np.meshgrid(np.arange(output_shape[1]), np.arange(output_shape[0]))
    inds = np.stack((u, v))
    inds = np.round(inds * [[[classes.shape[0] - 1]], [[classes.shape[1] - 1]]] /
                    [[[output_shape[0] - 1]], [[output_shape[1] - 1]]]).astype(np.int)
    classes = classes[inds[0], inds[1]]
    unique_classes = np.unique(classes)
    unique_classes = unique_classes[unique_classes!=0]
    instance_masks = classes[None] == unique_classes[:, None, None]
    return Seg2D(classes, instance_masks, unique_classes, probs)

argmax over the CNN’s per-class likelihood volume, at whatever resolution the network ran at, resized (nearest-index, the same remap-and-round idiom as the unprojection code) to the ground-truth image’s shape. No point cloud, no depth image, no camera pose, just “what did the network say about this exact frame.” Running the same assessment with this flag flipped and comparing the two accuracy numbers is the only clean way to separate “the CNN is bad at this class” from “the 3D fusion step lost information the CNN already had”, and it’s a genuinely useful pattern outside this one repo: whenever a system’s output passes through a lossy intermediate representation before being scored, keep a code path that scores the representation before the lossy step, or you can never tell which stage to go fix.

Frame scrubber, with the score computed live

The widget below decodes all 34 gt_XXXXXX.png/est_XXXXXX.png pairs client-side, reusing ../_shared/label-palette exactly as built and verified for Post 26, because two renders of the same class id can legitimately use different colours if their max_class differs, so only a decode-then-compare is correct, never a raw pixel-colour diff, and recomputes point accuracy, mIoU and FIoU from the decoded labels on every frame change, not from a table. That’s the “Classification” row’s definition specifically (get_results(s2d_est.classes, s2d_gt.classes, all_class_ids), segmentation2d_reproj.py:112-113) a direct per-pixel class comparison, not the instance-matched rows above, which need Post 27’s greedy matcher and aren’t reproduced here. Drag the divider (or the slider under the image) to wipe between ground truth and the estimate; toggle “show only disagreements” to collapse agreement to a flat grey and show only what the reprojection got wrong; scrub from frame 272 to frame 623 and watch the coverage line (the fraction of pixels the estimate actually labels) and the accuracy line move together.

With JavaScript enabled, this becomes a scrubber over all 34 real NYUv2 frame pairs with a draggable wipe comparison, a disagreement-only view, and accuracy / mIoU / FIoU recomputed from the actual pixels as you scrub, alongside a coverage chart across every frame.

Optional: the unprojection itself, orbitable

The scrubber above starts from images that are already reprojected. This second widget goes one step earlier and shows the unprojection alone: one 64×48 depth thumbnail (nyuv2_sample_93_depth.png, downsampled) turned into a point cloud with the pinhole maths from the top of this post, orbitable by drag, with a depth_max slider that visibly peels the far wall away as it drops.

With JavaScript enabled this becomes an orbitable point cloud with yaw, pitch and depth_max controls.

What I’d tell past me

The reprojection isn’t where the numbers go wrong. The reprojection is a faithful report of how much of the room the fusion step actually reconstructed, and index bookkeeping (seg_inds, carried unmodified through a depth gate, a KD-tree query and a distance threshold) is what makes that report possible instead of just a plausible-looking average. est_000272.png isn’t a bug in the metric. It’s SemanticFusion, at that exact camera pose, having built almost nothing there yet, and the scatter-back is honest enough to leave those pixels at zero rather than guessing. A 1.4% instance-accuracy number that came from anywhere else in this pipeline would be worth doubting. This one, once you trace where it comes from, is exactly as low as it should be.