APE, RPE, and why "how wrong is this trajectory" has five answers

Every algorithm in the evaluation platform that produces a camera trajectory (really just SemanticFusion, wired to ElasticFusion for pose) needs a single question answered: how wrong is the path it recovered? I built the odometry assessment expecting that to be one number. It is at least five: two different failure modes (drift versus jitter), three different norms to measure each one in, and an alignment step applied before any of it that can move the answer by an order of magnitude without touching a single pose.

Two questions, not one

segtester/metrics/odo_ape_rpe.py wraps two calls into evo, Michael Grupp’s trajectory-evaluation library, ape() and rpe(). They ask different questions about the same pair of trajectories:

• APE, absolute pose error: where did the camera end up, compared to where it should have? Punishes drift, because drift accumulates and every pose after the drift starts carries it.
• RPE, relative pose error: was each individual step the right size and direction? Punishes local jitter, and is blind to a slow accumulating bias, because it only ever compares a pose to the one before it.

Here’s ape() in full (segtester/metrics/odo_ape_rpe.py:10–56), because one detail in it is worth reading past the alignment plumbing for:

def ape(traj_ref, traj_est, pose_relation, align=False, correct_scale=False,
        align_origin=False, ref_name="reference", est_name="estimate"):

    # Align the trajectories.
    only_scale = correct_scale and not align
    if align or correct_scale:
        logger.debug(SEP)
        traj_est = trajectory.align_trajectory(traj_est, traj_ref,
                                               correct_scale, only_scale)
    elif align_origin:
        logger.debug(SEP)
        traj_est = trajectory.align_trajectory_origin(traj_est, traj_ref)

    # Calculate APE.
    logger.debug(SEP)
    data = (traj_ref, traj_est)
    ape_metric = metrics.APE(pose_relation)
    ape_metric.process_data(data)

    title = str(ape_metric)
    if align and not correct_scale:
        title += "\n(with SE(3) Umeyama alignment)"
    elif align and correct_scale:
        title += "\n(with Sim(3) Umeyama alignment)"
    elif only_scale:
        title += "\n(scale corrected)"
    elif align_origin:
        title += "\n(with origin alignment)"
    else:
        title += "\n(not aligned)"

    ape_result = ape_metric.get_result(ref_name, est_name)
    ape_result.info["title"] = title
    ...

rpe() is the same function shape with a delta/delta_unit pair added and the same if/elif chain repeated verbatim (odo_ape_rpe.py:59–116). That repeated chain is the nice detail: the plot title is the provenance record. Nobody looking at a saved ape.pdf months later has to go spelunking through a config file to find out whether the number in front of them used Sim(3) alignment: it’s printed on the plot, in words, because the code that computed the number is the same code that wrote the label. Compare that to the alternative I’ve seen in other benchmarks: a table of numbers in a paper with “aligned” in a footnote and no record of how.

Three ways to measure error on SE(3)

A pose is a rigid transform, not a point, so “the error” needs a pose relation before it means anything. OdometryConfig (segtester/configs/assessments/odometry.py:17–20) defaults to all three evo defines:

self.pose_relations: List[PoseRelation] = BCNF.OptionalMember(default_ret=[
    PoseRelation.full_transformation,
    PoseRelation.translation_part,
    PoseRelation.rotation_part])

Given the relative error pose $E = P_{\text{ref}}^{-1} P_{\text{est}}$ (rotation $R$, translation $t$), the three read it three different ways:

• translation_part: how far off in metres: $\lVert t \rVert$.
• rotation_part: how far off in orientation. This widget reports the angle in degrees, $\theta = \arccos\!\big(\tfrac{\operatorname{tr}(R)-1}{2}\big)$, because it’s the quantity a reader can sanity-check against the drift slider by eye. evo may define this as a matrix norm $\lVert R - I\rVert_F$ instead. I don’t have its source to check, so consider this my reading of the name rather than a reproduction.
• full_transformation: both at once, via the SE(3) logarithm (below). This is the one worth a caveat: it stacks translation (metres) and rotation (radians) into a single Euclidean norm, which means it’s mixing units in one number. That’s not a bug in this widget or in evo. It’s exactly why translation_part and rotation_part exist as separate options, so you’re never stuck reading a number that’s secretly half one thing and half another.

Five ways to align first: and the one that gets benchmarks gamed

Before any of that error is computed, the estimate is optionally registered onto the reference. segtester/assessments/odometry.py:11–17 enumerates every option the platform supports as one dict, and the run loop just below it (:57–58) tries every combination:

ALIGNMENT_OPTIONS = {
    "noalign": (False, False, False),
    "SE3_Umeyama_alignment": (True, False, False),
    "Sim3_Umeyama_alignment": (True, True, False),
    "scale_corrected": (False, True, False),
    "origin_alignment": (False, False, True),
}

for alignment_opt in self.conf.alignment_options:
    for pose_relation in self.conf.pose_relations:
        alignment_opt_tuple = ALIGNMENT_OPTIONS[alignment_opt]

Each entry is (align, correct_scale, align_origin), fed straight into the if align or correct_scale / elif align_origin branches quoted above. scale_corrected is the one worth slowing down for: only_scale = correct_scale and not align, so it runs the same alignment machinery as Sim3_Umeyama_alignment but keeps only the scale it finds, not the rotation or translation.

The argument that matters: a monocular SLAM system has no absolute scale, it can only ever recover a trajectory up to an unknown similarity transform, so Sim(3) alignment isn’t cheating, it’s the honest correction for a well-known ambiguity. An RGB-D system like the ElasticFusion this platform evaluates has metric depth built in and needs no scale correction at all. So reporting “with Sim(3) Umeyama alignment” without saying so is how benchmarks get gamed: apply it to a monocular result and you’ve fixed a real ambiguity; apply it to an RGB-D result and you’ve quietly cancelled out a real failure mode (a wrong scale is an error for a system that had metric depth to get it right).

Umeyama, worked

Sim3_Umeyama_alignment and SE3_Umeyama_alignment both call the same closed-form least-squares fit (Umeyama, 1991) between the estimate’s positions $\{x_i\}$ and the reference’s $\{y_i\}$, differing only in whether the recovered scale $c$ is applied or held at 1:

$$\bar x = \frac1n\sum_i x_i, \qquad \bar y = \frac1n\sum_i y_i, \qquad \Sigma_{xy} = \frac1n\sum_i (y_i-\bar y)(x_i-\bar x)^{\mathsf T}$$ $$\Sigma_{xy} = U D V^{\mathsf T} \ \text{(SVD)}, \qquad S = \operatorname{diag}\!\big(1,\,1,\,\operatorname{sign}(\det U \det V)\big), \qquad R = U S V^{\mathsf T}$$ $$c = \frac{\operatorname{tr}(DS)}{\sigma_x^2}, \quad \sigma_x^2 = \frac1n\sum_i \lVert x_i - \bar x\rVert^2 \qquad\qquad t = \bar y - cR\bar x$$

origin_alignment skips all of this and matches the first pose exactly: $T = P_{\text{ref},0}\,P_{\text{est},0}^{-1}$, applied rigidly to every pose. No least squares, no scale, it just asks “if I nail down where the trajectory started, how far does it wander from the truth from there.”

This is a $3\times3$ SVD, which is exactly what wasm/crates/calib-wasm/src/linalg.rs already has: post 41 wrote a hand-rolled one-sided Jacobi SVD for Zhang’s method, and it’s already loaded by two other posts on this site. I added traj.rs to the same crate rather than write a second SVD. One wrinkle: linalg::svd returns only the singular values and $V$, not $U$ (it’s a one-sided method, it only ever orthogonalises columns of $A$, never forms $U$ explicitly). $U$ is recoverable exactly as $\Sigma_{xy} V$, normalised per column, except this widget’s ground truth path is planar-ish (see below), which makes the smallest singular value of $\Sigma_{xy}$ come out at or near zero, and dividing by a near-zero singular value is a NaN waiting to happen. The fix is to build the third column of $U$ as $u_0 \times u_1$ instead of dividing it out, always a valid right-handed completion, and Umeyama’s own reflection fix (the $\operatorname{sign}(\det U \det V)$ term) still corrects the handedness regardless of how that column was constructed.

The log-map, and why RPE only ever asks about one frame

full_transformation needs the SE(3) logarithm: the 6-vector $\xi=(\rho,\omega)$ such that composing infinitesimal motion $\xi$ recovers the pose. The rotation half is ordinary Rodrigues:

$$\theta=\arccos\!\left(\frac{\operatorname{tr}(R)-1}{2}\right), \qquad \omega=\frac{\theta}{2\sin\theta}\begin{bmatrix}R_{32}-R_{23}\\R_{13}-R_{31}\\R_{21}-R_{12}\end{bmatrix}$$

The translation half is not just $t$: it’s $t$ pulled back through the inverse of SE(3)‘s left Jacobian, $\rho = V^{-1}t$, because composing rotation and translation isn’t commutative:

$$V^{-1} = I - \tfrac12[\omega]_\times + \left(\frac1{\theta^2} - \frac{1+\cos\theta}{2\theta\sin\theta}\right)[\omega]_\times^2$$

(that coefficient’s small-angle limit is $1/12$, wasm/crates/calib-wasm/src/traj.rs’s test suite checks it numerically rather than trusting the algebra). This is the “per-pose SE(3) composition and log-map over a few thousand poses” the crate exists for: every pose needs composing against the alignment transform, then every pair of poses needs a relative transform and a log-map, on every slider drag.

For RPE, that pair is always adjacent poses: exe_rpe_tests calls rpe(..., delta=1, delta_unit=Unit.frames, ...) (odometry.py:122–124), “for delta = 1 (frames) using consecutive pairs,” as the plot title puts it. Never a longer baseline. That’s a deliberate choice, not a limitation: a longer delta would start re-mixing in some of the drift APE already measures, and the whole point of having both metrics is to keep the two failure modes cleanly separated.

The plots, and one of them is wrong

The two 8-page PDF collections in assets/ (noalign_translation_partape.pdf, …rpe.pdf, the filename is {alignment_opt}_{pose_relation} from OdometryConfig.save_path’s default template, so even the filename is provenance) hold the error-against-index plot and the same error colour-mapped onto the trajectory, for all four projections evo draws (xy, xz, yz, xyz). Here’s the first, un-aligned, translation-part:

The widget below sidesteps this entirely by generating its own trajectory rather than trying to salvage this one, more on why in a moment.

What actually came out: SemanticFusion on ScanNet

assets/SCANNET/odom_main.tex has the one number this whole assessment produced for a real algorithm run: SemanticFusion, the only algorithm here with pose to score (ElasticFusion tracks it; 3DMV and MinkowskiEngine consume ground-truth pose and never estimate one):

Algorithm	Mean [m]	STD [m]	Min [m]	Max [m]	RMSE [m]
SF.	0.266	0.223	0.000	6.156	0.421

Lead with the max, not the mean: 6.156 m against a mean of 0.266 m is not a slowly growing error, it’s one catastrophic tracking loss dragging the whole trajectory’s RMSE up with it. RMSE squares every error before averaging, so one 6 m outlier among a few thousand roughly 0.2 m errors contributes as much to RMSE as hundreds of typical frames combined.

This printed table doesn’t have a median column, but the underlying stats do, segtester/assessments/summarizeresults.py::OdometryRes reads seven fields out of every saved stats.json, not five:

self.all_results = pd.DataFrame(columns=[
    "rmse", "mean", "median", "std", "min", "max", "sse",
])
...
self.all_results = self.all_results.append({
    ...
    "rmse": stat_data["rmse"],
    "mean": stat_data["mean"],
    "median": stat_data["median"],
    ...

Median and sum-of-squared-error are computed and stored every run, they were just never selected into the final LaTeX table. That’s exactly why the median column exists in the first place: it’s the number that tells you whether the mean is being dragged around by a handful of catastrophic frames, and it survives even when nobody remembered to print it.

Try it: corrupt a trajectory and watch the five answers move

With JavaScript enabled this becomes an interactive explorer: a generated ground-truth path, an estimate you corrupt with drift, jitter and scale-error sliders, five alignment options and three pose relations, and live APE/RPE recomputed on every change.

Three sliders corrupt the estimate: drift (a small heading bias re-applied every step, so it accumulates, realistic odometry drift), jitter (independent per-step noise, which doesn’t accumulate), and scale error (every step’s translation multiplied by a fixed factor before integrating, a monocular-style scale drift). All three are applied by re-integrating the reference’s own relative motions rather than perturbing absolute positions directly, which is what makes the APE/RPE distinction visible: drift shows up in APE (it compounds over the whole path) but barely moves RPE at $\Delta{=}1$ (each individual step is still roughly the size it should be).

The one-click version of the whole post: hit “pure scale error” (this sets drift and jitter to zero and dials in a 1.3× scale error), leave alignment on noalign, and watch APE explode: translation error growing without bound as the path gets longer. Now switch to Sim3_Umeyama_alignment and watch it collapse to near zero, while the RPE number sitting right next to it never moves. That’s the whole argument about monocular versus RGB-D evaluation, live, because a pure scale error is invisible to a metric that only ever compares adjacent poses.

The “one catastrophic frame” button injects a single large translational jump partway through the integration, a stand-in for the tracking loss odom_main.tex’s max of 6.156 m is telling you about. Turn it on with a small amount of jitter and watch the mean and RMSE readouts jump while the median barely moves: the same story the report’s own numbers tell, reproduced with a button instead of a re-run.

For your own data: the paste-or-upload panel at the bottom takes two TUM-format files (timestamp tx ty tz qx qy qz qw, per line, the format evo.tools.file_interface .read_tum_trajectory_file actually reads, and so what this harness’s own trajectories are in before evo ever sees them). Nothing is uploaded anywhere; parsing and every metric run in this tab. It’s a simpler matcher than evo’s: poses are matched by line order and truncated to the shorter file, with no timestamp association, which is fine for two exports of the same run but not for arbitrary unsynchronised logs, and the widget says so. Export metrics (JSON) writes out the full error array alongside the summary stats and the parameters used, for whichever trajectory is currently loaded.

The crate decision, and a number worth reporting honestly

I extended calib-wasm (option 1: traj.rs, one pub mod traj; line in lib.rs) rather than write this in plain TypeScript, mainly for reuse: the SVD traj.rs needs is the same SVD zhang.rs already has, tested, and shipped to every visitor who’s loaded posts 38 or 41. That grew the crate from 60.0 kB to 81.6 kB (+21.6 kB, 13 new tests, 41 passing in total).

But I want to be honest about how much that reuse mattered versus how much it was necessary. The actual computation per drag is one $3\times3$ SVD plus one se(3) log-map per pose, at 1,600 poses, a plain-JavaScript transliteration of the same algorithm (one-sided Jacobi SVD, the same log-map formula above) ran 1,600 log-maps in about 1.4 ms and 1,600 SVDs (far more than the one Umeyama actually needs per drag) in about 17 ms, measured in Node on this machine. Scaled to what the widget actually does (one SVD, 1,600 log-maps), that’s comfortably sub-2ms: real-time by a wide margin. This is a case where WASM wasn’t strictly necessary for performance: the linear algebra here is just too small to matter at either speed. I chose Rust anyway because the SVD was sitting right there, tested, in a crate three other posts already load; if calib-wasm hadn’t existed yet, I’d have written this one in TypeScript instead.

Four properties get tested directly, per the brief that suggested them: Umeyama recovers a known similarity transform exactly on noiseless synthetic data; alignment with scale fixed at 1 (SE3_Umeyama_alignment) leaves a scaled input mismatched where Sim3_Umeyama_alignment removes it (the widget’s own one-click demo, as a unit test); the SE(3) log-map round-trips through its exponential across four cases including the small-angle branch; and RPE is invariant to a constant global rigid transform applied to an otherwise-perfect estimate, where APE is not, the whole argument of the “two questions” section, checked by assertion rather than just asserted in prose.