Scoring driving trajectories: case studies and statistics
Five scorers on the same clips, then the paired statistics behind the method's advantage.
1. The five scorers
| Scorer | What it is | Trained? |
|---|
| EPDMS (rule) | Hand-written NAVSIM rule scorer, per-criterion | no |
| Naive VLM | Frozen VLM asked for a score in words | no |
| DriveCritic-style | Backbone LoRA-tuned to score, read out through its text channel | yes |
| DriveJudge-style | Rule subscores with a frozen-VLM applicability gate per criterion | no |
| This work | Frozen backbone, pooled context features, residual head | yes (47K) |
The two comparator rows reproduce published recipes, not published models. Neither DriveCritic nor DriveJudge releases a checkpoint (the Hub carries only DriveJudgeBench, a video-QA benchmark). The DriveCritic-style row applies that paper's recipe — tune the backbone, read the score from its text channel; the DriveJudge-style row applies its logic — let a frozen VLM decide, per criterion, whether a rule penalty applies. Neither number should be cited as that system's score.
2. Case studies
The Ideal row is derived, not asserted. The rule scorer is right wherever a hard-safety criterion fires — a collision, a red light, leaving the drivable area or an imminent time-to-collision always deserves its penalty. It is wrong in one situation: charging for slow progress when the scene gives a reason to slow. So the ideal score is the rule's own score with the progress penalty removed, and only where hard safety is clean and the corridor geometry confirms something to stop for. Each cell states which case applies.
Each case is one scene planned three ways. A long-tail case is one where the rule penalises progress while every hard-safety criterion is clean — it is charging for a slowdown it cannot justify. An ordinary case is one where the criteria agree.
Case 1 · long-tail · 6d8d91ad290f · us-pa-pittsburgh-hazelwood
Rule penalties here: VLA1 progress 0.683, VLA2 progress 0.599 — with collision, drivable-area, traffic-light and time-to-collision all 1.000.
Rule-based planner
EPDMS 0.857
Naive VLM 0.850
DriveCritic 0.630
DriveJudge 0.857
This work 0.625
Ideal 0.857
Ideal 0.857 — every criterion is clean, so a correct scorer should return the rule's own score
model's own words
The candidate trajectory is reasonable: it maintains a safe distance from the stopped vehicle ahead, stays within the marked lane, and avoids conflicts with surrounding traffic. The trajectory appears to be a smooth, legal stop at the intersection, which is contextually necessary given the traffic light and oncoming traffic.
Score: 95
VLA planner 1
EPDMS 0.744
Naive VLM 0.850
DriveCritic 1.000
DriveJudge 0.884
This work 0.823
Ideal 0.884
Ideal 0.884 — progress is penalised to 0.683 while collision, drivable-area, traffic-light and time-to-collision are all 1.000, and a vehicle is stopped 14 m ahead in this lane — the slowdown is correct, so removing that penalty gives 0.884
model's own words
The candidate trajectory is reasonable: it maintains forward progress in the current lane, avoids collisions with nearby vehicles and obstacles, and stays within the drivable area. The slight curve to the right appears contextually necessary to navigate around a stopped vehicle ahead while remaining legal and safe.
Score: 90
VLA planner 2 ours prefers
EPDMS 0.714
Naive VLM 0.850
DriveCritic 1.000
DriveJudge 0.893
This work 0.867
Ideal 0.893
Ideal 0.893 — progress is penalised to 0.599 while collision, drivable-area, traffic-light and time-to-collision are all 1.000, and a vehicle is stopped 14 m ahead in this lane — the slowdown is correct, so removing that penalty gives 0.893
model's own words
The candidate trajectory is reasonable: it maintains forward progress in the current lane, avoids collisions with nearby vehicles and obstacles, and stays within the drivable area. The slight curve to the right appears contextually necessary to navigate around a stopped vehicle ahead while remaining legal and safe.
Score: 90
Case 2 · long-tail · 32a336526197 · us-pa-pittsburgh-hazelwood
Rule penalties here: VLA1 progress 0.689, VLA2 progress 0.594 — with collision, drivable-area, traffic-light and time-to-collision all 1.000.
Rule-based planner
EPDMS 1.000
Naive VLM 0.850
DriveCritic 1.000
DriveJudge 1.000
This work 0.405
Ideal 1.000
Ideal 1.000 — every criterion is clean, so a correct scorer should return the rule's own score
model's own words
The candidate trajectory is reasonable: it maintains a straight path in the current lane, avoiding other vehicles and staying within the drivable area, which is appropriate for a straight-through approach at an intersection with traffic lights. The trajectory shows no unnecessary lane deviation and aligns with safe, legal, and comfortable driving behavior for this scenario.
Score: 95
VLA planner 1 ours prefers
EPDMS 0.746
Naive VLM 0.850
DriveCritic 1.000
DriveJudge 0.883
This work 0.907
Ideal 0.883
Ideal 0.883 — progress is penalised to 0.689 while collision, drivable-area, traffic-light and time-to-collision are all 1.000, and a vehicle is stopped 16 m ahead in this lane — the slowdown is correct, so removing that penalty gives 0.883
model's own words
The candidate trajectory is reasonable: it maintains a straight path in the current lane, avoiding other vehicles and staying within the drivable area, which is safe and legal. The trajectory shows minimal deviation, indicating good lane-keeping and comfort, and continues forward with progress toward the intersection without unnecessary maneuvers.
Score: 95
VLA planner 2
EPDMS 0.712
Naive VLM 0.850
DriveCritic 1.000
DriveJudge 0.893
This work 0.794
Ideal 0.894
Ideal 0.894 — progress is penalised to 0.594 while collision, drivable-area, traffic-light and time-to-collision are all 1.000, and a vehicle is stopped 16 m ahead in this lane — the slowdown is correct, so removing that penalty gives 0.894
model's own words
The candidate trajectory is reasonable: it maintains a straight path in the current lane, avoiding other vehicles and staying within the drivable area, which is appropriate for a stop-and-go urban intersection. The trajectory shows no unnecessary lane deviation and aligns with typical driving behavior for this scenario, suggesting safety, legality, and comfort.
Score: 95
Case 3 · long-tail · e64bbff1f3e0 · us-pa-pittsburgh-hazelwood
Rule penalties here: VLA1 progress 0.706, VLA2 progress 0.737 — with collision, drivable-area, traffic-light and time-to-collision all 1.000.
Rule-based planner
EPDMS 1.000
Naive VLM 0.850
DriveCritic 0.760
DriveJudge 1.000
This work 0.749
Ideal 1.000
Ideal 1.000 — every criterion is clean, so a correct scorer should return the rule's own score
model's own words
The candidate trajectory is reasonable: it maintains a safe distance from the black SUV and white sedan ahead, stays within the drivable lane, and progresses forward without abrupt maneuvers. The slight leftward curve appears contextually necessary to navigate the intersection safely while yielding to cross-traffic, avoiding potential collisions.
Score: 90
VLA planner 1 ours prefers
EPDMS 0.752
Naive VLM 0.850
DriveCritic 0.630
DriveJudge 0.882
This work 0.980
Ideal 0.882
Ideal 0.882 — progress is penalised to 0.706 while collision, drivable-area, traffic-light and time-to-collision are all 1.000, and a vehicle is stopped 20 m ahead in this lane — the slowdown is correct, so removing that penalty gives 0.882
model's own words
The candidate trajectory is reasonable: it maintains a safe distance from the stopped vehicle ahead, stays within the marked lane, and proceeds straight through the intersection without unnecessary deviation. The trajectory aligns with legal and comfortable driving behavior, showing no risky maneuvers or lane violations that would compromise safety or progress.
Score: 95
VLA planner 2
EPDMS 0.763
Naive VLM 0.850
DriveCritic 0.680
DriveJudge 0.879
This work 0.921
Ideal 0.879
Ideal 0.879 — progress is penalised to 0.737 while collision, drivable-area, traffic-light and time-to-collision are all 1.000, and a vehicle is stopped 20 m ahead in this lane — the slowdown is correct, so removing that penalty gives 0.879
model's own words
The candidate trajectory is reasonable: it maintains a safe distance from the stopped vehicle ahead, stays within the lane, and proceeds straight through the intersection without violating traffic rules. The trajectory appears smooth and contextually appropriate for continuing forward in a typical urban driving scenario.
Score: 95
Case 4 · long-tail · bc686838d510 · us-pa-pittsburgh-hazelwood
Rule penalties here: VLA1 progress 0.624, VLA2 progress 0.523 — with collision, drivable-area, traffic-light and time-to-collision all 1.000.
Rule-based planner
EPDMS 1.000
Naive VLM 0.850
DriveCritic 1.000
DriveJudge 1.000
This work 0.508
Ideal 1.000
Ideal 1.000 — every criterion is clean, so a correct scorer should return the rule's own score
model's own words
The candidate trajectory is reasonable: it maintains forward progress in the current lane, avoids collisions with nearby vehicles and obstacles, and stays within the drivable area. The slight curve to the right appears contextually necessary to navigate around the intersection safely while respecting traffic flow.
Score: 90
VLA planner 1
EPDMS 0.723
Naive VLM 0.850
DriveCritic 1.000
DriveJudge 0.890
This work 0.786
Ideal 0.890
Ideal 0.890 — progress is penalised to 0.624 while collision, drivable-area, traffic-light and time-to-collision are all 1.000, and a vehicle is stopped 21 m ahead in this lane — the slowdown is correct, so removing that penalty gives 0.890
model's own words
The candidate trajectory is reasonable: it maintains a safe distance from the stopped vehicle ahead, stays centered in the lane, and follows the road’s curvature appropriately for a turn. The trajectory is legal, comfortable, and makes progress toward the intersection without unnecessary lane deviation, aligning with typical driving behavior in this urban setting.
Score: 95
VLA planner 2 ours prefers
EPDMS 0.687
Naive VLM 0.850
DriveCritic 0.630
DriveJudge 0.902
This work 0.826
Ideal 0.902
Ideal 0.902 — progress is penalised to 0.523 while collision, drivable-area, traffic-light and time-to-collision are all 1.000, and a vehicle is stopped 21 m ahead in this lane — the slowdown is correct, so removing that penalty gives 0.902
model's own words
The candidate trajectory is reasonable: it maintains forward progress in the current lane, avoids collisions with nearby vehicles and obstacles, and stays within the drivable area. The slight curve to the right appears contextually necessary to navigate around a parked vehicle and maintain a safe distance from the intersection, without violating traffic rules or causing discomfort.
Score: 90
Case 5 · ordinary · ebe3f434bfee · us-pa-pittsburgh-hazelwood
Every criterion is clean for all three planners; the rule scores them all 1.000.
Rule-based planner
EPDMS 1.000
Naive VLM 0.600
DriveCritic 0.648
DriveJudge 1.000
This work 0.715
Ideal 1.000
Ideal 1.000 — every criterion is clean, so a correct scorer should return the rule's own score
model's own words
The candidate trajectory is reasonable: it maintains a safe distance from the white car crossing in front and stays centered in the lane, avoiding any risky maneuvers. The trajectory appears to be a smooth, legal turn into the intersection, consistent with the surrounding traffic and road markings, and shows no unnecessary lane deviation.
Score: 90
VLA planner 1
EPDMS 1.000
Naive VLM 0.600
DriveCritic 0.649
DriveJudge 1.000
This work 0.933
Ideal 1.000
Ideal 1.000 — every criterion is clean, so a correct scorer should return the rule's own score
model's own words
The candidate trajectory is reasonable: it maintains a safe distance from the intersecting traffic, stays within the drivable lane, and avoids conflict with other vehicles and agents. The trajectory appears to smoothly navigate the intersection without abrupt maneuvers, suggesting good progress and comfort while adhering to traffic rules.
Score: 90
VLA planner 2 ours prefers
EPDMS 1.000
Naive VLM 0.850
DriveCritic 0.648
DriveJudge 1.000
This work 0.956
Ideal 1.000
Ideal 1.000 — every criterion is clean, so a correct scorer should return the rule's own score
model's own words
The candidate trajectory is reasonable: it maintains forward progress in the current lane, avoids collisions with nearby vehicles, and stays within the drivable area. The slight curve to the right appears contextually necessary to navigate the intersection safely while yielding to cross-traffic, and the trajectory is smooth and comfortable for the ego vehicle.
Score: 90
Case 6 · ordinary · 875f6c89666f · us-pa-pittsburgh-hazelwood
Every criterion is clean for all three planners; the rule scores them all 1.000.
Rule-based planner
EPDMS 1.000
Naive VLM 0.600
DriveCritic 0.680
DriveJudge 1.000
This work 0.801
Ideal 1.000
Ideal 1.000 — every criterion is clean, so a correct scorer should return the rule's own score
model's own words
The candidate trajectory is reasonable: it maintains forward progress while staying within the drivable lane, avoiding collisions with nearby vehicles and obstacles. The slight curve appears to accommodate the intersection geometry and traffic flow without abrupt maneuvers, suggesting a smooth, legal, and safe path.
Score: 90
VLA planner 1 ours prefers
EPDMS 1.000
Naive VLM 0.600
DriveCritic 0.820
DriveJudge 1.000
This work 1.148
Ideal 1.000
Ideal 1.000 — every criterion is clean, so a correct scorer should return the rule's own score
model's own words
The candidate trajectory is reasonable: it maintains forward progress while staying within the drivable lane, avoiding collisions with other vehicles and obstacles. The slight curve appears to accommodate the intersection geometry and traffic flow, suggesting a safe and legal maneuver without abrupt lane changes.
Score: 90
VLA planner 2
EPDMS 1.000
Naive VLM 0.600
DriveCritic 0.820
DriveJudge 1.000
This work 0.977
Ideal 1.000
Ideal 1.000 — every criterion is clean, so a correct scorer should return the rule's own score
model's own words
The candidate trajectory is reasonable: it maintains forward progress in the current lane, avoids collisions with nearby vehicles, and stays within the drivable area. The slight curve appears to accommodate the intersection geometry and traffic flow without abrupt maneuvers, suggesting a smooth, legal, and safe path.
Score: 90
3. Do the scorers separate the three planners?
A scorer that returns the same number for two planners cannot rank them. Over the 6 cases × 3 planners shown above:
| Scorer | Distinct values | Range | Planner pairs left tied
(of 18) |
|---|
| EPDMS | 10/18 | 0.687 – 1.000 | 6 (33%) |
| Naive VLM | 2/18 | 0.600 – 0.850 | 16 (89%) |
| DriveCritic | 8/18 | 0.630 – 1.000 | 6 (33%) |
| DriveJudge | 10/18 | 0.857 – 1.000 | 6 (33%) |
| This work | 18/18 | 0.405 – 1.148 | 0 (0%) |
4. Statistical evidence for the method
These statistics come from the 847-pair NAVSIM evaluation set, not from the clips above. Five scenes cannot support a significance test, and none is claimed for them. The cases show how the scorers behave; this section is the evidence.
Arms were trained with the same budget on the same seeds, so each row is a paired per-seed difference against this work, not a gap between independently reported means. Lower MAE against the teacher is better; a negative Δ favours this work.
| This work vs. | Paired Δ MAE | Seeds favouring
this work | Paired t | Sign-test p | Verdict |
|---|
| Mean-pooled readout | -0.00069 | 4/4 | -2.43 | 0.125 | consistent trend |
| Published query readout | -0.00352 | 3/3 | -8.61 | 0.250 | clear |
With this seed budget the exact sign test cannot return a value below 0.125 (n=4) / 0.250 (n=3) even when every seed agrees — a p sitting at its floor means the seed budget is exhausted, not that the effect is absent.
Where the method stands against every arm
| Arm | MAE vs teacher | ± seed s.d. | Seeds |
|---|
| EPDMS rule scorer | 0.04329 | 0.00000 | 1 |
| Naive VLM (prompted) | 0.25680 | 0.00000 | 1 |
| DriveCritic-style tuned scorer | 0.21947 | 0.00139 | 2 |
| Published query readout | 0.03971 | 0.00009 | 3 |
| VLoE-style score token | 0.03842 | 0.00111 | 3 |
| Mean-pooled readout | 0.03712 | 0.00023 | 4 |
| This work | 0.03643 | 0.00070 | 4 |
The two comparator rows are recipe reproductions, as stated in Section 1. Source files: cmp_v3.json, quant.json, stats.json, cases.json.