The harness builds the model, a fixed checker scores it, and builds without a reference Lego LDraw file also receive a likeness score from LLMs and from people.
Instructions or photos → LDraw
The task prompt gives each harness a 4-hour time budget per build to create an LDraw model from the benchmark’s PDF or photos and render four views.
Each build is an independent attempt. The saved model and renders are passed to the judge.
Evaluate the model and compute a score
Fixed checks evaluate the model’s parts, transforms, connections and overlaps, then compute the build score. A reference Lego LDraw file is used for matching when one exists. The checker also says how much of the build matches an example model.
The scoring rules are deterministic. Human opinions and LLM ratings do not contribute to this score.
Rate how much the build looks like the car
Three LLMs compare the rendered build with the photos and read the checker’s note on example overlap. A build that copies an example model scores near zero, even when it looks like a finished LEGO car.
Each rating is kept. Their average is the LLM likeness, shown as a percentage.
Manually score how the build looks
For these benchmarks, humans compare the rendered build with the input photos and manually rate its likeness from 0 to 10.
Human ratings stay separate from the build score and are shown as a percentage. The published score averages the LLM percentage and the human percentage, and uses whichever one exists when the other is missing.