Beam can't see an image. So it learned to write its own eyes.
A text-only Beam agent writes its own computer-vision pipeline, learns from exact grader feedback, and improves without weight updates. Its best 10-round run reaches 0.179 held-out partial credit on 64 unseen real apartment plans and 0.803 room F1 on clean plans.
Beam actually learned: after the grader reported zero rooms on scans, Beam diagnosed its thresholding failure and adopted adaptive thresholding. Public partial credit jumped from 0.022 to 0.179.
The problem. Many floor plans exist only as images or scans. A room graph (rooms + doors) answers "can a wheelchair reach every room?", "how do I get out?" and "how many rooms?". This environment teaches a text-only model to build that graph from a real plan, and grades it exactly.
In one screen.
reset() / step(code), strict and dense rewards, 24 practice + 64 held-out plans, Docker grading, and a GRPO example. Strict reward remains 0.000 versus 0.242 for the reference: rooms are emerging; doors are the open research problem.Data, grader, Docker files and every run: huggingface.co/datasets/hamoodey/floorplan-graph-rl
Plan p17 (clean style). Panel 3: rooms recovered by Beam's own program (Beam low, round 9). Panel 4: the step-free route to the exit, a networkx shortest path over the true door graph; this is what a recovered graph is for. Beam's doors are not right yet (door F1 0), so its own route would not be trustworthy.
| Evidence | Result |
|---|---|
| Beam low, 10 feedback rounds | 0.000 → 0.182 public partial; 0.179 held-out partial |
| General tips playbook, 3 repeats each | 0.032 → 0.075 held-out partial |
| Clean-plan room recovery, best 10-round run | 0.803 room F1 |
| Strict held-out reward | 0.000 for every Beam setup; 0.242 reference |
from code.env import FloorplanEnv, extract_code
env = FloorplanEnv(set="public") # 24 practice plans; "hidden" only for the final score
prompt = env.reset() # instruction + contract + style guide
reward, partial, feedback, info = env.step(extract_code(model_reply))
next_prompt = prompt + previous_code + feedback # the in-context loop used here
def floorplan_reward(completions, **kw): # GRPO-style reward (e.g. TRL GRPOTrainer)
return [sum(FloorplanEnv("public").step(extract_code(c if isinstance(c, str) else c[-1]["content"]))[:2])
for c in completions] # strict + dense partial creditUnsaturated: strict reward is still 0.000 for every model loop here, while the hand-written reference reaches 0.242. Lots of room to train.
reasoning_content, Beam low, 10-round run)Round 5 (practice-plan partial credit 0.022 → 0.179): “My previous approach using simple thresholding produced 0 rooms on scan styles and poor results. I need a more robust approach.”
Round 7 (practice-plan partial credit 0.078 → 0.174): “The previous attempt has low IoU due to morphological closing causing over-merge and room splitting. Let me focus on better wall detection and not over-engineer doors yet.”
The grader is exact (rooms at IoU 0.7, doors within 0.5 m). Feedback uses only the 24 practice plans; the 64 held-out plans are from other buildings and never feed back. Every prompt, reasoning trace, program and score is in runs/.
Known limits. Kimi K3 did not finish (no reply to its round-1 request in 31 min); Beam high effort finished (held-out 0.016 in a single 3-round run; 0.176 with best of 4 samples per round). Most numbers are one run; the playbook result has 3 repeats with a wide spread. Doors are nearly unsolved (door F1 0.000 on every setup but one). We did not update Beam's weights.
FloorplanEnv.reset() gives the task prompt; step(code) runs Beam's run.py and returns the strict reward, a dense partial-credit reward and per-style feedback (room F1, door F1, top failures).task/environment, task/tests).We did not update Beam's weights; in these runs Beam improves only in-context, by retrying with the environment's feedback. The environment is ready for RL fine-tuning of Beam (open weights).
| Model | Rounds | Best public strict | Best public partial | Held-out strict (64 plans) | Held-out partial | Best round |
|---|---|---|---|---|---|---|
| Beam low effort, 3 rounds (repeat 1) | 3 | 0.000 | 0.000 | 0.000 | 0.000 | round 1 |
| Beam low effort, 3 rounds (repeat 2) | 3 | 0.000 | 0.026 | 0.000 | 0.035 | round 3 |
| Beam low effort, 3 rounds (repeat 3) | 3 | 0.000 | 0.058 | 0.000 | 0.061 | round 2 |
| Beam low + playbook, 3 rounds (repeat 1) | 3 | 0.000 | 0.101 | 0.000 | 0.102 | round 1 |
| Beam low + playbook, 3 rounds (repeat 2) | 3 | 0.000 | 0.000 | 0.000 | 0.000 | round 1 |
| Beam low + playbook, 3 rounds (repeat 3) | 3 | 0.000 | 0.136 | 0.000 | 0.123 | round 1 |
| Beam high effort, 3 rounds | 3 | 0.000 | 0.010 | 0.000 | 0.016 | round 2 |
| Beam high + playbook, 3 rounds | 3 | 0.000 | 0.000 | 0.000 | 0.000 | round 1 |
| Beam low, 10 rounds, each revises the best code so far | 10 | 0.000 | 0.182 | 0.000 | 0.179 | round 9 |
| Beam high, 5 rounds, best of 4 samples per round | 5 | 0.000 | 0.181 | 0.000 | 0.176 | round 4 |
| Kimi K3: no reply to round 1 in 31 min; did not finish. | ||||||
| Earlier baseline: Opus 5.5, one 2-hour agent run with tools (a different setup; strict score) | - | - | - | 0.032 | - | - |
| Reference solution (hand-written, for scale) | - | - | - | 0.242 | 0.586 | - |
Strict = the official score: plan score = room F1 x door F1 (rooms matched one to one at IoU ≥ 0.7, a door counts when its rooms match and its centre is within 0.5 m); gate g = min(1, mean clean / 0.6); strict = g x mean over the 48 hard plans. Partial = mean over plans of (room F1 + door F1) / 2, the dense RL signal. Public scores drive the feedback; held-out plans are used only for the final score.
| Run | Room F1 clean | outline+clutter | clutter+scan | outline+scan | Door F1 (mean) | Strict | Partial |
|---|---|---|---|---|---|---|---|
| Beam low effort, 3 rounds (repeat 1) | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
| Beam low effort, 3 rounds (repeat 2) | 0.045 | 0.028 | 0.000 | 0.203 | 0.000 | 0.000 | 0.035 |
| Beam low effort, 3 rounds (repeat 3) | 0.149 | 0.011 | 0.036 | 0.291 | 0.000 | 0.000 | 0.061 |
| Beam low + playbook, 3 rounds (repeat 1) | 0.695 | 0.025 | 0.013 | 0.087 | 0.000 | 0.000 | 0.102 |
| Beam low + playbook, 3 rounds (repeat 2) | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
| Beam low + playbook, 3 rounds (repeat 3) | 0.813 | 0.035 | 0.013 | 0.105 | 0.005 | 0.000 | 0.123 |
| Beam high effort, 3 rounds | 0.059 | 0.024 | 0.024 | 0.018 | 0.000 | 0.000 | 0.016 |
| Beam high + playbook, 3 rounds | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
| Beam low, 10 rounds, each revises the best code so far | 0.803 | 0.043 | 0.036 | 0.550 | 0.000 | 0.000 | 0.179 |
| Beam high, 5 rounds, best of 4 samples per round | 0.789 | 0.010 | 0.050 | 0.560 | 0.000 | 0.000 | 0.176 |
| Reference solution | 0.969 | 0.524 | 0.779 | 0.651 | 0.441 | 0.242 | 0.586 |
The "playbook" is a 46-line list of general computer-vision techniques added to the prompt (code/playbook.md); no answers, no plan-specific hints. With tips the mean is about 2.4x higher, but n = 3 and one repeat in each group scored 0, so this is not a firm result.
| Group | Runs | Held-out strict | Held-out partial |
|---|---|---|---|
| beam-low (no playbook) | 3 | mean 0.000, spread 0.000 to 0.000 (runs: 0.000, 0.000, 0.000) | mean 0.032, spread 0.000 to 0.061 (runs: 0.000, 0.035, 0.061) |
| beam-low + playbook | 3 | mean 0.000, spread 0.000 to 0.000 (runs: 0.000, 0.000, 0.000) | mean 0.075, spread 0.000 to 0.123 (runs: 0.102, 0.000, 0.123) |
Each round Beam gets its previous program and the grader's report (per-style room F1 / door F1, top failures from the public plans) and writes a new run.py. Below: Beam's own reasoning (reasoning_content) in the rounds where its score rose, with the score before and after.
| Round | Public partial | Public strict | Room F1 clean / outline+clutter / clutter+scan / outline+scan | Lines of code | Output tokens (reasoning) | Files |
|---|---|---|---|---|---|---|
| 1 | 0.000 | 0.000 | 0.000 / 0.000 / 0.000 / 0.000 | 109 | 1696 (577) | run.py · reasoning |
| 2 | 0.025 | 0.000 | 0.151 / 0.050 / 0.000 / 0.000 | 101 | 1136 (154) | run.py · reasoning |
| 3 | 0.018 | 0.000 | 0.118 / 0.022 / 0.000 / 0.000 | 101 | 1701 (710) | run.py · reasoning |
| 4 | 0.022 | 0.000 | 0.000 / 0.000 / 0.000 / 0.178 | 105 | 1312 (229) | run.py · reasoning |
| 5 | 0.179 | 0.000 | 0.707 / 0.117 / 0.123 / 0.486 | 107 | 1216 (152) | run.py · reasoning |
| 6 | 0.078 | 0.000 | 0.373 / 0.085 / 0.125 / 0.042 | 97 | 1058 (103) | run.py · reasoning |
| 7 | 0.174 | 0.000 | 0.739 / 0.110 / 0.088 / 0.458 | 97 | 1173 (171) | run.py · reasoning |
| 8 | 0.176 | 0.000 | 0.708 / 0.116 / 0.122 / 0.459 | 160 | 2329 (696) | run.py · reasoning |
| 9 | 0.182 | 0.000 | 0.739 / 0.116 / 0.107 / 0.497 | 128 | 2067 (708) | run.py · reasoning |
| 10 | 0.180 | 0.000 | 0.739 / 0.116 / 0.097 / 0.491 | 168 | 1911 (74) | run.py · reasoning |
Round 2 (public partial 0.000 → 0.025): “The previous submission failed because it wrote no rooms? Actually it says pred 0 rooms. Probably adaptive threshold or contour issue due to image dimensions mismatch? cv2.imread on grayscale 8-bit should work. But maybe plan.png is larger? We resized. The wall mask maybe all white? Let's not overthink; output empty likely due to exception? But status ok. Could be min_area too high? No. Given time budget, I'll provide a slightly improved but stil ...”
Round 5 (public partial 0.022 → 0.179): “My previous approach using simple thresholding produced 0 rooms on scan styles and poor results. I need a more robust approach. Let me think about what's needed: detect walls, close small gaps, find free regions. The previous code closed gaps with morphology but maybe threshold was wrong, or morphology too aggressive. Given the complexity and time, I'll improve: use Otsu threshold, invert properly, handle scan styles with adaptive thresholding. A ...”
Round 7 (public partial 0.078 → 0.174): “The previous attempt has low IoU due to morphological closing causing over-merge and room splitting. Let me focus on better wall detection and not over-engineer doors yet. Key: rooms IoU low. Need better thresholding, keep wall gaps. Let me write improved version using Otsu + closing small gaps under 0.5m only, proper free space. Also handle scan rotation? The polygons move with image, so no need to unrotate. Let me improve: use adaptive threshol ...”
Left: the rooms (colour), doors (red dot; green = door to outside) and door edges recovered by Beam low, 10 rounds, each revises the best code so far (round 9), drawn over the plan. Right: the true room graph. The blue line is a wheelchair route to the exit: a networkx shortest path over the door graph from the room farthest from the exit to the outside door, computed only from the recovered map. Blue dot = start.
Floor plans exist mostly as images. A room graph turns one into something a program can answer questions about: is every room reachable from the exit, what is the step-free route for a wheelchair, how many bedrooms, which rooms are interior. The environment is a ready-made, exactly graded RL task for teaching a text-only model to do visual work through code, with dense partial credit so early programs still get a learning signal.
Plan geometry: Swiss Dwellings v3.0.0, Zenodo, DOI 10.5281/zenodo.7788422, CC BY 4.0. Plans re-drawn from that geometry. Built by Hamood (hamoodey) for Reflection's Build with Beam contest. Beam: Beam-1.0-501B-A23B via the Reflection API.
plan.png and manifest.json.