feat(evals): Habitat navigation benchmark: text-only arms, planner arm, video, parallel cases

[spomichter]

Contribution path

Draft. Stacked on feat/typesafe-agent (the TypeSafe module this benchmark evaluates), so the diff is the benchmark only. The TypeSafe v2 rewrite is #4212; the eval needs only set_trace_dir, world_state and agent_idle from the module and rebases onto whichever lands. Spec: engineering/typesafe-nav-benchmark/spec.md in the agents repo.

Problem

Compare the TypeSafe reactive agent, dimOS's planner, and coding agents with and without dimOS on long-horizon go-to-object navigation in real Habitat scenes, on an even text-only footing, with navigation-quality metrics, per-case video, and parallel execution.

Solution

One launch serves every arm: habitat-nav mcp-server demo-objects nav-skills, plus the agent's own modules.

• Habitat native publishes odom (PoseStamped) and a body-frame lidar scan, so type-safe-agent and raw-robot-bridge autoconnect by name. No relays, no remap blueprints.
• nav-skills: go_to(x, y, wait_s), stop_navigation, finish over the planner's goal / goal_reached / stop_movement topics. Goals carry the robot's floor z: HM3D houses are multi-level and a z=0 goal snaps onto furniture.
• TopicAgent (dimos/evals/agents/topic.py): evaluate whatever is already running. Publishes the instruction (text, or its (x, y) as a PointStamped) on one topic and returns when another goes truthy; trajectory from a module's set_trace_dir trace. The planner arm and the TypeSafe arm are two configurations of it.
• Raw bridge: topics allowlist, world_state/json (the same builder TypeSafe reads) and finished/json; ROBOT.md lists only the served topics.
• HabitatEnvironment: a navmesh tour from the scene file is driven before the task so the planner has a map (it plans only over observed surface, like dimsim_house explores first); task_start_ts; record_topics.
• nav_metrics: reached, time to object (run duration when never reached), facing, bumps (held command without displacement), path length, straightness, turn reversals; written as nav_metrics.json beside the recording.
• Sim: raw_topics, extra_env, --rerun-open none, opt-in rrd, video (Xvfb + dimos-viewer + ffmpeg, viewer window only).
• dimos evals run --case --parallel --container --repeat: one dimos evals run per case, in a container when given; docker/eval/Dockerfile.
• Suite dimos.evals.suites.habitat_nav with an example HM3D scene file (furniture-sized boxes, 4 cases, tour). The scene file layout is the detection3d_array_to_dict one from Ruthwik/feat/dimsim object annotations #4208 plus cases and tour.

Since the first push

• HSSD: ten scenes from feat(habitat): HSSD scene object and wall ground truth as Detection3DArray #4211's ground truth, misc/habitat/nav_cases.py picks targets with an easy / medium / hard mix (clearance, geodesic distance, detour) and writes cases plus a navmesh mapping tour per scene; 80 cases, end point on the object's centre. HSSD ships no navmesh: dimos/simulation/habitat/navmesh.py computes and caches one.
• Two MLS planner fixes the benchmark surfaced: goal snapping tries one step of z before the robot's height (a goal on an object snapped to its top), and local_map / region_bounds pair across the last four messages (latest-of-each never matched when the cloud lagged a tick, the planner starved).
• TypeSafe agent: task brief in the state, distance_m to the object's edge, a finished question that stops and publishes finished. Stool case: 300 s timeout with 14 bumps -> finished in 10 s, score 1.0.
• --video per case, dimos evals media captions and grids, every case instruction opens with a task brief.

Matrix on one HSSD case (stool, 3 m): planner 0.91, TypeSafe 1.00 (10 s), Pi no-dimOS Astra 1.00 / Fable 1.00 / GPT-5.6 1.00 / Opus 0.91, dimcode Fable 0.91 / Astra 0.72.

How to Test

# planner alone
dimos evals run dimos.evals.suites.habitat_nav --agent dimos.evals.agents.topic \
  --set send=goal --set send_type=point --set done=goal_reached --set done_type=Bool
# TypeSafe agent (TYPESAFE_API_KEY)
dimos evals run dimos.evals.suites.habitat_nav --agent dimos.evals.agents.topic \
  --set 'modules=["type-safe-agent"]' --set trace=TypeSafeAgent
# coding agent with dimOS / without dimOS
dimos evals run dimos.evals.suites.habitat_nav --agent dimos.evals.agents.dimcode --set model=gpt-6-astra
dimos evals run dimos.evals.suites.habitat_nav --agent dimos.evals.agents.pi --set no_dimos=true --set model=gpt-6-astra

Verified on the example scene, chair case (score 0.8 each): planner reached in 10.9 s; Pi + Astra without dimOS wrote a Zenoh observer and navigator and reached in 34 s; dimcode + Astra with dimOS reached in 16 s with one go_to. viewer.mp4 captured headless. 285 unit tests, ruff, mypy and pre-commit pass.

Not yet run: the TypeSafe arm (no key on the box) and the container path of --parallel (Docker registry unreachable on the box; job command unit-tested).

Checklist

• I have read and approved the CLA.

[codecov]

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
5987	1	5986	103

View the full list of 1 ❄️ flaky test(s)

dimos.e2e_tests.test_voice_browser::test_hold_to_talk_ships_a_decodable_recording[firefox]

Flake rate in main: 100.00% (Passed 0 times, Failed 6 times)

Stack Traces | 22.8s run time

voice_bridge = ('http://127.0.0.1:49902/', [])
fake_mic_page = <Page url='http://127.0.0.1:49902/'>

    def test_hold_to_talk_ships_a_decodable_recording(
        voice_bridge: tuple[str, list[AudioChunk]], fake_mic_page: Page
    ) -> None:
        url, chunks = voice_bridge
        chunks.clear()  # the module fixture is shared across both engines
        fake_mic_page.goto(url)
        mic = fake_mic_page.get_by_test_id("chat-audio_in-mic")
        # Enabled == transport connected; the manifest already placed the panel.
        expect(mic).to_be_enabled(timeout=120_000)
        expect(mic).to_have_attribute("data-state", "idle")
    
        mic.hover()
        fake_mic_page.mouse.down()
>       expect(mic).to_have_attribute("data-state", "recording", timeout=15_000)
E       AssertionError: Locator expected to have attribute 'recording'
E       Actual value: arming 
E       Call log:
E         - Expect "to_have_attribute" with timeout 15000ms
E         - waiting for get_by_test_id("chat-audio_in-mic")
E       
E       Aria snapshot:
E       - button "hold to talk": talk

chunks     = []
fake_mic_page = <Page url='http://127.0.0.1:49902/'>
mic        = <Locator frame=<Frame name= url='http://127.0.0.1:49902/'> selector='internal:testid=[data-testid="chat-audio_in-mic"s]'>
url        = 'http://127.0.0.1:49902/'
voice_bridge = ('http://127.0.0.1:49902/', [])

dimos/e2e_tests/test_voice_browser.py:104: AssertionError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}