ClimbrerCoach

A real-time climbing coach that runs in your browser.

It tracks your body, finds the holds, and tells you (live, with an overlay and a voice) which limb to move to which hold next, why, and roughly how hard the route is.

Live demo: add your Vercel URL here after deploying (see Deploy).

Try it in 10 seconds

Open the app. Demo mode starts immediately, with no camera, permissions, or API key. A simulated climber works up a wall, and every move it makes is chosen by the real planner. Click the wall to add or remove holds and watch it re-plan. Switch between Greedy, A* and Beam, and open Summary for a heatmap, height timeline and grade breakdown.

Camera and Video run the same pipeline on a webcam or a clip from your device. Tap a hold to teach it the hold colour, and optionally click the four wall corners to calibrate. Frames never leave your browser.

What it does

Capability	How it works
Pose tracking	MediaPipe Pose Landmarker, in the browser (WebGPU/WASM), lazy-loaded so Demo stays light
Hold detection	Tap-to-sample HSV colour (circular hue mean), then connected components in a Web Worker
Wall calibration	4-point homography from pixels to wall metres, rejecting degenerate corner picks
Contacts	Nearest hold per limb, smoothed with hysteresis so contacts don't flicker
Balance	Signed distance from the centre of mass to the hull of the holds that stay on during the move (three points of contact)
Planning	Greedy ranking, plus A* and beam search with true multi-move lookahead
Grade	A documented, interpretable heuristic mapped to the V-scale (not a trained model)
Coach	Instant rule-based cue; upgraded to a Claude cue when the API is available
Summary	Hold-usage heatmap, height over time, grade factors, JSON export

Architecture

flowchart LR
    subgraph Browser
      SRC[Demo sim / camera / video] --> POSE[MediaPipe pose]
      SRC --> HOLDS[Hold detector<br/>Web Worker]
      POSE --> H[Homography px→m]
      HOLDS --> H
      H --> PIPE[CoachPipeline<br/>contacts · balance · planner · grade]
      PIPE --> UI[Canvas overlay + HUD]
      PIPE --> REC[(Session recorder)]
      PIPE --> RULE[Rule cue]
    end
    PIPE -- move changed, ≤1 per 3 s --> API[/api/coach · Go/]
    API -- key set --> CLAUDE[Claude]
    API -- no key / any error --> RULES[Rule cue]

Loading

Path	Language	Responsibility
src/core/	TypeScript	Pure, DOM-free planning brain: geometry, stability, reach, ranking, A*/beam, contacts, grade, homography, the frame pipeline, the simulator, the rule cue. Fully unit-tested.
src/vision/	TypeScript	MediaPipe wrapper and the hold detector (pure core + worker).
src/ui/	React + Canvas	Stages (Demo, Camera/Video), overlay renderer, sidebar, coach, summary.
backend/coach/	Go	/api/coach: validation, per-IP rate limit, rule cue, Claude client (official Go SDK). Never 5xx.
backend/web/, cmd/server/	Go	Production server: static app with SPA fallback and immutable asset caching, plus the API.
api/coach.go	Go	Vercel function wrapping the same handler.
fixtures/rule-cues.json	JSON	Run by both the TS and Go test suites so the browser and server cues can't drift.

How the planner works

• State: the hold index under each limb (-1 means off the wall).
• Successors: any reachable hold for a movable limb. A hand target must be within arm reach (plus a dynamic-move allowance) and within body span of each planted foot. A foot target must sit below the hands. Feet never share a hold.
• Edge cost: a positive step cost plus sideways travel, reach, balance during the move, and a hand-matching penalty.
• Heuristic: the remaining height to the top divided by the maximum gain per move.
• Fallback: when the top is beyond the search budget, A* and beam return the path to the most promising state, so there is always a next move when one exists.

The greedy ranker scores height as progress from where the limb is now. See src/core/plan.ts and src/core/rank.ts.

Run it locally

You need Node 22+ and Go 1.25+.

npm install
npm run dev          # app on http://localhost:5173 (Demo works on its own)
npm run dev:api      # optional, in another terminal: Go coach API on :8787 (Vite proxies /api)

npm test             # Vitest: core, vision, UI helpers
go test ./...        # Go: handler, rate limiter, rule-cue parity, static server
npm run e2e          # Playwright against the production build served by Go

Without the Go API running, the app still works: the coach falls back to the in-browser rule cue. To use Claude, put ANTHROPIC_API_KEY in your shell (see .env.example) before npm run dev:api. The model defaults to claude-opus-5-5 (override with COACH_MODEL).

Docker

docker compose up --build    # http://localhost:8080 (set ANTHROPIC_API_KEY to enable Claude)

The image is multi-stage: Node builds the app, Go builds a static binary, and the runtime is distroless with just the binary and the built files.

Deploy

On Vercel, import the repo and deploy. Vercel builds the Vite app and picks up api/coach.go as a Go function. Optionally set ANTHROPIC_API_KEY (and COACH_MODEL) in the project's environment variables.

Honesty notes

• The grade is a heuristic (four documented factors in grade.ts), not a trained model. The summary shows exactly which factors drove it.
• The coach labels its source. You see "Rules" until a Claude reply arrives, and the badge says which backend is live.
• Demo is a simulation. The body is scripted, but every hand move it makes is the planner's choice through the same pipeline the camera uses.
• Camera and video frames never leave the browser. Only a small move description (limb, distances, balance, grade) goes to /api/coach.

What changed from v1

v1 (tagged v1) was a Python desktop app (OpenCV + MediaPipe + FastAPI). v2 is a rewrite:

• It runs in the browser with zero install, and has a Go backend.
• Balance is measured correctly. v1 measured the centre of mass against the feet only, so on a vertical wall it read "off balance" almost always.
• A* actually plans ahead. v1 searched over holds reachable from the current pose only, so it rarely found a path and often suggested nothing.
• The ranking no longer favours feet. v1 scored the absolute height of the target hold, so any foot move outranked any hand move.
• The body has limits. A body-span constraint stops the "infinitely stretched climber" v1 allowed.