← hilos.simmer.io

Decisions

The design of the TypeScript build: what was chosen, why, and the numbers behind it. It was the spec the agents built from, kept up to date as things were measured and changed.

Status key: ✅ decided and built · 📏 an estimate, not yet measured

Constraints we build: #5 (nothing lost, required), #6 (sole stays put), #4 (iPad memory). #1 (full mesh) comes free: we edit the real 700k-face mesh, no proxy. #2 (next frame) and #3 (picking) are measured and reported, built as far as the design allows.

Measured facts about the shoe

From brief/shoe/shoe.glb (script: scripts/measure.mjs, to be committed):

A stroke at Ghost's default 60 mm radius moves about a third of the upper. That number drives the undo and save format below.

1. The tool

✅ One tool: Grab. It's Ghost's Morph done properly, using the same conventions (G key, radius in mm, a ring cursor lying on the surface) so a designer doesn't relearn anything.

• Hover: a ring on the surface shows the radius, and a soft gradient inside it shows the falloff. Over the sole the ring turns grey to show the sole is locked.
• Press and drag: grabs the region under the pen at press time, then moves it in the screen plane through the grabbed point. Release commits the edit.
• Directions come from the view. For "raise the toe," view from the side and drag up. For "slim the heel," view from the top and drag inward. "Push a panel out" works the same way from a view that faces along the panel.
• Radius: [ and ] keys, plus a slider in mm. Range 10–80 mm, default 40 mm. ✅ The 80 mm cap bounds undo memory (§3).
• Falloff: a smooth polynomial curve. It uses only + - * / (no Math.pow or exp), so the same input produces exactly the same output in every browser. Undo and sync rely on that.
• ✅ Back-half fade (replaces the facing filter): the weight depends only on position: falloff × a fade along the grabbed normal (fully on in front of the grabbed point, off at r/2 behind it) × the seam ramp. This keeps the far wall across the shoe opening still, while the lining just behind the grabbed wall moves with it, preserving the shell's thickness. Why: the first design used vertex normals, and on thin details (laces about 3 mm across, the collar rim) neighbouring normals flip, which gave weights jumping 0.99 across a 0.05 mm edge, so a drag tore "curtains" of stretched triangles. Found by the wasm agent; tests/9-no-tears pins it (the weight's slope along every edge ≤ 1.54/r + 3/r + 0.15 per mm).
• iPad: the pen edits and fingers navigate (one finger orbits, two fingers pan and zoom). Undo and redo are toolbar buttons. Two- and three-finger taps (the Procreate convention) are built in the wasm build only, not in main.
• Touch with no pen (iPhone, or an iPad without a Pencil): until a pen is detected, one finger edits while the Grab tool is active and two fingers navigate. Switching is automatic on the first pointerType === 'pen' event.
• Device testing: a real iPad became available on 2026-10-29 (model and iPadOS version recorded once it is paired), so iPad numbers come from the iPad itself, not a proxy. Earlier plans used an iPhone as the stand-in; any iPhone numbers that remain in the write-up are labelled as iPhone. Numbers come from Safari Web Inspector on a Mac over USB (Timelines tab: frame and JavaScript time; Memory timeline), served over HTTPS (local HTTPS dev server or a hosted preview URL). Whether the iPad has a Pencil decides whether pen input was tested on a device, and the write-up says which.
• Desktop: left-drag on the shoe edits; left-drag off it, or Alt-drag anywhere, orbits; right-drag pans and the wheel zooms (OrbitControls defaults). Cmd/Ctrl+Z undoes and Shift+Cmd/Ctrl+Z redoes.
• ✅ Pen pressure: ignored in v1, because pressure doesn't map naturally onto a grab. Pen events (pointerType: 'pen', with pressure) are simulated in Playwright through the Chrome DevTools Protocol (Input.dispatchMouseEvent) to test pen-vs-finger routing. We have no tablet, so real pressure feel is untested.

Additional feature (one only): ✅ an "original" overlay toggle that shows the untouched shoe as a faint ghost, so you can compare before and after.

Left out, and why (this goes in the write-up):

• Symmetry: a shoe is not mirror-symmetric (medial and lateral sides differ), so mirroring would be wrong.
• Smooth brush: Ghost already has one.
• Push along the normal: it would need per-vertex directions in every edit record, roughly 3× the undo memory. It's the next tool I'd add.
• Editing the sole: locked by design (#6).
• Remeshing and subdivision: not allowed by the brief.

2. Source of truth

✅ There is one owner: a ShoeModel class, plain TypeScript, no React.

• The CPU typed arrays are the truth for the current shape.
• An integer displacement accumulator stores each upper vertex's offset in units of 0.01 mm (Int32×3). Every vertex position is computed as base + acc × 0.01.
• The GPU is a cache. Buffers are re-uploaded from the CPU arrays whenever needed, for example after a context loss.
• IndexedDB holds the durable truth: the base mesh (cached once by its SHA-256) and an append-only log of operations.
• React holds UI state only (tool, radius, toggles). It never holds mesh data.

✅ Only two things may write positions: applyEdit/revertEdit, and the live preview of the stroke in progress. The preview writes base + (acc + w×D_now) × 0.01 and never touches acc until the stroke is committed. That means cancelling a stroke (for example on context loss) restores the shape exactly.

✅ The sole is never written. Its arrays are read-only after loading.

3. What an edit is

✅ An edit record contains:

{ id: uuid,
  mask: bitset over upper verts (185,465 bits = 23 KB),
  weights: Uint16 per set bit,
  D: Int32×3 (in 0.01 mm) }

Each vertex moves by round(w/65535 × D), and that delta is added to acc.

• Undo subtracts the same integer deltas, which are recomputed from the record, so the shape comes back bit-exact. Redo adds them again.
• Edits commute: the deltas are integer additions, so the order they're applied in doesn't change the result. That's what makes reload replay and multi-tab merging trivial (§4).

Size per edit, 📏 estimated from the measured vertex counts:

100 steps of undo at worst ≈ 26 MB.

Why this format:

• Storing per-vertex float before and after positions would be 28 B per vertex, about 1.7 MB per 60 mm stroke, so 100 steps would be roughly 170 MB.
• The mask plus weights format works because a grab moves every vertex in its region along the same vector D.

✅ Why changes per edit, not full snapshots or pure replay:

• Pure replay can't undo from the parameters alone: working out which vertices a grab moved needs the shape before the stroke, which is what undo is trying to restore. So it has to rebuild from the start (cost grows with history) or from snapshots at 2.2 MB each. It would also force the falloff to be bit-identical across the TypeScript and Rust builds, which stored weights avoid.
• The saved log already is a replay, of cheap integer additions (reload ≈ 100 ms).
• If memory ever got tight: keep the last ~20 steps as changes, and turn older ones into parameters plus one snapshot.

Memory budget for #4, 📏 to be verified on device:

Two caveats:

• ✅ Assumption (stated in the write-up, not asked): the canvas's own colour and depth buffers are not "the shoe." The brief lists vertex data, indices, acceleration structures and undo history. The shoe uses no screen-sized buffers (picking runs on the CPU spatial index, with no ID or picking render target), so its ~77 MB doesn't depend on resolution. For completeness, the write-up notes that a native-resolution canvas adds about 32 MB on the 11" iPad Pro we test on (2388×1668, colour + depth at 4 bytes each) and about 45 MB on a 12.9" (2732×2048), and that we lower the pixel ratio while the shoe moves, as the brief allows.
• The GLB's 17 MB load buffer is released after parsing. A test asserts that nothing keeps holding it.

4. Save and sync

✅ The log is append-only. It holds three kinds of operation: add(edit), undo(editId) and redo(editId).

• Each operation is written to IndexedDB in its own transaction when the pen lifts.
• A save sends only new operations, gzipped, about 150 KB each. It never sends the mesh.

✅ Server sync: built and deployed. One Cloudflare Worker serves the built app as static assets and runs the sync API, with one SQLite-backed Durable Object per shoe holding the ordered log ops(version, op_id, bytes) (worker/). One Worker rather than Pages plus a separate Durable Object Worker: one deploy, one live version, same origin.

• A Durable Object processes one request at a time per shoe, so assigning versions needs no locks. Inserts happen with no await in between, so they commit together.
• Wire format: binary, not JSON (base64 would add a third to ~150 KB ops). A request is repeated [u32 length][op bytes]; a response adds u32 version to each record (src/sync/framing.ts, shared by client and Worker).
• POST /shoes/:id/ops?since=N (gzipped) stores ops whose id is new and skips ids it has seen, so a retry is safe. The response holds everyone else's ops after N, never the caller's own (no echo of ~150 KB per save), plus X-Shoe-Head and X-Shoe-Next. The client pages with GET ?since=<Next> until Next reaches Head.
• The server validates each op exactly (length per kind, mask bits = weight count, no bits past the vertex count), because one stored bad op would break that shoe for every client, forever.
• Client (src/sync/client.ts): an outbox store in IndexedDB, written in the same transaction as the log. One server response is recorded atomically: remote ops join the log, acknowledged outbox entries go, the cursor moves. A Web Lock per shoe stops two tabs syncing at once. Syncs on load (at most 3 s before opening from the local log), after every saved op, every 30 s and on online, backing off up to 5 min on failure. The HUD says up to date only when the outbox is empty.
• Ops from other devices take effect on the next load, not live: an undo from another device can name an edit this tab never held, and keeping every remote edit in memory would break the memory budget. The HUD counts ops waiting for a reload. Live application is future work.
• Replay matches undo and redo by edit id, not stack position. With ops from two devices in one log, the edit an undo names isn't always the newest one (B's log can read add e2, add e3, undo e2).
• No authentication: an unguessable shoe id (a UUID; opening the app without one starts a new shoe at its own URL) is the only key. Stated as a prototype limitation.
• /demo copies a pre-edited demo shoe's ops into a fresh shoe for each visitor, so reviewers never edit the shared one.
• Deploying uses a Cloudflare API token limited to Workers Scripts Edit, stored in 1Password and read through op run (scripts/deploy.sh).
• Known gap: if a shoe's Durable Object were ever wiped, a client whose cursor is past the server's head wouldn't notice and re-send. The fix (treat head < since as a reset and re-post the whole log; dedup makes that safe) isn't built.
• This maps directly onto the Liveblocks room that Ghost already uses.

The same shoe in two tabs: both append to the shared IndexedDB log and outbox, one syncs at a time (Web Lock), and each sees the other's edits on reload. Live updates over BroadcastChannel were planned and not built. An edit computed in one tab was weighted against geometry the other tab hadn't seen yet; that's acceptable for a grab, because edits commute.

5. When things fail

6. Budgets and tests

The tests are the spec. HILOS reads them first. ✅ A few sharp tests, one per guarantee:

1. The sole stays put (#6): after 50 random strokes, including some at the seam, the sole's positions and indices are byte-identical to the GLB. So are the upper's 3,161 seam vertices, and the minimum y stays at 0.
2. Undo is exact: 50 random strokes, then undo all of them, gives upper positions bit-identical to the loaded shoe. A redo/undo round trip is also identical.
3. Memory budget (#4): memoryReport() sums every retained buffer after 100 strokes at 80 mm, which must stay at or under 96 MB minus the framebuffer allowance. Cross-checked by hand with the Safari Web Inspector on an iPad.
4. Reload loses nothing (#5): run strokes, kill the in-memory model, rebuild from IndexedDB (fake-indexeddb), and positions are identical. A stroke in progress at the kill is the only thing missing.
5. Saves never resend the mesh (#5): the bytes saved per stroke are far below the mesh size, and the saved payload never contains the base positions.
6. Context loss (#5, end-to-end): in Playwright, start a stroke, lose and restore the context, finish the stroke; the pixels and positions are correct and editing continues.
7. Frame budget (#2): a benchmark of the per-frame update at 60 and 80 mm stays at or under 10 ms on CI. Real numbers come from a desktop and an iPad, using the pointer event's timeStamp to time the work, reported in an on-screen performance HUD.
8. Picking after an edit (#3): a raycast at a moved point hits the new surface, not the old one.
9. No tears: the weight's slope along every edge stays bounded, so thin details don't tear.
10. Sync loses nothing (#5, across devices): against the real Worker logic over node:sqlite, two devices converge with an interleaved undo, offline ops survive a reload and go out later, a lost response stores nothing twice, and a /demo copy leaves the demo alone. e2e/10 repeats the offline case and /demo in Chromium against wrangler dev.
11. Cross-build fixture: recorded op logs and the shape hash after each op (tests/fixtures/cross-build/), so the Rust/wasm build can prove it computes bit-identical positions.

Per-frame work for #2: the first plan updated about 62k vertices and their normals on the CPU every frame (estimated 2–5 ms). Measured, it took 17–25 ms, so the plan B became the build: the displacement runs in the vertex shader during a stroke, and the spatial index is refit only for the dirty nodes. The measured breakdown is in "Frame budget" below.

7. Bigger meshes (10×: 7M faces)

Not built. How I'd approach it:

• Keep rendering and the pen loop in the browser. Streaming video from a GPU server can't meet "next frame at 60 fps" under a pen; network round-trip time kills the feel.
• Stream triangles: a full-resolution mesh broken into clusters with levels of detail, kept at full resolution near the brush, with the heavy kernels in a WebAssembly worker or WebGPU compute.
• The server keeps the canonical full mesh and replays the same small edit operations.
• That's where WebAssembly earns its place; at today's size, plain TypeScript fits the budget.

Frame budget: how each frame stays under 10 ms (measured, desktop Node, densest 80 mm spot)

✅ Decided and built:

• Plan A (updating everything on the CPU every frame) is dropped: 17–25 ms per frame here, and 11.7 ms even in native Rust (wasm agent).
• Partial refit: each triangle maps to its leaf and each node to its parent, built once with shapecast. A refit marks the dirty leaves plus their ancestors and calls refit(nodeIndices). Each node is marked once, avoiding the duplicate-list trap the wasm agent hit.
• Picks are correct again after the positions and refit phases, a few frames after release. The model's pickable flag says when.
• A new press while settling finishes the remaining work at once: one long frame, only if strokes are less than about half a second apart. 📏 Measure how often that happens on the iPad.
• Undo and redo use the same preview: weights × ±D are shown on the GPU at once, and normals catch up as it settles.
• Memory, worst case (every undo step an 80 mm grab at the densest spot): 92.0 of 96 MB (GPU 20.4, CPU ~39.6, picking index ~7.5, undo 24.8). The picking index uses leaves of 24 triangles (5.7 MB vs 9.4 MB, picks still in the microseconds). Only 4 MB of headroom. If it's needed: drop the CPU copy of the sole normals (2.3 MB), or store weights as u8 (−12 MB at worst).

Threads and GPU

✅ Background threads (workers):

• Loading and one-time setup: parsing the GLB (Ghost does this too), finding the seam, grouping vertices that share a position, the vertex-to-triangle table, and building the picking index (three-mesh-bvh has worker builders).
• Persistence and sync: encoding, gzip, IndexedDB writes, server requests, and replay on reload.
• Maybe: updating the picking index after a stroke. It needs shared memory, which requires the COOP/COEP headers (set through the Worker's _headers file). Only if the measured update time is over budget.
• Not the per-frame drag, and not rendering. Pointer events arrive on the main thread, so a worker adds a hop and timing jitter, and React is idle during a stroke.

✅ GPU:

• Built: displace in the vertex shader during a stroke (weights uploaded once at pen-down, one vector per frame). Works on WebGL2; the CPU arrays get the real update when the pen lifts. Updating positions and normals on the CPU every frame was the first plan and measured 17–25 ms, so it was dropped.
• The main build stays on WebGL2. Ghost uses it, and HILOS's forced context loss most likely uses the WEBGL_lose_context extension; WebGPU loses its device differently. WebGPU is explored on the wasm branch (wgpu).

Stack and structure

✅ Vite, React, TypeScript, React Three Fiber and drei (their stack), plus three-mesh-bvh, Vitest, fake-indexeddb and Playwright.

src/core/      ShoeModel, edit records, falloff, seam   (no React, no three renderer)
src/persist/   IndexedDB op log + outbox, replay by edit id
src/sync/      wire framing, sync client
src/view/      R3F scene, brush cursor, input, perf HUD
worker/        sync Worker + one Durable Object per shoe
tests/         the tests above (e2e/ for the browser ones)

✅ Hosting: one Cloudflare Worker (static assets + sync API) at hilos.simmer.io (custom domain; the workers.dev URL also works), with X-Robots-Tag: noindex. The GLB is served from the same origin, never from a public repo. The Rust/wasm web build is going on its own static Worker at wasm.simmer.io.

Questions for Beichen

None sent. Instead, the write-up states our assumptions:

• #4: the canvas framebuffer is not counted as "the shoe" (see §3).
• #2: latency is measured from the pointer event's timeStamp to the requestAnimationFrame callback that submits the draw, because browsers don't expose when a frame is actually presented.
• #5: has a live server (see §4).