🥽 Companion Scan

Scan a room with your phone, then explore it in 3D. Walk a space with your phone’s camera, and Companion Scan builds a 3D map that you can open, label, and measure in a desktop editor, all stored on your own Hub.

Install & availability

Available now, as a beta. Install from the Companion Hub App Store. The store lists it as Companion Scan.

Overview

Companion Scan is a browser-based AR room scanner and a desktop world editor. On the phone it uses the free 8th Wall  SLAM engine binary, with a native WebXR fallback, to track the space and build a live coloured point cloud while a vision-language model labels the objects it sees. On the desktop, the editor places one or more scans in a shared, metric, geo-anchored 3D world that you can inspect, align, label, measure, and share by link.

Scan files, camera frames, and detections go only to your Hub and the Ollama it uses. See What leaves your network for the few third-party requests the pages make.

What a world is

A world is a metric frame, in meters, anchored to a GPS origin. It holds one or more scans, and each scan is split into typed layers: point cloud, meshes, floors, detections, labels, and the camera’s path. The fourth dimension is time: every frame is timestamped, and edits are an append-only history, so one world can hold several scans of the same place captured at different moments. The editor opens the world named default unless you ask for another with ?world=<name>.

Key features

• Web AR scanning — 8th Wall SLAM tracks the room and builds a live point cloud. A native WebXR scanner at /webxr.html runs on browsers that support it.
• Object detection — each capture sends a frame to your Hub, where an Ollama vision model returns labelled boxes. The labels become 3D pins at their positions in the room.
• Scan quality guidance — a coverage bar, warnings for fast movement, low light, and reflective surfaces, and a verdict chip (Ready, Review, or Sparse). Done opens the export panel, and export asks you to confirm when the scan is sparse.
• Three ways to capture — tap to capture, auto-capture as you walk, or keyframe streaming that sends posed frames to the Hub for reconstruction jobs.
• Live view on the desktop — Start Live Stream shows the scan growing in the editor in real time.
• Autosave and history — the first capture uploads silently. Sessions stay on the phone, and the history drawer (Scan Memory) lists them with a map of where you scanned.
• Desktop world editor — scans in one shared frame, with render modes, layer toggles, a measure tool, a map, semantic tags, notes, and move, rotate, and scale controls.
• Server processing — from the phone’s panel, ask the Hub for a floor plan, a scene description, or an object report. Uploads are compressed and get lower-detail previews that the editor loads first.
• Share links by role — owner, editor, viewer, and scanner links for a world.
• Installable and offline-tolerant — a PWA with an offline queue for uploads.

The scanning experience

On your phone: scan a room

Open the app’s HTTPS address, allow camera and motion access, choose Scan a room, and sweep the space slowly. The HUD in the top-left corner shows tracking status, the live point count, the coverage bar, and the verdict chip. The buttons on the right open the desktop editor, the Review and Export panel, keyframe streaming, auto-capture, and the history drawer.

Tap the capture button to identify objects. Each detection appears as a card and as a 3D pin in the scene. When you’re done, select Done to review and export.

On the desktop: structure the world

Open the editor at /viewer.html?world=default on your app’s address, or tap the desktop button on the phone’s action rail. The Scans tab lists every scan in the world. Select one to focus the view on it.

The World tab shows the world’s totals, the share links, and the processing queue.

How it works

From a phone sweep to a shareable world, the data moves through these stages:

flowchart TD
    PH["Phone: 8th Wall SLAM or WebXR<br/>pose and feature points"] -->|"point cloud<br/>plus a JPEG per capture"| DET["Detection on your Hub<br/>Ollama vision model"]
    DET -->|"boxes placed in 3D"| PINS["Labelled 3D pins"]
    PINS -->|"autosave / Export GLB"| HUB["Hub storage<br/>scans · worlds · summaries"]
    HUB -->|"compress and make previews"| ED["World editor<br/>metric frame · layers · measure · tags"]
    ED -->|"queue jobs"| JOBS["Processing jobs<br/>splats and dense reconstruction need GPU tools"]
    JOBS -->|"artifacts attach to the scan"| ED
    ED -->|"role links"| SHARE["Share links<br/>owner · editor · viewer · scanner"]

Reconstruction methods

Setup

ollama pull moondream:latest

Usage

Viewing a completed scan

Orbit, pan, and zoom the scene. The render buttons along the top switch between Solid, Points, Wire, and Semantic (recoloured by tag). The Layers tab toggles each layer on its own: the point cloud, meshes, floors, detections, labels, and camera path. Map pins every GPS-tagged scan on a map.

Measuring

Select Measure, then click two points. Because the world is metric, the readout is in real meters.

Tagging and editing

Select a scan in the Scans tab to open the Edit tab. There you can move, rotate, or scale the scan, assign a semantic tag, add notes, or hide it. Every edit is saved as an append-only patch.

Sharing

The World tab has owner, editor, viewer, and scanner links. Each carries a token in its address. The store build doesn’t enforce those tokens by default, so treat the app’s address itself as the thing to protect.

Exporting

From the phone’s Review and Export panel, Export GLB saves the scan as a .glb file and uploads it to your Hub, and Export Summary saves a summary of the scan.

What leaves your network

Scan files, camera frames, and detections go to your Hub and, for labelling, to the Ollama it uses. The pages themselves make a few third-party requests:

• Fonts load from Google Fonts, and the map library loads from unpkg.com.
• The maps draw tiles from the CARTO basemap, which uses OpenStreetMap data.
• When location access is on, the phone asks OpenStreetMap’s Nominatim service to turn your coordinates into an address.

Troubleshooting

Tracking never starts, or “Move camera to initialise tracking…” never goes away Camera or motion permission was denied. Re-enable it in your browser or OS settings (iOS: Settings → Safari → Motion & Orientation Access, and camera access for the site) and reload over the HTTPS link.

“HTTPS required” on the start screen Camera and AR APIs only work on a secure origin. Open the app’s HTTPS link, not a plain-HTTP address.

The scan has no labels, or capturing reports an error The app couldn’t use an Ollama vision model. Check <app address>/health as described in Give the app a vision model, pull a vision model, and restart the app, because it checks Ollama only when it starts.

Uploading a video or photos fails This is a known problem in the store build: the upload scripts import a file that isn’t in the image. Use live AR scanning instead.

Splat training or dense reconstruction is skipped These jobs need GPU tools that the store image doesn’t include. The job reports a skipped reason.

The reconstructed model has holes Coverage was too low. Capture more angles of the problem areas and scan again. A world can hold several scans of the same space.

Links

• Companion Spatial
• Photo Time Machine
• Earth
• Companion Memory