What the Machine Sees

A small app from my old website: choose a photo, and the vision model Mirage found objects, text and the scene in it, entirely in the browser.

Article Created Updated Also in German: Was die Maschine sieht

Reviewed by adrian on

What the machine sees was a small app in the playground of my old website. You chose a photo, and a vision model marked what it recognised: objects with outlines, text in the image and a short description of the scene. Everything ran in the browser tab, and the photo was never uploaded. The playground did not move to the new site, so the app is offline; the model itself is on Hugging Face.

I came across it through a demo on GitHub that runs Mirage by Cactus Compute in the browser. The demo has no licence, so I wrote my own interface and only took the parts that are under the Apache License 2.0: the WebAssembly build of the cactus engine and the model file.

Setup

  • The engine (cactus.js, cactus.wasm) and the model (mirage.cact, 21.6 MB) lay on my own server. The first visit loaded about 23 MB, after that the browser cache had them.
  • The photo is read with createImageBitmap, drawn into a canvas, scaled to at most 1024 pixels on the long edge and handed to the engine as RGB pixels.
  • The engine answers with JSON: objects with name, confidence, box and outline, text found in the image, and the scene. The model answers in English.
  • The only network requests are downloads: the page, the scripts, the engine and the model. I checked that in the browser's network log.

What it did

The photo always fitted into the stage, whatever its format. Describing started as soon as a photo was chosen; a test photo took 0.9 seconds and gave 20 objects. Pointing at something in the photo or in the list highlighted it and dimmed the rest. A click pinned it and showed a card with a crop from the photo and everything the model said about it. Arrow keys stepped through the findings, a slider hid those below a confidence threshold, and a photo could also be dropped onto the stage or pasted.

Problems on the way

  • The demo's cactus.js did not run at all: a semicolon was missing between two statements. I found it by bisecting the file and patched it, with a note in the folder.
  • My site's Content Security Policy blocks blob: images, so the photo cannot simply go into an <img>. A canvas works.
  • The first version was a long page on which tall photos ran out of the window. The second one is laid out like an app, with toolbar, stage and sidebar.

Based on

  • Project: Mirage vision model in the browser (raw/2026/2026-11-18/project-mirage-vision-model-in-the-browser.md) [notes]

More on this

Topic

  • On-Device AI AI models that run on the user's own device, in an app or in the browser, so the input stays there and no server does the work.

Uses

  • cactus Open-source engine by Cactus Compute that runs small AI models for text, speech and images locally on consumer devices.
  • Mirage Small vision model by Cactus Compute that finds objects with outlines, text and the scene in a photo and returns them as JSON.

Mentioned

  • Cactus Compute Company behind cactus, an engine for running small AI models on consumer devices, and models such as the vision model Mirage.

Show in the graph view