What the machine sees was a small app in the playground of my old website. You chose a photo, and a vision model marked what it recognised: objects with outlines, text in the image and a short description of the scene. Everything ran in the browser tab, and the photo was never uploaded. The playground did not move to the new site, so the app is offline; the model itself is on Hugging Face.
I came across it through a demo on GitHub that runs Mirage by Cactus Compute in the browser. The demo has no licence, so I wrote my own interface and only took the parts that are under the Apache License 2.0: the WebAssembly build of the cactus engine and the model file.
Setup
- The engine (
cactus.js,cactus.wasm) and the model (mirage.cact, 21.6 MB) lay on my own server. The first visit loaded about 23 MB, after that the browser cache had them. - The photo is read with
createImageBitmap, drawn into a canvas, scaled to at most 1024 pixels on the long edge and handed to the engine as RGB pixels. - The engine answers with JSON: objects with name, confidence, box and outline, text found in the image, and the scene. The model answers in English.
- The only network requests are downloads: the page, the scripts, the engine and the model. I checked that in the browser's network log.
What it did
The photo always fitted into the stage, whatever its format. Describing started as soon as a photo was chosen; a test photo took 0.9 seconds and gave 20 objects. Pointing at something in the photo or in the list highlighted it and dimmed the rest. A click pinned it and showed a card with a crop from the photo and everything the model said about it. Arrow keys stepped through the findings, a slider hid those below a confidence threshold, and a photo could also be dropped onto the stage or pasted.
Problems on the way
- The demo's
cactus.jsdid not run at all: a semicolon was missing between two statements. I found it by bisecting the file and patched it, with a note in the folder. - My site's Content Security Policy blocks
blob:images, so the photo cannot simply go into an<img>. A canvas works. - The first version was a long page on which tall photos ran out of the window. The second one is laid out like an app, with toolbar, stage and sidebar.
- Topic
-
- On-Device AI AI models that run on the user's own device, in an app or in the browser, so the input stays there and no server does the work.
- Uses
- Mentioned
-
- Cactus Compute Company behind cactus, an engine for running small AI models on consumer devices, and models such as the vision model Mirage.
- Tags