Photo Sidecars with a Vision Model

The script vision.ts of my old site: EXIF data and a vision model's alt text, titles and tags in a JSON file next to each photo, generated locally before the deploy.

Article Created Updated Also in German: Foto-Sidecars mit einem Vision-Modell

Reviewed by adrian on

The photo stream of my old Astro site needed alt text, titles and camera data for every picture. The script scripts/vision.ts wrote them into a JSON sidecar next to each JPG. This page collects three posts from March 2026: where the script runs, how it gets through rate limits, and its move from OpenAI to Claude.

What the script does

For every JPG without a .json sidecar it:

  1. reads the EXIF data with exiftool (camera, lens, aperture, ISO, focal length, shutter speed, GPS);
  2. sends the image to a vision model for alt text, title ideas and tags;
  3. writes both into a JSON file next to the image.

The sidecar drives the photo pages: alt text for accessibility, titles for the detail page, EXIF for the metadata panel.

Run it where the originals are

At first the VPS generated the sidecars itself. The deploy worked, but no sidecars arrived. The script read the photos in the repository, while the deploy synced a different folder: the photo library in my Obsidian vault on the Mac. And rsync cannot upload files that were never created in the folder it syncs.

The fix was to treat the VPS as a target only. The Mac has the originals, the API key and exiftool, so it generates the sidecars, and the server receives finished files. vision.ts got an optional source folder, and the deploy script runs it first:

pnpm run vision -- "$PHOTO_SRC"

rsync -az --delete \
  --include='*/' \
  --include='*.md' --include='*.mdx' \
  --include='*.jpg' --include='*.jpeg' --include='*.JPG' --include='*.JPEG' \
  --include='*.json' \
  --exclude='.DS_Store' --exclude='*' \
  "$PHOTO_SRC" "$VPS:$REMOTE_PHOTOS/"

After that the server pulls the repository and rebuilds the container with podman-compose. The VPS no longer needs an API key, exiftool or Node for metadata.1

Rate limits on large batches

The first run over a whole trip's photos died on 429 Too Many Requests, with $3.16 of credit left. A positive balance does not help against the tokens-per-minute limit of the account. The script already had bounded concurrency and retries with exponential backoff, so the job was to choose values that keep it moving:

pnpm vision -- --concurrency=1 --retries=10 --backoff-ms=2000

Start there, watch the log for 429 retries, and go to --concurrency=2 only after a clean run. Above 2, on an account without a higher limit tier, the script spends more time in backoff than it saves. For more throughput, raise the tier. Still open at the time: downscaling the images before upload (1024 px on the long edge is enough to describe a scene), which saves tokens and time.2

From OpenAI to Claude

The move to the Claude API mostly changed the request; EXIF, concurrency, batching and the command-line flags stayed the same.

  • Package and key: @anthropic-ai/sdk instead of openai, ANTHROPIC_API_KEY instead of OPENAI_API_KEY.
  • Client: new Anthropic({ maxRetries: 0 }), because the script has its own backoff loop and two retry layers waste quota.
  • Structured output: OpenAI used response_format with a json_schema and a JSON.parse afterwards. Claude gets a tool vision_response with an input_schema (title_ideas, description, tags) and tool_choice: { type: "tool", name: "vision_response" }. The answer arrives as toolUseBlock.input, an object, without parsing.
  • Image block: OpenAI takes a data URL in image_url; Anthropic takes a source block with type: "base64" and its own media_type.
  • Rate limits: instead of checking a raw status and parsing the error text, error instanceof Anthropic.RateLimitError, with the wait time from the retry-after header of the exception.
function extractRetryAfterMs(error: unknown): number | null {
  if (!(error instanceof Anthropic.RateLimitError)) return null;
  const retryAfter = error.headers?.get("retry-after");
  if (retryAfter) {
    const seconds = Number.parseFloat(retryAfter);
    if (Number.isFinite(seconds) && seconds > 0) return Math.ceil(seconds * 1000);
  }
  return null;
}

The model was claude-opus-4-6. For a batch that runs before a deploy, quality mattered more than speed.3

In current versions of the SDK, error.headers is a fetch Headers object, so the value has to be read with .get("retry-after"). An index access like headers["retry-after"] returns undefined, and the script then falls back to its own backoff without notice.4

Takeaways

  • Preprocessing has to run against the folder the deploy syncs, not a copy in the repository.
  • Work that needs keys and tools belongs on the machine with the originals; the server only needs what it serves.
  • A 429 is usually the throughput limit, not the balance.

  1. Generating Photo Sidecars Locally Before a VPS Deploy, 14 March 2026. ↩

  2. Vision Rate-Limit Runbook for Photo Sidecars, 15 March 2026. ↩

  3. Migrating the Vision Script from OpenAI to Claude, 19 March 2026. ↩

  4. Anthropic TypeScript SDK, APIError in src/core/error.ts, as of November 2026. ↩

Based on

  • Generating Photo Sidecars Locally Before a VPS Deploy (raw/2026/2026-03-14/generating-photo-sidecars-locally-before-a-vps-deploy.md) [local]
  • Vision Rate-Limit Runbook for Photo Sidecars (raw/2026/2026-03-15/vision-rate-limit-runbook-for-photo-sidecars.md) [runbook]
  • Migrating the Vision Script from OpenAI to Claude (raw/2026/2026-03-19/migrating-the-vision-script-from-openai-to-claude.md) [claude]
  • Anthropic TypeScript SDK, error.ts [sdk]

More on this

Topic

  • AI / Language Models What I use language models for: agents, local models, RAG and small programs in which a model handles one task.

Uses

  • Claude API Anthropic's programming interface to the Claude models. Billed separately from a claude.ai subscription, with keys from the Claude Console.

Mentioned in

  • Astro Justified Gallery Layout My npm package @altner/astro-justified-gallery-layout: a small justified gallery for Astro with EXIF, LQIP previews, a virtual variant and a lightbox.
  • Building the Photo Stream The photo stream of my old Astro site: JSON sidecars next to each JPG, import.meta.glob, a justified grid and batches revealed on scroll.
  • My Old Astro Website My old site at adrian-altner.de: Astro with the Node adapter, typed content collections, IndieWeb features and photos as files with JSON sidecars.
  • Syndicating Photos to Flickr A script that makes Flickr match the photo tree of my old site: upload, delete, move between albums, and write the Flickr ID and stats back into the sidecars.

Show in the graph view