Caching Webmention Avatars Locally

An Astro helper that downloads webmention avatars during the build, dedupes and sweeps them, so the CSP stays strict and readers never hit the avatar host.

Article Created Updated Also in German: Webmention-Avatare lokal cachen

Reviewed by adrian on

After the security headers went live with a strict Content Security Policy and img-src 'self' data:, the next scan showed a new problem: the mention photos on my post pages were gone. Opening img-src to 'self' data: https: would have treated the symptom. Instead I cached the avatars at build time, so the CSP stayed strict and readers never sent a request to the avatar host.

The problem

My Webmentions.astro component (Receiving and sending webmentions) rendered the facepile and the replies with avatar URLs straight from the webmention.io API:

<img
    src="https://avatars.webmention.io/files.mastodon.social/0ec46...jpg"
    alt=""
    loading="lazy"
/>

That caused three problems:

  1. CSP. img-src 'self' data: blocks external images, and adding https: would make the directive almost useless.
  2. Reader privacy. Every visit to a page with mentions made the browser call avatars.webmention.io. The host saw IP address, user agent, referer and the path of the post. Under GDPR Art. 5.1.c (data minimisation) that is an unnecessary disclosure of personal data.
  3. Availability. When webmention.io was down or slow, avatars were missing or held up rendering.

The idea

The mentions were already fetched at build time, so every avatar URL was known while the page was rendered. The images could be downloaded in the same pass, stored locally and the HTML pointed at the local path. That needs four pieces:

  • a cache folder under public/ that survives between builds;
  • a mirror into dist/, because Astro copies public/ to dist/ before rendering, so files added to public/ later would be missing from the current build;
  • deduplication, so one author who liked and reposted the same page is downloaded once;
  • a sweep at the end of the build that removes avatars nobody links to any more.

The helper

src/lib/avatar-cache.ts:

import { createHash } from 'node:crypto';
import { existsSync } from 'node:fs';
import { copyFile, mkdir, readdir, unlink, writeFile } from 'node:fs/promises';
import path from 'node:path';

const PUBLIC_DIR = path.resolve(
    process.cwd(),
    'public',
    'images',
    'webmention',
);
const DIST_DIR = path.resolve(process.cwd(), 'dist', 'images', 'webmention');
const URL_PATH = '/images/webmention';

const EXT_BY_MIME: Record<string, string> = {
    'image/jpeg': 'jpg',
    'image/png': 'png',
    'image/webp': 'webp',
    'image/gif': 'gif',
    'image/avif': 'avif',
    'image/svg+xml': 'svg',
};

const memo = new Map<string, Promise<string | null>>();
const usedFilenames = new Set<string>();

async function mirrorToDist(filename: string, srcPath: string) {
    try {
        await mkdir(DIST_DIR, { recursive: true });
        const dest = path.join(DIST_DIR, filename);
        if (!existsSync(dest)) await copyFile(srcPath, dest);
    } catch {
        // dist/ doesn't exist during `astro dev`, ignore
    }
}

async function download(url: string): Promise<string | null> {
    const hash = createHash('sha1').update(url).digest('hex').slice(0, 16);
    await mkdir(PUBLIC_DIR, { recursive: true });

    try {
        const res = await fetch(url, { signal: AbortSignal.timeout(10_000) });
        if (!res.ok) return null;
        const mime = (res.headers.get('content-type') ?? '')
            .split(';')[0]
            .trim()
            .toLowerCase();
        const ext = EXT_BY_MIME[mime] ?? 'jpg';
        const filename = `${hash}.${ext}`;
        const filepath = path.join(PUBLIC_DIR, filename);
        if (!existsSync(filepath)) {
            const buf = Buffer.from(await res.arrayBuffer());
            await writeFile(filepath, buf);
        }
        usedFilenames.add(filename);
        await mirrorToDist(filename, filepath);
        return `${URL_PATH}/${filename}`;
    } catch {
        return null;
    }
}

export function cacheAvatar(url: string | undefined): Promise<string | null> {
    if (!url) return Promise.resolve(null);
    if (!/^https?:\/\//i.test(url)) return Promise.resolve(url);
    let pending = memo.get(url);
    if (!pending) {
        pending = download(url);
        memo.set(url, pending);
    }
    return pending;
}

async function sweepDir(dir: string): Promise<number> {
    if (!existsSync(dir)) return 0;
    const entries = await readdir(dir);
    let removed = 0;
    await Promise.all(
        entries.map(async (name) => {
            if (!usedFilenames.has(name)) {
                await unlink(path.join(dir, name));
                removed++;
            }
        }),
    );
    return removed;
}

export async function sweepCache(): Promise<{ removed: number }> {
    const publicRemoved = await sweepDir(PUBLIC_DIR);
    await sweepDir(DIST_DIR);
    return { removed: publicRemoved };
}

Four details that are easy to miss:

  • The SHA1 of the URL is the file name. The same avatar always gets the same name, whatever build it appears in.
  • The extension comes from the MIME type in the response header. webmention.io serves some avatars behind URLs without an extension.
  • The memo map prevents duplicate downloads within one build. The component calls cacheAvatar through Promise.all, so an author who liked and reposted would otherwise be fetched twice in parallel.
  • usedFilenames plus sweepCache make it a mark-and-sweep cache. Every hit marks a file as used, and at the end the sweep deletes everything unmarked, such as the avatar of a like that was taken back.

In Webmentions.astro the cache step ran right after fetchMentions:

---
import { cacheAvatar } from '~/lib/avatar-cache';
// ...
const all = await fetchMentions(targetStr);

await Promise.all(
    all.map(async (m) => {
        if (m.author?.photo) {
            const local = await cacheAvatar(m.author.photo);
            if (local) m.author.photo = local;
            else delete m.author.photo;
        }
    }),
);
---

If a download failed (timeout, 404), author.photo was removed and the component fell back to an avatar with the author's initials.

The sweep as an Astro integration

sweepCache may only run after every page has been rendered, or it would delete avatars a later page still needs. Astro's astro:build:done hook is that moment. src/integrations/avatar-cache-sweep.ts:

import type { AstroIntegration } from 'astro';
import { sweepCache } from '../lib/avatar-cache';

export default function avatarCacheSweep(): AstroIntegration {
    return {
        name: 'avatar-cache-sweep',
        hooks: {
            'astro:build:done': async ({ logger }) => {
                const { removed } = await sweepCache();
                if (removed > 0) {
                    logger.info(
                        `swept ${removed} orphaned webmention avatar${removed === 1 ? '' : 's'}`,
                    );
                }
            },
        },
    };
}

Registered in astro.config.mjs:

import avatarCacheSweep from './src/integrations/avatar-cache-sweep';

export default defineConfig({
    integrations: [
        mdx(),
        sitemap({
            /* ... */
        }),
        avatarCacheSweep(),
    ],
});

astro:build:done never fires in astro dev, so the sweep only runs on astro build. A dev server cannot evict an avatar that the next production build still needs.

Why /images/webmention/

My first attempt used /webmention-avatars/ at the site root. That worked, but it missed the cache rule I already had in Caddy:

header /images/* Cache-Control "public, max-age=604800"

Under /images/webmention/ the avatars got a week of browser caching with no extra configuration.

The cache folder does not belong in the repository. It fills itself during a build when empty and otherwise stays from the previous run, so .gitignore gets:

public/images/webmention/

On a CI without a persistent cache the avatars are fetched again on every build. They are small JPEGs, so that does not matter.

The result

The built HTML now pointed at the site itself:

<img src="/images/webmention/556ae159c81f2fc1.jpg" alt="" loading="lazy" />
  • The CSP stayed strict. img-src 'self' data: was enough.
  • No third-party requests on page load. Readers fetched avatars from my server, and the mention host never saw them.
  • Availability was decoupled. The avatars depended only on a successful build, not on webmention.io being reachable when someone read the page.
  • The cache cleaned itself. Retracted likes and deleted comments took their avatar with them on the next build.

Takeaways

  • External resources known at build time belong in a local cache. The argument people make for self-hosting Google Fonts applies to avatars, icons and badges as well.
  • CSP and privacy pointed to the same fix. Once the avatars no longer load from outside, the CSP need not be relaxed and readers' IP addresses stay with me.
  • A cache without garbage collection keeps data it should drop. Mark-and-sweep takes a few lines.

Based on

  • Caching webmention avatars locally at build time (raw/2026/2026-04-22/caching-webmention-avatars-locally-at-build-time.md) [post]

More on this

Topic

  • IndieWeb Publishing on your own domain first and connecting personal websites with open standards such as Webmention and Microformats.

Builds on

  • Receiving and Sending Webmentions Incoming webmentions through webmention.io, rendered at build time in Astro, and outgoing ones sent by webmention.app after every deploy.
  • Security Headers in Caddy A strict Content Security Policy, HSTS and no-referrer in a Caddy site block, checked with Webbkoll, securityheaders.com and the Mozilla Observatory.

Uses

  • Astro Tool for building websites from content such as Markdown. Ships plain HTML by default and only loads JavaScript where it is needed.
  • Caddy Web server and reverse proxy that obtains and renews TLS certificates on its own, configured in a short Caddyfile.

Show in the graph view