Visitor Statistics from the Server Log with Matomo

A self-hosted Matomo fed hourly from Caddy's access log: no tracking script, no cookies, masked IP addresses.

I wanted to know which pages of my website are read and how people find them, without a tracking script and without a cookie banner. The web server logs every request anyway. So a script hands the page views from Caddy's log to my own Matomo once an hour, and Matomo turns them into reports. The pattern comes from another site of mine on the same server, which has worked like this for a while.

How it fits together

  • Caddy writes a JSON access log, rolled daily and kept for 7 days.
  • At minute 55 of every hour, cron runs a small Python script (standard library only). It reads the new lines, keeps the page views and passes them to Matomo's own importer import_logs.py.
  • At minute 5, Matomo's archiving builds the reports.
  • Once a week, the search report reads the numbers through Matomo's API.

What counts as a page view

Every request is in the log: images, scripts, htmx swaps, bots. The script keeps a line only if all of this holds:

  • GET, status 200, content type text/html, and not under /static/, /health or /api/.
  • A browser navigation: Sec-Fetch-Mode: navigate, Sec-Fetch-Dest: document and a Sec-Fetch-Site header. The last one keeps out the iframes of my small web apps and headless crawlers.
  • No HX-Request header, so htmx swaps inside a page do not count.
  • No DNT: 1 and no Sec-GPC: 1. Whoever switches on Do Not Track or Global Privacy Control is not passed on.
  • The visitor's address is not in a list of data-centre networks whose crawlers pose as browsers.

The Sec-Fetch-Site rule came from the first dry run. It counted 168 page views in one day, and 120 of them came from a single visitor. That visitor used an old Chrome on Linux, never sent a referrer, and sent Sec-Fetch-Mode without Sec-Fetch-Site. Real browsers send these headers together. With the rule, 47 page views were left.

What the script changes before the hand-over

  • The real IP behind Cloudflare: the site runs through Cloudflare's proxy, so Caddy sees a Cloudflare address. If the request came from a Cloudflare network, the script uses CF-Connecting-IP instead. The website's own rate limiter follows the same rule.
  • No query string: the script cuts it from the URL, so search terms from the site search never reach Matomo.

Reading the log reliably

  • The script remembers the inode and the byte offset of the log in a small state file. When the log has been rolled since the last run, it first reads the old file to the end, found by its inode.
  • It reads only complete lines and moves the offset only after a successful import. A failed run is repeated by the next one.
  • Caddy compresses rolled files by default. Then the old file cannot be finished, so the log block needs roll_uncompressed.
  • It copies import_logs.py out of the Matomo container on every run (docker cp), so the importer always matches the installed version.
  • It calls the importer with --idsite, --log-format-name=caddy_json, --recorders=2 and --auth-config=<file>. The token sits in that file ([auth], token_auth = …, mode 600), not on the command line, where any user could see it in the process list.
  • --dry-run only counts. I ran it a few times before the first real import.

Setting up Matomo

  1. Add the site under Administration, Websites, Manage. The UI did not show the new site's ID, so I read it from the database.
  2. Two users with one token each. One user has write access to this site only, for the import. The other has view access only, for the weekly report. Since Matomo 5 a user comes in by invitation, and you can copy the invite link from the user list. Then sign in as that user in a private window, go to Personal, Security, and create a token with "Only allow secure requests".
  3. Check the access rights. In my case both users also had access to the other site at first, and I removed it.
  4. Privacy (Administration, Privacy, Anonymize data). These settings apply to every site in the instance: - Mask IP addresses by 2 bytes. - "Also use the Anonymized IP addresses when enriching visits": Yes. Otherwise Matomo locates visitors from the full IP, down to the city if a city database is installed. - Referrer: remove query parameters. - Delete old raw data after 90 days. I left the deletion of aggregated reports off, because they hold only counts.
  5. Save each section separately. Matomo asks for the password, and before that nothing is stored. I checked the stored options in the database afterwards.

The importer's help text says a token needs admin access, or visitor IPs will be wrong. Write access was enough here: after the first import, Matomo had stored the visitors' masked addresses, not the server's.

When the Matomo login is refused

My Matomo only accepts logins from my VPN network. The VPN on my Mac only routes that network, so the browser went over the public internet and Matomo refused it. The fix was two lines in /etc/hosts on the Mac. The first points Matomo's hostname to the server's VPN address. The second points it to ::1 for IPv6, because the browser preferred the public IPv6 address from DNS. Since nothing listens on port 443 locally, the browser falls back to IPv4 through the tunnel. Matomo now works only while the tunnel is up.

Privacy statement

The statistics belong in the privacy statement. Mine now says, in the section on server log files:

  • what goes to Matomo, and that it shortens the IP by two bytes;
  • that pages arrive without search terms;
  • that Matomo derives the approximate country, browser and device;
  • that Do Not Track and Global Privacy Control are respected, and that visitors can object by email;
  • that individual visits are deleted after 90 days;
  • the legal basis, legitimate interest under Art. 6(1)(f) GDPR.

The script's filter and the Matomo settings have to stay in line with that text.

Pitfalls

  • A date range asked for too early stays wrong. I queried a range before its days were archived and got zero visits. The same range asked for after archiving was correct. The weekly report runs on Monday morning, when the days are long archived.
  • Tokens expire. Mine run for 180 days. After that the import stops until a new token is in the file. Matomo sends a reminder 30 days before, if it can send email.

Result

The first real import brought 46 page views, 9 visits from three countries, all with masked IP addresses.

More on this

Topic
  • Self-Hosting Running services on a server you control instead of using a provider's hosted version, with the upkeep that comes with it.
Uses
  • Caddy Web server and reverse proxy that obtains and renews TLS certificates on its own, configured in a short Caddyfile.
  • Matomo Open-source web analytics that runs on your own server and can import web server logs instead of using a tracking script.
  • Python General-purpose programming language with highly readable syntax. The standard for data analysis, automation and quick prototypes.
See also

Show in the graph view →