Web vitals, crawlers and IndexNow

This page covers three parts of analytics: the web vitals from real visitors, the AI and search crawler hits from your access logs, and the IndexNow submit of changed pages.

Web vitals

The tracking script measures the Core Web Vitals of real visitors. It needs no change to your tag and no third-party library.

Metric	What it measures	Good up to	Poor above
LCP	Largest Contentful Paint	2,500 ms	4,000 ms
INP	Interaction to Next Paint	200 ms	500 ms
CLS	Cumulative Layout Shift	0.1	0.25
TTFB	Time to First Byte	800 ms	1,800 ms

The thresholds come from web.dev.

Read the web vitals

ce vitals example.com --period 7d
ce vitals example.com --path /pricing

The answer has the 75th percentile (p75) of each metric for the site and its top pages. Each value has n, the share of good values and the rating: good, needs_improvement or poor.

The MCP tool is web_vitals. The site page shows a Web vitals panel.

Confidence	Reports
High	100 or more
Medium	20 or more
Low	Fewer than 20

Note

The web vitals are not tested for significance. Exit code 4 means that the period has no reports yet.

What the script sends

The script sends one report for each page load, when the page is hidden first. A report has the page URL and the metrics that the browser measured. No report comes from a browser without the APIs, from automation, or from a file: page.

A report keeps no visitor ID and no location. A report is not a page view, so the page view numbers do not change.

A page with bad vitals that loses search position makes a slow_page_losing_position finding.

AI crawlers from access logs

A JavaScript tag cannot see crawlers. To see GPTBot, ClaudeBot and the other crawlers, push the access log of your web server to Checkend.

ce logs push example.com /var/log/nginx/access.log

The token needs write access to the project of the site.

Push the log each day

Run the push from cron on the web server, for example with the log of yesterday:

CHECKEND_URL=https://stats.example.com CHECKEND_TOKEN=ce_... \
  ce logs push example.com /var/log/nginx/access.log.1

For a compressed log, read it from stdin:

zcat access.log.2.gz | ce logs push example.com -

Formats and limits

• Formats: the combined format of Nginx and Apache, and the JSON access log of Caddy.
• At most 10 MB in one push. Split a larger file.
• A line longer than 16 KiB refuses the push.
• Lines older than 31 days, or in the future, are not kept.
• A Caddy line of another host is not kept.
• Crawler hits do not count against the daily event limit.

A second push of the same lines adds nothing to the data. The answer of each push counts the hits that it found and sent, so a second push shows the same count again.

What Checkend keeps

Checkend keeps only the lines of known crawlers: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Bingbot, Applebot, CCBot, Bytespider, Amazonbot, Meta-ExternalAgent and DuckAssistBot.

For each hit, it keeps the UTC day, the path without the query, the bot name and the status. It keeps no IP address and no user agent. It drops the lines of people and other clients.

Note

Checkend does not check the user agent against the IP ranges of the crawler operators. A client can claim to be GPTBot. Google-Extended and Applebot-Extended are robots.txt tokens only: their fetches show as Googlebot and Applebot.

Read the crawler hits

ce crawlers example.com --period 7d
ce crawlers example.com --bot GPTBot

The answer has the hits, the pages, the errors (status 400 or more) and the last day of each crawler, and the top pages. The MCP tool is ai_crawlers. The site page shows an AI crawlers panel.

IndexNow

IndexNow tells Bing and other IndexNow search engines that pages changed. Checkend sends:

• the pages that the crawler finds changed (title, meta description, H1 or canonical URL),
• the page of each agent change on a finding with a page,
• the URLs that you submit.

Turn IndexNow on

Only an admin can turn IndexNow on. There is no MCP action for it, because an agent must not send data to an outside service without the permission of an admin.

1. Turn IndexNow on for the site:

   ce indexnow set example.com --on

   Checkend makes a key of 32 hex characters.

2. Show the status and the key file snippet:

   ce indexnow status example.com

3. Make the key file available at https://example.com/<key>.txt. Use one of the two modes below.

The key file

The search engines read the key file at the root of your site. A key file in a folder covers only the URLs of that folder.

Mode proxy (default). Checkend serves the key at <public URL>/indexnow/<key>.txt. Map /<key>.txt on your site to it. Nginx example:

location = /<key>.txt {
    proxy_pass https://stats.example.com/indexnow/<key>.txt;
    proxy_set_header Host stats.example.com;
    proxy_ssl_server_name on;
}

ce indexnow status also gives snippets for Caddy and Next.js.

Mode manual. Put a file <key>.txt with the key in the web root of your site:

ce indexnow set example.com --on --mode manual

To make a new key, add --new-key.

Submit URLs

ce indexnow submit example.com https://example.com/pricing --reason "new prices"

The MCP tool indexnow can also submit URLs (write access). One submit takes at most 100 URLs, and only URLs on the host of the site. A URL that was submitted in the last 24 hours does not go to the queue again.

The submit job

• The job runs every 10 minutes (--indexnow-every).
• It checks the key file first, and again after 7 days. When the key file is missing, it waits one hour and shows the next step.
• It sends at most 1,000 URLs of a site in 24 hours (--indexnow-max-per-day).
• The default endpoint is https://api.indexnow.org/indexnow. Select another one with --indexnow-endpoint, for example https://www.bing.com/indexnow.

ce indexnow status shows the key check, the last answer, the error with the next step, and the queue. Each submit writes an audit entry.