# Web vitals, crawlers and IndexNow > Read the web vitals of real visitors, see AI crawler hits from your access logs, and send changed pages to IndexNow. This page covers three parts of analytics: the web vitals from real visitors, the AI and search crawler hits from your access logs, and the IndexNow submit of changed pages. ## Web vitals The tracking script measures the Core Web Vitals of real visitors. It needs no change to your tag and no third-party library. | Metric | What it measures | Good up to | Poor above | |---|---|---|---| | LCP | Largest Contentful Paint | 2,500 ms | 4,000 ms | | INP | Interaction to Next Paint | 200 ms | 500 ms | | CLS | Cumulative Layout Shift | 0.1 | 0.25 | | TTFB | Time to First Byte | 800 ms | 1,800 ms | The thresholds come from [web.dev](https://web.dev/articles/vitals). ### Read the web vitals ```sh ce vitals example.com --period 7d ce vitals example.com --path /pricing ``` The answer has the 75th percentile (p75) of each metric for the site and its top pages. Each value has n, the share of good values and the rating: `good`, `needs_improvement` or `poor`. The MCP tool is `web_vitals`. The site page shows a **Web vitals** panel. | Confidence | Reports | |---|---| | High | 100 or more | | Medium | 20 or more | | Low | Fewer than 20 | > **Note** > The web vitals are not tested for significance. Exit code 4 means that the period has no reports yet. ### What the script sends The script sends one report for each page load, when the page is hidden first. A report has the page URL and the metrics that the browser measured. No report comes from a browser without the APIs, from automation, or from a `file:` page. A report keeps no visitor ID and no location. A report is not a page view, so the page view numbers do not change. A page with bad vitals that loses search position makes a `slow_page_losing_position` finding. ## AI crawlers from access logs A JavaScript tag cannot see crawlers. To see GPTBot, ClaudeBot and the other crawlers, push the access log of your web server to Checkend. ```sh ce logs push example.com /var/log/nginx/access.log ``` The token needs write access to the project of the site. ### Push the log each day Run the push from cron on the web server, for example with the log of yesterday: ```sh CHECKEND_URL=https://stats.example.com CHECKEND_TOKEN=ce_... \ ce logs push example.com /var/log/nginx/access.log.1 ``` For a compressed log, read it from stdin: ```sh zcat access.log.2.gz | ce logs push example.com - ``` ### Formats and limits - Formats: the combined format of Nginx and Apache, and the JSON access log of Caddy. - At most 10 MB in one push. Split a larger file. - A line longer than 16 KiB refuses the push. - Lines older than 31 days, or in the future, are not kept. - A Caddy line of another host is not kept. - Crawler hits do not count against the daily event limit. A second push of the same lines adds nothing to the data. The answer of each push counts the hits that it found and sent, so a second push shows the same count again. ### What Checkend keeps Checkend keeps only the lines of known crawlers: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Bingbot, Applebot, CCBot, Bytespider, Amazonbot, Meta-ExternalAgent and DuckAssistBot. For each hit, it keeps the UTC day, the path without the query, the bot name and the status. It keeps no IP address and no user agent. It drops the lines of people and other clients. > **Note** > Checkend does not check the user agent against the IP ranges of the crawler operators. A client can claim to be GPTBot. Google-Extended and Applebot-Extended are `robots.txt` tokens only: their fetches show as Googlebot and Applebot. ### Read the crawler hits ```sh ce crawlers example.com --period 7d ce crawlers example.com --bot GPTBot ``` The answer has the hits, the pages, the errors (status 400 or more) and the last day of each crawler, and the top pages. The MCP tool is `ai_crawlers`. The site page shows an **AI crawlers** panel. ## IndexNow IndexNow tells Bing and other IndexNow search engines that pages changed. Checkend sends: - the pages that the crawler finds changed (title, meta description, H1 or canonical URL), - the page of each agent change on a finding with a page, - the URLs that you submit. ### Turn IndexNow on Only an admin can turn IndexNow on. There is no MCP action for it, because an agent must not send data to an outside service without the permission of an admin. 1. Turn IndexNow on for the site: ```sh ce indexnow set example.com --on ``` Checkend makes a key of 32 hex characters. 2. Show the status and the key file snippet: ```sh ce indexnow status example.com ``` 3. Make the key file available at `https://example.com/.txt`. Use one of the two modes below. ### The key file The search engines read the key file at the root of your site. A key file in a folder covers only the URLs of that folder. **Mode `proxy` (default).** Checkend serves the key at `/indexnow/.txt`. Map `/.txt` on your site to it. Nginx example: ```nginx location = /.txt { proxy_pass https://stats.example.com/indexnow/.txt; proxy_set_header Host stats.example.com; proxy_ssl_server_name on; } ``` `ce indexnow status` also gives snippets for Caddy and Next.js. **Mode `manual`.** Put a file `.txt` with the key in the web root of your site: ```sh ce indexnow set example.com --on --mode manual ``` To make a new key, add `--new-key`. ### Submit URLs ```sh ce indexnow submit example.com https://example.com/pricing --reason "new prices" ``` The MCP tool `indexnow` can also submit URLs (write access). One submit takes at most 100 URLs, and only URLs on the host of the site. A URL that was submitted in the last 24 hours does not go to the queue again. ### The submit job - The job runs every 10 minutes (`--indexnow-every`). - It checks the key file first, and again after 7 days. When the key file is missing, it waits one hour and shows the next step. - It sends at most 1,000 URLs of a site in 24 hours (`--indexnow-max-per-day`). - The default endpoint is `https://api.indexnow.org/indexnow`. Select another one with `--indexnow-endpoint`, for example `https://www.bing.com/indexnow`. `ce indexnow status` shows the key check, the last answer, the error with the next step, and the queue. Each submit writes an audit entry. ## Prompt for a coding agent ```text Find the slowest pages of example.com and their causes. Replace example.com with my site. 1. Run `ce vitals example.com --period 7d`. 2. List the pages with a p75 LCP over 2.5 s, a p75 INP over 200 ms or a p75 CLS over 0.1. 3. For each page, find the probable cause in the code of this repository: large images, render-blocking scripts, layout shifts or slow server responses. 4. Propose one fix for each page. Do not change files yet. ```