Search Console and Bing

Checkend joins Google Search Console and Bing Webmaster Tools data with your page views and conversions. The search rules use this data to make findings, for example a query on position 8 to 20.

Connect Google Search Console

Checkend uses the OAuth client of your own Google Cloud project. Checkend never ships a shared OAuth client.

Make the OAuth client

1. In the Google Cloud console, make a project.
2. Turn on the “Google Search Console API”.
3. Set up the OAuth consent screen. Add the scope https://www.googleapis.com/auth/webmasters.readonly.
4. Make an OAuth client of the type “Web application”.
5. Add the redirect URI <public URL>/v1/search/oauth/callback, for example https://stats.example.com/v1/search/oauth/callback.

ce search status <site> also shows the redirect URI.

Give the client to the server

Start the server with the client ID as a flag and the client secret as an environment variable:

CHECKEND_GOOGLE_CLIENT_SECRET=<client secret> \
  checkend serve --google-client-id <client ID>.apps.googleusercontent.com

With the installer, put both in /etc/checkend/checkend.env, then restart the service.

Caution

Keep the client secret out of the command line. Use the environment variable, so that the secret is not in the process list.

Connect a site

1. Run the connect command:

   ce search connect example.com --google

   Or select Connect Search Console on the Search tab, or use the MCP tool connect_search_console.

2. Open the consent page that Checkend prints. Use a Google account with access to the property.

3. Approve the access within 15 minutes.

The browser goes back to the Search tab.

The property is a domain property by default (sc-domain:example.com). For a URL-prefix property, use --property:

ce search connect example.com --google --property https://example.com/

The property must cover the site.

Connect Bing Webmaster Tools

1. In Bing Webmaster Tools, open Settings → API Access, and copy the API key.

2. Give the key to Checkend on stdin:

   echo "$BING_KEY" | ce search connect example.com --bing --api-key-stdin

Checkend checks the key. The site must be a verified site of the key’s user.

Caution

Keep the Bing API key safe. It gives access to all sites of its user.

How Checkend keeps the tokens

Checkend seals the Google refresh token and the Bing API key with AES-256-GCM. The key is the file <data>/secret.key. A copy of the database without the key file does not give the tokens.

Danger

Back up secret.key with the data folder. Without it, you must connect each source again.

The sync

A job syncs the search data each hour. Change the interval with --search-every.

• Google: the newest day is today minus 2 days (UTC). Each run gets the newest 4 days again, because Google changes recent data. The first run gets the newest 7 days. Then the backfill goes back 31 days in each run, until it reaches 16 months.
• Bing: Bing gives query rows and page rows, without a date range. Bing updates its data each week.

Checkend keeps the search rows permanently. The raw-event retention does not delete them. So you keep the history after the 16 months that Google gives.

After an error, the sync waits and tries again. The connection shows the error and the next step.

Run a sync now, or check the status:

ce search sync example.com
ce search status example.com

Disconnect

ce search disconnect example.com google --yes

Disconnect deletes the stored token and revokes it at Google. The synced rows stay.

Read the search data

ce search example.com --period 28d --by query
ce search example.com --by page
ce search example.com --by query --position 8-20

The answer has clicks, impressions, CTR and the average position, by query or by page. Page rows also have the Checkend page views, visitors, page views from Organic Search, the visitors who completed a goal, and the current title.

Flag	Values
--period	7d, 28d (default), 90d, 365d. The period ends at the newest search date.
--by	query or page
--position	A range, for example 8-20 for striking-distance rows
--source	google, bing or all

The MCP tool is search_performance. The web UI has a Search tab.

Limits of the search data

• Search Console data is 2 to 3 days late.
• Search Console gives at most 50,000 rows a day for each search type, and it leaves out rare queries. So the query totals are not equal to the page totals.
• Exit code 4 means that the site has no search data yet.

The page crawler

A small crawler stores the title, meta description, H1, canonical URL and indexability of each page with traffic or search impressions. An agent needs this to propose a better title.

ce pages example.com --path /pricing

The MCP tool is get_page.

• The crawler runs each hour (--crawl-every). It crawls at most 100 pages of a site in each run, and a page again after 7 days.
• It obeys robots.txt. It reads the CheckendBot group, else *. When robots.txt gives a server error, it does not crawl.
• It waits 2 seconds between two requests to a site. It reads at most 2 MB of a page.
• Its user agent is Mozilla/5.0 (compatible; CheckendBot/1.0; +<public URL>).
• It does not follow redirects. It stores a redirect as “redirects to ”.
• It refuses loopback, private and special addresses. --crawler-allow-private turns this protection off.

A page is indexable when it has status 200, is an HTML page, has no noindex, and has a canonical URL that is the page itself.

URL inspection

With a Search Console connection, Checkend also inspects the top pages with the URL Inspection API. It uses at most 50 inspections a day for each site. A failed inspection makes an index_problem finding. ce pages and get_page show the newest inspection.

Search findings

The search data feeds these rules: striking_distance_query, low_ctr_for_position, content_decay, cannibalization, impressions_without_visits, index_problem and slow_page_losing_position. See The closed loop.