The closed loop

View as Markdown

Most analytics tools stop when they show a number. Checkend continues. It tells the agent what to fix, records the change, measures the result, and keeps the result as memory.

Finding  ->  Change (annotation)  ->  Measured outcome  ->  Memory (the brief)
   ^                                                              |
   +--------------------------------------------------------------+

Agents do not remember between runs. Checkend keeps what was tried on each site and what worked.

The four parts

Part	What it is
Finding	A fact that a rule found, with evidence. For example: “The page views of /pricing fell 31 %.”
Change	An annotation: a record of what the agent or a person changed, and when.
Outcome	The measured result of the change: the numbers 14 days before and 14 days after, with a statistical test.
Memory	The brief. It gives the outcomes of earlier changes to the next agent run, newest first.

A typical run

The agent calls get_brief (or ce brief example.com). It reads the open findings.
The agent claims a finding, so that other agents do not work on it.
The agent makes the change, for example a new title on /pricing.
The agent marks the finding done, with the text of the change. Checkend records an annotation.
Checkend starts to measure on the next UTC day.
After the 14 days after the change day, Checkend sets the outcome.
The next brief shows the outcome. A change that made things worse comes first, with a warning.

With ce:

ce findings list --status open
ce findings claim f_8k2
ce findings done f_8k2 --text "Title changed on /pricing" --link https://github.com/acme/web/pull/7
ce findings show f_8k2

With MCP, the agent uses the tools get_findings, update_finding and annotate.

Finding statuses

Status	Meaning
`open`	A rule found the fact. Nobody works on it.
`claimed`	A token claimed the finding. Another token gets an error with the name of the claimer.
`done`	The change is made. The finding has an annotation of the change.
`measuring`	Checkend measures. This starts at 00:00 UTC on the day after the change day.
`worked`	The primary metric moved in the wanted direction, significant at 95 %.
`got_worse`	The primary metric moved in the wrong direction, significant at 95 %. The change made it worse.
`no_clear_change`	The test had enough data, and found no change.
`too_few_to_compare`	The test cannot reach 95 % with the data that it has.
`dismissed`	A person or an agent decided not to act on the finding.

The allowed changes:

From	Possible actions
`open`	claim, done, dismiss
`claimed`	done, dismiss, reopen
`done`	reopen. Checkend moves it to `measuring` on the next UTC day.
`measuring`	reopen, remeasure (admin). Checkend sets the outcome when the after window is complete.
An outcome status	reopen, remeasure (admin)
`dismissed`	reopen

Done needs a change. Give an existing annotation of the same site, or a text for a new annotation. Each status change goes to the history of the finding and to the audit log. Use --dry-run (or dry_run in MCP) to check a change without making it.

A rules run does not reopen a finding that is not open. A person or an agent can reopen it.

How Checkend measures an outcome

The windows. The change day is the UTC day of the annotation. The before window is the 14 days before the change day. The after window is the 14 days after it. The change day is in neither window. Each window has each weekday two times.

The primary metric decides. It is the metric that the change wants to move. Checkend uses one of two tests:

Kind of metric	Test	Examples
A rate	Two-proportion z-test	Conversion rate, search CTR, the share of good page loads
A count	Poisson rate test	Page views, 404 events, search clicks, search impressions

Checkend also shows secondary metrics with the same tests: page views, visitors, conversions and the conversion rate. They do not decide.

The decision, in this order:

A window with fewer than 12 of 14 days with data: too_few_to_compare.
The primary metric changed, significant at 95 %: worked in the wanted direction, else got_worse.
Not significant, and the test has less than 80 % power to find a change of the site’s minimum detectable effect: too_few_to_compare.
Else: no_clear_change.

For a search finding, Checkend waits until the search data covers the after window. Search Console data is 2 to 3 days late.

The outcome sensitivity

The minimum detectable effect (MDE) is the smallest change that the test must find with 80 % power. It is a setting of each site:

The default is 10 %.
The range is 5 % to 50 %.
A small MDE finds small changes, but needs more data.
A large MDE needs less data, but misses small changes.

As an example, at an MDE of 10 %, a count needs about 1,648 events in the 14 days before for 80 % power to find a rise of 10 %. At 20 %, it needs about 431 events.

Only an admin can change the MDE:

ce sites set example.com --outcome-mde 20

There is no MCP tool for this setting. An agent must not change how its own work is judged. Each outcome stores the MDE that it used. A new setting does not change old outcomes.

Remeasure

An admin can compute an outcome again, for example after a change of the MDE:

ce findings remeasure f_8k2

Remeasure uses the MDE that the site has now. It adds a new outcome and keeps the old outcomes in the history. It can change the status.

The outcome record

Each outcome stores the windows, the values before and after with n, the test, the z statistic, the p-value, the power, the MDE, the result, the reason, the change, the data date and who measured. ce findings show <id> shows the newest outcome and the history.

The memory: the brief

The brief is the main entry point for an agent: get_brief in MCP, ce brief in the CLI, GET /v1/brief in the REST API. It has:

Part	Content
Warnings	One sentence for each `got_worse` outcome: the change, the finding and the numbers, then “Do not repeat this change”. Read them first.
Top changes	Page views, visitors and conversions of each site, and its top pages, against the period before, with a significance test.
Open findings	The open and claimed findings, high priority first.
Annotations	The changes and releases of the period.
Outcomes	The findings with an outcome, newest first: what was tried and what happened.
Data dates	The time of the newest event of each site.

The default period is 7 days.

Annotations

An annotation records a change on a date. It has a text (1 to 500 characters), a kind and a time. It can link to a finding and to a URL.

Kind	Use
`agent_change`	A change by an agent. This is the default.
`release`	A release. Record it from CI.
`note`	Any other note.

ce annotate example.com "Title changed on /pricing" --finding f_8k2
ce annotate example.com "v1.4.0" --kind release

The site chart shows each annotation as a line.

The rules

Rules make findings. Each finding has a type, an action type, evidence, an estimated impact, a priority and a confidence level. The rules job runs each hour. An admin can run it now with ce findings run.

The thresholds below are choices, not measured needs.

Traffic and conversion rules

These rules compare the last 7 complete UTC days with the 7 days before. A site with a shorter history gets no drop, spike or conversion findings.

Type	Condition	Action type
`traffic_drop`	The page views of a page fell at least 25 %, with at least 20 page views in one of the two weeks, and p < 0.05.	`investigate`
`traffic_spike`	The same, but a rise. The page must have page views in the week before.	`investigate`
`conversion_drop`	The conversion rate of a goal fell at least 20 %, with at least 10 conversions in the week before, and p < 0.05.	`investigate`
`traffic_without_conversions`	A page with at least 50 page views in the week, and no visitor of the page converted. Not for sites in strict mode.	`improve_content`
`not_found_with_traffic`	At least 3 `404` events on a page in the week.	`fix_redirect`

For these rules, the priority is high when the impact is at least 10 % of the site’s number, medium from 2 %, else low. The confidence is high when p < 0.01, medium when p < 0.05.

Search and speed rules

These rules use the 28 days that end at the newest Search Console date. They need a Search Console connection. See Search Console and Bing.

Type	Condition	Action type
`striking_distance_query`	A query with an average position of 8 to 20 and at least 200 impressions.	`improve_content`
`low_ctr_for_position`	A page with a CTR below 60 % of the expected CTR for its position. The expected CTR comes from the other pages of the site.	`rewrite_title`
`content_decay`	The clicks of a page fell at least 30 % against the same 28 days one year before.	`improve_content`
`cannibalization`	Two or more pages each get a large part of the impressions of the same query.	`merge_pages`
`impressions_without_visits`	A page with search clicks, but no Checkend page view.	`fix_tracking` or `fix_redirect`
`index_problem`	The URL Inspection of the page shows a failure or a blocked indexing state.	`fix_index`
`slow_page_losing_position`	A page lost at least 1.0 position, and a core web vital (LCP, INP or CLS) is not good.	`improve_speed`

Get told about changes

Webhooks send finding.created and finding.updated when a rule finds something or an outcome is set. An agent can then start without a poll. See Webhooks.