# Backups and upgrades

> Back up and restore the data folder, keep backups and the lake in an S3-compatible bucket, set the retention and upgrade the server.

Checkend keeps all data in one data folder, for example `/var/lib/checkend`. This page shows how to back it up, restore it, limit its size and upgrade the server.

> **Danger**
> Do not lose `<data>/secret.key`. It opens the stored Search Console and Bing tokens and the webhook secrets. Without it, you must connect each search source and add each webhook again.

## What a backup contains

- `app.sqlite` (the metadata) and `lake.sqlite` (the lake catalog), as consistent copies,
- the Parquet files of the lake (with the lake on a bucket, the backup reads them from the bucket),
- `secret.key`,
- `manifest.json`, with the size and the SHA-256 of each file.

The event log is not in a backup. The events that were not yet in the lake at the time of the backup are lost after a restore.

> **Caution**
> Keep a backup as safe as the data folder. It contains `secret.key`.

## Make a backup by hand

A backup works while the server runs:

```sh
checkend backup create --data /var/lib/checkend --to /backups/checkend
```

With the lake on a bucket, add the same `--lake-bucket` as the server:

```sh
checkend backup create --data /var/lib/checkend --to /backups/checkend \
  --lake-bucket s3://<bucket>/checkend
```

The scheduled backup uses the `--lake-bucket` flag of the server.

List the backups in a folder:

```sh
checkend backup list --from /backups/checkend
```

## Scheduled backups

Add these flags to `CHECKEND_FLAGS` in `/etc/checkend/checkend.env`:

```sh
--backup-dir /backups/checkend --backup-every 24h --backup-keep 7
```

| Flag | Meaning | Default |
|---|---|---|
| `--backup-dir` | A folder, or `s3://<bucket>/<prefix>`. Without it, there is no scheduled backup. | None |
| `--backup-every` | The interval. | 24 hours |
| `--backup-keep` | The count of backups to keep. | 7 |

With the installer, the systemd service can write only to `/var/lib/checkend`. Add the backup folder to `ReadWritePaths`:

1. Open an override file:

   ```sh
   sudo systemctl edit checkend
   ```

2. Add these lines:

   ```ini
   [Service]
   ReadWritePaths=/backups/checkend
   ```

3. Restart the service:

   ```sh
   sudo systemctl restart checkend
   ```

## The bucket

An S3-compatible bucket can hold the backups and the Parquet files of the lake. Examples are AWS S3, MinIO, Cloudflare R2 and Hetzner Object Storage. The bucket must exist. Make it in the console of your provider.

Set the bucket access in the environment of the server. For the service, put the variables in `/etc/checkend/checkend.env`:

| Variable | Meaning | Default |
|---|---|---|
| `CHECKEND_S3_ENDPOINT` | The endpoint URL. | `https://s3.<region>.amazonaws.com` |
| `CHECKEND_S3_REGION` | The region. | `us-east-1` |
| `CHECKEND_S3_ACCESS_KEY_ID` | The access key ID. | None |
| `CHECKEND_S3_SECRET_ACCESS_KEY` | The secret access key. | None |
| `CHECKEND_S3_URL_STYLE` | `path`, or `vhost` for the bucket in the host name. | `path` |

Checkend uses Signature Version 4. An example for Hetzner Object Storage in `fsn1`:

```sh
CHECKEND_S3_ENDPOINT=https://fsn1.your-objectstorage.com
CHECKEND_S3_REGION=fsn1
CHECKEND_S3_ACCESS_KEY_ID=...
CHECKEND_S3_SECRET_ACCESS_KEY=...
CHECKEND_S3_URL_STYLE=vhost
```

[Hetzner documents](https://docs.hetzner.com/storage/object-storage/getting-started/using-s3-api-tools/) the endpoint `https://<location>.your-objectstorage.com` and URLs with the bucket in the host name.

> **Note**
> Checkend was tested only with MinIO and path-style URLs. It was not tested with Hetzner or with other providers.

### Backups in the bucket

Use `s3://<bucket>/<prefix>` in place of a folder, in `--backup-dir`, `--to` and `--from`:

```sh
checkend backup create --data /var/lib/checkend --to s3://<bucket>/backups
```

### The lake on the bucket

The Parquet files of the lake can go to the bucket. The lake catalog (`lake.sqlite`) and the metadata (`app.sqlite`) stay in the data folder.

1. Add `--lake-bucket s3://<bucket>/checkend` to `CHECKEND_FLAGS` in `/etc/checkend/checkend.env`.
2. For a server that already has data, stop the server:

   ```sh
   sudo systemctl stop checkend
   ```

3. Copy the Parquet files to the bucket. A second run copies only the new files.

   ```sh
   sudo -u checkend env $(grep CHECKEND_S3 /etc/checkend/checkend.env | xargs) \
     checkend lake push --data /var/lib/checkend --to s3://<bucket>/checkend
   ```

4. Start the server:

   ```sh
   sudo systemctl start checkend
   ```

5. Run `checkend doctor`. The check `lake_files` must be OK.

   ```sh
   checkend doctor --data /var/lib/checkend --lake-bucket s3://<bucket>/checkend
   ```

6. When the server works on the bucket, delete `/var/lib/checkend/objects`.

With `--lake-bucket`, the first start also downloads the DuckDB extension `httpfs`. DuckDB keeps the bucket key in memory only. The writer sends new data to the bucket about every 5 seconds. The compaction job merges the small files each hour. The retention and the compaction also work on the bucket.

The server does not start when the catalog names data files and the lake store has none of them. The cause is a wrong `--lake-bucket`, or a missing `checkend lake push`. The message tells you what to do.

> **Caution**
> Do not give the same `--lake-bucket` prefix to two servers. Each server has its own catalog, and the cleanup of one server deletes the files of the other.

### Litestream

[Litestream](https://litestream.io) copies `app.sqlite` and `lake.sqlite` to the bucket each second. It is optional, and Checkend does not start it. Checkend was tested with Litestream v0.5.17.

The Checkend source has an example configuration, `deploy/litestream.yml`. It takes its values from the environment. Copy it to the server, and give its path to `-config`:

```sh
CHECKEND_DATA=/var/lib/checkend CHECKEND_LITESTREAM_BUCKET=<bucket> \
CHECKEND_LITESTREAM_PREFIX=checkend/litestream \
CHECKEND_S3_ENDPOINT=... CHECKEND_S3_REGION=... CHECKEND_S3_ACCESS_KEY_ID=... CHECKEND_S3_SECRET_ACCESS_KEY=... \
  litestream replicate -config litestream.yml
```

Run Litestream as the user `checkend`, as its own systemd service. `checkend doctor` shows a `litestream` check. With the lake on a bucket, it warns when Litestream does not copy `lake.sqlite`.

> **Caution**
> Keep `secret.key` and a scheduled backup too. Litestream does not copy `secret.key`. Without it, the sealed tokens and webhook secrets do not open.

### Restore from the bucket

Use these steps on a new server with the same bucket.

1. Install Checkend with `--no-start`.
2. Copy `secret.key` from a backup into the new data folder. Give it mode 0600 and the owner `checkend`.
3. Restore the two SQLite files into the data folder:
   - With Litestream:

     ```sh
     litestream restore -config litestream.yml -o /var/lib/checkend/app.sqlite /var/lib/checkend/app.sqlite
     litestream restore -config litestream.yml -o /var/lib/checkend/lake.sqlite /var/lib/checkend/lake.sqlite
     ```

   - Without Litestream, restore the newest backup. The restore writes a Parquet file to the bucket only when the bucket does not have it.

     ```sh
     checkend backup restore --from s3://<bucket>/backups --data /var/lib/checkend \
       --lake-bucket s3://<bucket>/checkend
     ```

4. Add the same `--lake-bucket` to `CHECKEND_FLAGS`.
5. Start the server.
6. Run `checkend doctor`. The checks `lake_files` and `data_folder` must be OK.

The events of the last seconds before the loss are lost. They were in the event log, and not yet in the lake.

## Restore a backup

1. Stop the server:

   ```sh
   sudo systemctl stop checkend
   ```

2. Restore the newest backup:

   ```sh
   checkend backup restore --from /backups/checkend --data /var/lib/checkend
   ```

   To restore an older backup, add `--id <id>`. Get the ID from `checkend backup list`. With the lake on a bucket, add `--lake-bucket`. The restore then writes only the Parquet files that the bucket does not have.

3. Start the server:

   ```sh
   sudo systemctl start checkend
   ```

The restore checks every file before it writes. A data folder that already has data needs `--force`. The old data then moves to a folder next to it.

> **Caution**
> Make sure that the user `checkend` owns the restored files before you start the server. A restore as root can leave files that the server cannot write:
>
> ```sh
> sudo chown -R checkend:checkend /var/lib/checkend
> ```

## Retention

Checkend keeps the raw events for 24 months by default. A nightly job deletes the older months as whole months.

- The daily rollups stay permanently, so the numbers of old days stay.
- The search rows stay permanently.
- After a month is deleted, a visitor counts one time on each day of that month.

Show and change the retention (admin):

```sh
ce retention
ce retention set analytics --months 18 --dry-run
ce retention set analytics --months 18
```

## The disk guard

The disk guard protects the server from a full disk. It runs every 10 minutes.

When the disk use goes above the limit, it deletes the oldest raw month first. It never deletes the current or the previous month. The rollups stay.

| Flag | Meaning | Default |
|---|---|---|
| `--disk-limit` | The disk use in percent above which the guard deletes. `0` turns it off. | 90 |
| `--disk-budget` | The raw-data budget of a module, for example `analytics=50GiB`. | None |
| `--disk-guard-every` | The interval. | 10 minutes |

The disk guard tells you in three ways:

- an audit entry `data.delete_raw` for each deleted month,
- the webhook event `disk.pressure`,
- a failed `disk` check in `ce health` and `checkend doctor`.

## Upgrades

### With the installer

1. Make a backup.
2. Run the installer again:

   ```sh
   curl -fsSL [INSTALL_URL] | sudo sh
   ```

   It downloads the new version only when the version is different. It replaces the binaries and restarts the server.

3. Check the server:

   ```sh
   checkend doctor --data /var/lib/checkend --public-url https://stats.example.com
   ```

### With Docker

1. Make a backup.
2. Pull the new image.
3. Start a new container on the same volume.

### Migrations

The new binary runs the pending migrations at start. You never run a migration by hand. Migrations only go forward.

Before it migrates `app.sqlite`, Checkend copies the file to `app.sqlite.before-migration-<n>` (mode 0600). This is the rollback copy.

> **Caution**
> Delete `app.sqlite.before-migration-<n>` when the new version works. The copy can hold old secrets in clear, for example the webhook secrets of an older version.

## Check the health

After a backup, a restore or an upgrade, run the checks in [Troubleshooting](/docs/guides/troubleshooting.md).