Monitoring
This page covers how to tell whether Endurain is healthy, where its logs go and what to watch. Endurain has no metrics endpoint: the signals are the two health endpoints, the logs and a few database tables.
Health endpoints
Both are unauthenticated and need no X-Client-Type header.
| Endpoint | Checks | Fails with |
|---|---|---|
GET /api/v1/about |
That the web process is up. Returns the name, version and licence | Nothing but a dead process |
GET /api/v1/health |
That a database connection can be obtained and answers SELECT 1, that the shared state backend (Redis, when configured) answers, and that the audit trail is healthy. Returns {"status": "ok", "state": "ok", "audit": "ok"} |
503 with Retry-After: 5 when the pool is exhausted or the database is unreachable |
The container image's own HEALTHCHECK calls /api/v1/about every 30 seconds, with a 120 second start period. That tells Docker the process runs, not that the database works. For a load balancer or an orchestrator that should stop sending traffic to a wedged instance, probe /api/v1/health instead.
When only the shared state backend does not answer, /api/v1/health still returns 200, with {"status": "degraded", "state": "unavailable", "audit": "ok"}. Restarting or removing the instance from a load balancer would not bring Redis back, and the instance still serves most routes: rate limits fall back to per-process counters (so the effective limit is up to the number of replicas times higher until Redis returns), while features that need shared state, such as MFA, WebSocket tickets and sync claims, answer 503. Alert on "status": "degraded" in the body instead of on the status code. With the default local profile the state is in memory, so state is always ok.
audit: "degraded" also answers 200 with "status": "degraded", and the reason is logged at DEBUG as The audit trail is degraded: <reason>, so set LOG_LEVEL=debug to read it; the body never says which. It means one of these:
- tomorrow's audit partition does not exist, or the maintenance run has never recorded one;
- the latest sealed audit checkpoint is older than an hour (a checkpoint that was never sealed is not flagged);
- an audit record for a refused sign-in or permission failed to write on this instance within the last hour;
- the audit health itself could not be read.
Check that the scheduler runs and that the audit maintenance job completes (see the job tables and the logs), then that PostgreSQL has the daily partitions of jasil_audit_log for the next days, and search the logs for Audit record ... was not written. A problem with the partitions or the checkpoint is shared by every replica; a write failure is only that of the instance you probed.
flowchart TD
P[Probe] --> A{GET /api/v1/about}
A -- no answer --> D[Process down: restart it]
A -- 200 --> H{GET /api/v1/health}
H -- 200, status ok --> OK[Healthy]
H -- 200, state unavailable --> R[State backend unreachable: check Redis]
H -- 200, audit degraded --> U[Audit trail unhealthy: check the scheduler, the partitions and the logs]
H -- 503 --> W[Web process up, database unreachable or connection pool exhausted]
Neither endpoint says anything about background work. A worker that has stopped, or a queue that is backing up, shows only in the job tables and the logs.
Logs
| Where | What |
|---|---|
| Container output (stdout) | One JSON object per line, in production and demo |
app.log in LOGS_DIR |
The same records. In development it is human-readable text |
| Uvicorn access log | Off unless LOG_ACCESS=true |
Set the level with LOG_LEVEL (critical, error, warning, info, debug, trace). Raise it to debug only while investigating, because debug logs routine work such as every stored record.
app.log rolls over at 10 MiB and keeps five old files, but only when a single process writes it. With several web workers, a dedicated worker or several replicas, the file is not rotated, so send the stdout JSON to your platform's log collection instead and keep the file small or off.
Finding one request
Every response carries an X-Request-ID header, and every error body carries the same value as request_id. Logs written while handling that request carry it too. A durable background job's log lines carry the id of the request that queued it, so one id finds both the request and its background work. Without durable jobs, work handed to a local pool runs with the context of the request that started it, so its log lines carry the same id. Ask the user for the id, then search for it.
docker compose logs endurain | grep <request_id>
The log levels
| Level | Means |
|---|---|
| ERROR | A failure nobody asked for, with its traceback |
| WARNING | A refusal or a degraded path an operator may need to see: a denied caller, a dropped event subscriber, a provider that refused |
| INFO | A change someone would audit: an account created, approved, deleted or linked, settings changed, a background job queued or finished |
| DEBUG | Routine work |
Messages worth alerting on:
| Message | Level | Meaning |
|---|---|---|
Dropped an event subscriber: the subscriber pool is full |
WARNING | The in-process queue was full and a thumbnail, notification or cleanup was skipped. A scheduled job repairs it. See Background work |
A <pool> job failed |
ERROR | A job on a local pool raised. The traceback follows |
Too many <pool> jobs in progress |
(503 to the client) | A bounded local pool is full, and the request was refused up front |
Inconsistent deployment configuration |
Startup error | Startup stops. The message names the setting |
In the app
Administrators can read the newest records in Settings > Server settings > Logs. It is available only when one process writes the log file, so it is off on the public demo and on multi-process deployments.
What to watch
- Database connections. Each web process holds up to 60. See Sizing.
/api/v1/healthgoing503under load usually means the pool or PostgreSQL'smax_connectionsis exhausted. - Dead-lettered jobs (durable jobs only).
SELECT count(*) FROM jasil_processing_jobs WHERE status = 'dead_letter'. They are kept for review and never pruned. See Troubleshooting. - Pending jobs that do not shrink. On the durable profile with
JOBS_RUN_IN_PROCESS_WORKER=false, this means no worker is running. - Disk space for
LOGS_DIRand the data volume, or your bucket's size on object storage. See Storage and backups. - Audit trail.
auditin the/api/v1/healthbody turnsdegradedwhen tomorrow's partition is missing, the latest checkpoint is over an hour old or a refusal row failed to write recently. A refused sign-in or permission whose audit row cannot be written is logged at ERROR asAudit record ... was not written, and the request goes on. An accepted change whose row cannot be written fails with the change. Every audit row needs its day's partition, which the maintenance job creates a week ahead, so check that the scheduler runs. - Mail. Failed emails are retried only with durable jobs on. A test email in the server settings tells you whether the settings work.
- Strava. Its rate limits are per application. When one is hit, syncs stop until it expires and the log says so.