Background work and failure handling
Endurain does some work after a request has returned: map thumbnails, reverse geocoding, notifications, emails, imports and provider syncs. This page explains how that work is delivered, what happens when it fails or is dropped, and how each module behaves on a single process versus a distributed deployment.
For the settings that choose the mode, see Scaling and background jobs. For how to write a subscriber, see Events and subscribers.
Two kinds of background work
The distinction explains most of the behaviour below.
| Kind | Example | If it is lost |
|---|---|---|
| Derived work reacts to something that already happened and can be recomputed from the database | A thumbnail, a location, deleting a file after an activity is deleted | A scheduled job later finds what is missing and repairs it |
| A command is something a user asked for | An import, an upload, a provider sync, an email | Nothing repairs it. Its result is recorded on a job or state record, a failure is shown to the user, and the user starts it again |
Every piece of derived work has a reconciliation net, a scheduled job that repairs lost work. Commands deliberately have none; their module says why in subscriber_registry.py, and the reasons are summarised in the table per module below.
How an event is delivered
An event is a fact such as activity.created. The module that causes it publishes it in the same step as the change. What happens next depends on JOBS_ENABLED and the event bus.
flowchart TD
P[Service changes data and publishes an event] --> J{JOBS_ENABLED?}
J -- yes: durable --> O[Event written to the outbox in the same transaction]
O --> R[Relay creates one job per subscriber]
R --> W[A worker claims each job with a lease and runs it]
W -- fails --> B[Retry with backoff]
B --> W
B -- attempts used up --> DL[Dead letter: kept for review]
W -- succeeds --> D[Done]
J -- no: best effort --> C[Commit first, then publish]
C --> E{Event bus}
E -- memory, one process --> Q[Queue on a small in-process pool]
E -- Redis --> RB[Consumer thread of each process]
Q --> H[Handler runs once. A failure is logged and dropped]
RB --> H
H --> N[Scheduled reconciliation repairs what was lost]
DL --> N
Two points are easy to miss:
- Durable: the outbox row is committed with the data, so an event cannot be lost between "the activity was saved" and "something was told". If the transaction rolls back, the event goes with it.
- Best effort: the event is published after the commit. If the process dies in between, or a handler fails, nothing retries it. A handler that raises is logged and swallowed so that it can never break the request that caused the event.
Handlers must therefore be idempotent (delivery is at least once on the durable path) and must not assume they ran (best effort can lose them).
What the durable path guarantees
| Behaviour | Value |
|---|---|
| Attempts before a job is dead-lettered | JOBS_MAX_ATTEMPTS, 5 |
| Wait before a retry | JOBS_BACKOFF_BASE_SECONDS (5 s), doubling each attempt, up to JOBS_BACKOFF_MAX_SECONDS (1 hour), randomised to 50 to 100 % of that delay |
| A job claimed by a worker that goes silent | Requeued once its lease (JOBS_LEASE_SECONDS, 300 s) lapses. A running worker renews it every 15 s. The reaper runs every minute |
| A single job that runs too long | Abandoned after 30 minutes and handed to the reaper, even if its worker is healthy |
| A dead-lettered job | Kept for review. Never pruned automatically |
| Completed jobs and relayed outbox rows | Pruned after JOBS_RETENTION_DAYS, 90 days |
stateDiagram-v2
[*] --> pending: Relay creates the job
pending --> claimed: Worker claims it with a lease
claimed --> completed: Handler returns
claimed --> pending: Handler raises, retry later with backoff
claimed --> pending: Lease lapsed, worker presumed dead
pending --> dead_letter: Attempts used up
claimed --> dead_letter: Last attempt failed
completed --> [*]
dead_letter --> [*]
What happens when a job dead-letters
For derived work, nothing more: the reconciliation net will fill the gap. For a command a user is waiting on, the module registers an on_dead_letter callback that closes the user-visible state, so the user sees "failed" instead of "processing" until a timeout. Those are:
- activity uploads, provider refreshes and bulk imports (the job is marked failed),
- Garmin Connect and Strava syncs (the sync state becomes failed, so another can be queued),
- profile imports and exports, gear deletions, bulk visibility edits and account deletions (the state becomes failed).
A scheduled pass that runs every minute delivers the callbacks, so one runs also when the job died without raising, for example a worker killed mid-job, whose own error handling never ran.
The same event, two deployments
The delivery flowchart above shows the choices. This is what actually runs, step by step, when an activity is stored, side by side. The settings that select each column are in Scaling and background jobs, and the topologies are drawn in How events are delivered.
sequenceDiagram
autonumber
participant API as API process
participant DB as PostgreSQL
participant Q as Queue or relay
participant W as Thumbnail handler
participant S as Scheduler
rect rgb(235, 245, 255)
Note over API,S: Local: JOBS_ENABLED=false, one process
API->>DB: Commit the activity
API->>Q: Publish activity.created on the in-process bus
Q->>W: Queue it on the 2-thread pool, 500 waiting at most
W->>DB: Render and record the thumbnail
Note over Q,W: If the queue is full, the handler is skipped and a warning is logged
S->>DB: Hourly: find activities with no thumbnail
S->>W: Render the missing ones
end
rect rgb(240, 255, 240)
Note over API,S: Distributed: JOBS_ENABLED=true, several replicas and workers
API->>DB: Commit the activity and the outbox row together
Q->>DB: Relay, every 5 s: turn the outbox row into one job per subscriber
W->>DB: Claim the job with a lease, renewed every 15 s
W->>DB: Render and record the thumbnail
Note over Q,W: A failure retries with backoff, then dead-letters after 5 attempts
S->>DB: Reaper, every minute: requeue jobs whose lease lapsed
S->>DB: Hourly net, under a lock: only one replica runs it
end
The processes involved differ too:
flowchart LR
subgraph L[Local, one container]
LA[API process] --- LP[In-process pools: subscribers, ingestion, Garmin sync, profile transfer]
LA --- LS[Scheduler, no lock needed]
end
subgraph D[Distributed]
DA1[API replica 1]
DA2[API replica 2]
DW1[Worker 1]
DW2[Worker 2]
PG[(PostgreSQL: outbox and jobs)]
RD[(Redis: state, events, realtime)]
LK{{Coordination lock}}
DA1 & DA2 -- publish --> PG
DA1 & DA2 -- relay every 5 s, reap every minute --> PG
DW1 & DW2 -- claim jobs, SKIP LOCKED --> PG
DA1 & DA2 & DW1 & DW2 --- RD
DA1 & DA2 -- scheduled jobs --> LK
end
What each part does:
| Part | Runs where | Job |
|---|---|---|
| Outbox | PostgreSQL | Holds an event, committed in the same transaction as the change that caused it |
| Relay | Every API process, every 5 s | Fans an outbox row out into one job per subscriber. Safe on several replicas: it coordinates with SELECT ... FOR UPDATE SKIP LOCKED and the fan-out is idempotent |
| Worker | The API process (JOBS_RUN_IN_PROCESS_WORKER=true) or a dedicated APP_ROLE=worker container |
Claims jobs with a lease and runs the handlers. Workers compete for jobs, so adding workers adds throughput |
| Reaper | Every API process, every minute | Requeues jobs whose lease lapsed, and abandons a job that has run for 30 minutes |
| Scheduler | Every API process | Runs the scheduled jobs. Jobs with a lock run on one replica per interval |
| Event log | PostgreSQL | One row per event and subscriber for diagnostics, pruned after EVENT_LOG_RETENTION_DAYS |
Workers do not run the scheduler, the relay or the reaper, so with JOBS_RUN_IN_PROCESS_WORKER=false at least one API process must stay up for jobs to be relayed. Workers do not apply database migrations either; one started before the API waits until the schema is current before it claims jobs.
When something is dropped
| Situation | What happens | How it is repaired | How you see it |
|---|---|---|---|
| The in-process subscriber queue is full (500 waiting) | The subscriber is skipped | The reconciliation net of that module | WARNING Dropped an event subscriber: the subscriber pool is full |
| A best-effort handler raises | Logged and ignored. No retry | The reconciliation net, or the user retries a command | ERROR with the traceback |
| The process stops with work queued on an in-process pool | The queued work is lost | Derived work: the nets. Commands: the user retries | Nothing is logged for work that never started |
| A bounded local pool for a command is full (uploads, imports, syncs, Garmin Connect and Strava links, profile import and export, gear and account deletion, bulk edit) | The request is refused before anything is staged, with 503 and a message to try again shortly |
The client retries | 503 Too many <name> jobs in progress; try again shortly |
| The pool fills between accepting the request and starting the job | The job is closed as failed instead of left pending | The user retries | Logged, and the job shows failed |
| A durable job fails every attempt | Dead-lettered, and the user-visible state is closed | A net for derived work, or the user retries | Row in jasil_processing_jobs with status dead_letter |
| Redis is down while a realtime hint is sent | The hint is lost | The client refetches (every minute, on reconnect, when the menu opens) | WARNING Dropped a realtime hint for key ...: publish failed |
| A notification's handler is dropped | The notification is not created, and not recreated later | None: notifications have no net | WARNING for the dropped subscriber |
Notifications are the one derived item without a repair, on purpose: the activity or follow row is the source of truth and the notification is a transient signal.
Scheduled jobs
Recurring jobs are declared by each module and run by the scheduler of the API process. Most hold a named lock, so only one replica runs them.
| Job | Every | Lock | What it does |
|---|---|---|---|
| Generate missing thumbnails | 60 min | yes | Net for thumbnail generation |
| Remove orphaned thumbnails | 24 h | yes | Net for thumbnail cleanup |
| Backfill missing locations | 60 min | yes | Net for reverse geocoding |
| Remove orphaned retained activity files | 24 h | yes | Net for file cleanup |
| Remove orphaned activity photos | 24 h | yes | Net for photo cleanup |
| Remove activity deletion records past their retention | 24 h | yes | Retention |
| Queue a provider refresh for every user with Strava or Garmin linked | 60 min | yes | Fan-out, one job per user |
| Queue again provider refreshes whose rate limit cleared | 5 min | yes | Releases deferred refreshes |
| Fail ingestion jobs that stopped progressing | 60 min | yes | Closes jobs stuck in processing |
| Prune ingestion job records | 24 h | yes | Retention |
| Remove staged ingestion blobs nothing reads | 24 h | yes | Cleanup |
Remove failed bulk-import files past JOBS_RETENTION_DAYS |
24 h | yes | Retention |
| Pull the last day of Garmin Connect health data | 4 h | yes | One sync per linked user |
| Delete the Strava data of users who revoked access on Strava | 60 min | yes | Deletes it once the 7-day grace period has passed without a relink |
| Remove staged profile imports and exports | 6 h each | yes | Cleanup |
| Queue again account deletions that never finished | 60 min | yes | Net for account deletion |
| Remove user photos no user names | 24 h | yes | Cleanup |
| Remove gear and gear component deletion records past their retention | 24 h each | yes | Retention |
| Remove health record deletion records past their retention | 24 h | yes | Retention |
| Purge read notifications past retention | 24 h | yes | Retention |
| Authentication clean-ups (reset and sign-up tokens, OAuth states, sessions, rotated tokens) | 1 to 60 min | yes | Delete what has expired; an email change's rows stay 7 days, for its "this wasn't me" link |
| Delete unverified sign-ups whose verification link expired | 60 min | yes | Deletes the account row; each one is audited as users.user.purge |
| Revoke identity provider tokens of ended sessions | 15 min | yes | Net for identity provider token revocation |
| Remove identity provider icons no provider names | 24 h | yes | Cleanup |
| Prune known devices unseen for 400 days | 24 h | yes | Retention, in batches |
| Delete API keys revoked or expired more than 30 days ago | 60 min | yes | Retention, in batches |
| Refresh trusted-proxy hostnames | 1 min | no | Keeps TRUSTED_PROXIES names current |
| Prune event log, outbox and job bookkeeping | 24 h | yes | Retention |
| Maintain the audit trail | 5 min | yes | Creates the audit partitions ahead of time, seals checkpoints and applies the audit retention |
A job with a lock uses the platform's coordination lock. With the default noop:// lock on one process it always runs. With several processes the default is postgres-advisory://, and a replica that does not get the lock skips that run and logs it at DEBUG. A replica also skips a run when another replica already completed that job in the same interval, so a locked job runs once per interval across replicas. A job without a lock runs on every scheduler process, so it is written to be safe to repeat.
Strava's hourly activity pull is the provider refresh fan-out above; its only scheduled job of its own is the deletion of revoked data.
Startup also queues a catch-up run of the thumbnail, location, provider refresh and Garmin health jobs, so a restart does not wait a full interval.
Module by module
Each section says what triggers the work, what happens locally and when distributed, and what is lost if it is dropped. "Distributed" here means DEPLOYMENT_PROFILE=distributed, which requires JOBS_ENABLED=true, Redis, S3 and a shared lock. A multi-worker local deployment sits between the two: the shared state moves to Redis but the other rules are the local ones unless you also enable durable jobs.
Activity thumbnails
| Triggers | activity.created (render), activity.deleted (remove), and a change of the tile server (re-render everything, when that option is on) |
| Local | The handler runs on the 2-thread subscriber pool, after the request has returned. A map needs tiles, so a slow tile server delays only thumbnails |
| Distributed | One durable job per subscriber on a worker, with retries. Thumbnails are written to the bucket |
| If dropped | The hourly net renders any activity that still has no thumbnail. The daily sweep removes thumbnails whose activity is gone. The tile-change re-render has no net: it is an admin command, so repeat the settings save |
Reverse geocoding
| Trigger | activity.created |
| Local | Runs on the subscriber pool, throttled by the geocoding rate limit. A lookup that would wait more than a few seconds for its turn, or that the service refuses or fails, is left to the backfill |
| Distributed | Durable job with retries. A lookup that would wait more than a few seconds for its turn, or that the service refuses or fails, runs again when the service allows, without using up a retry. The rate limit holds for every replica and worker together |
| If dropped | The hourly backfill fills in activities that still have no location, in batches that respect the rate limit |
Retained files and activity photos
| Trigger | activity.deleted |
| Local | The handler deletes the blobs on the subscriber pool |
| Distributed | Durable job |
| If dropped | A daily sweep lists the storage area and removes blobs whose activity no longer exists |
Notifications
| Triggers | activity.created, follower requested, accepted and removed |
| Local | Created on the subscriber pool. A hint then goes to the browser in the same process |
| Distributed | Durable job. The hint travels over the Redis channel to whichever replica holds your socket |
| If dropped | The notification is not created and not recreated. The creation is idempotent, so a redelivered event does not duplicate it |
Emails: sign-up and password reset
| Triggers | A sign-up verification, an invitation to choose a password for an account an administrator created, an approval request to administrators, an approved notice, a password reset request, an email-change confirmation, a notice to the previous address after an email change |
| Local | The email is sent from the subscriber pool. A failed send is logged and not retried |
| Distributed | Durable job. A failed send is retried with backoff, which is why email retries need JOBS_ENABLED=true |
| If dropped | Nothing repairs it. A lost reset email is recovered by requesting another, a lost verification email by signing up again or an administrator, a lost email-change confirmation by resending it from Settings. The user and token rows stay the source of truth. A password reset request answers the same for every address and only stages the event; the subscriber decides (account state, the five-minute cooldown, the audit row), so a dropped event leaves no cooldown or audit row behind |
Activity uploads, provider refreshes and bulk imports
These are commands. Their state lives on an ingestion job row, which the client polls.
| Local | After the job row is committed, the work runs on a 2-worker pool that holds up to 100 jobs. A full pool refuses the request with 503 before a job row exists |
| Distributed | The job row and the outbox event commit together and a worker runs the job. A bulk import runs in bounded passes (a time and file budget) and continues itself as the same job, so large imports interleave with other work. A crash resumes from the saved position |
| Fan-out | Every hour one refresh job is queued per user who linked Strava or Garmin Connect. A user whose previous refresh is still running is skipped |
| After a restart | Local with one web worker: at startup every open upload and bulk import is submitted to the pool again (a bulk import continues after its saved position), and a folder import that was still being packed is failed. Provider refreshes are failed instead and the next hourly fan-out queues them again |
| If dropped | A job that stops progressing is failed by the hourly reaper (after 24 hours without progress). A dead-lettered job is closed as failed. The user uploads or starts the import again. Re-running is safe: duplicate detection skips what was already imported |
Garmin Connect
| Linking | The login runs in the background on a pool of 4. The attempt's status (queued, running, succeeded or failed, with mfa_required while the login waits for a code) lives in the shared state, so any replica can answer the status poll. The MFA code is passed through the same shared state. A second attempt while one runs is refused with 409 |
| Gear and health syncs | Local: a 2-worker pool holding up to 50. Distributed: a durable job. Only one sync of a kind per user at a time (409 otherwise) |
| Recurring | Every 4 hours, one health sync per linked user, under a lock, so replicas do not repeat it |
| If dropped | A failed sync is marked failed so another can be queued. A dead-lettered sync is closed the same way. The recurring pull covers the last day again |
Profile import and export
These are commands with a state record the client polls (an import's state lives 4 hours, an export's 2 hours).
| Local | Run on a dedicated pool of 1 worker holding 3 jobs, apart from the subscriber pool, because an archive can take minutes. A full pool refuses the request with 503 before the archive is staged |
| Distributed | Durable job. The staged archive is in the bucket, so any worker can take it |
| If dropped | A job the pool could not take is closed as failed and its archive removed. Unused staged archives are swept every 6 hours. The user starts again |
Account deletion
This is a command with a state record that an administrator polls (the state lives 2 hours).
| Local | Runs on a dedicated pool of 1 worker holding 5 deletions, in one pass to the end. A full pool refuses the request with 503 |
| Distributed | Durable job. It runs in bounded passes (a third of the job lease, at least 30 seconds) and continues itself as the same job until the activities are gone, then erases the photo and the user. Between passes the deletion stays running and a reactivation still cancels it |
| If dropped | A dead-lettered deletion is closed as failed. The hourly sweep queues a stalled one again, up to 5 times, then an administrator deletes the account again |
Server settings
A change to the tile server publishes server_settings.tile_settings_changed in the same transaction as the save, and the thumbnail module reacts. The settings are cached for a few seconds in each process, and the content security policy for a new tile domain reaches the other replicas within about 30 seconds.
Followers and activities
These modules publish events and subscribe to none. activity.created, activity.updated and activity.deleted and the follower events are consumed by the modules above.
Realtime hints
Delivered best effort over the WebSocket, and over a Redis channel when several processes run. See Realtime updates.
A revoked session (sign-out, revocation, password change or reset, a deactivated or deleted account, a detected refresh-token reuse) publishes auth.sessions_revoked when its transaction commits. A bus handler in the websocket module turns it into a control message on the realtime channel, and every API replica closes the sockets of that session with close code 4401. Nothing repairs a lost close: the socket ends with its 15-minute connection lifetime.
Subscriber catalog
Every durable job is one subscriber reacting to one event. The subscriber id is what you see in jasil_processing_jobs.subscriber_id and jasil_event_log.handler_name, and it is stable across refactors, so you can filter on it. See Troubleshooting for the queries, and Events and subscribers for how an id is chosen.
"Runs without durable jobs" says where the handler runs when JOBS_ENABLED=false. A bus handler is queued on the 2-thread subscriber pool and a failure is dropped. A command pool is a bounded pool that the request submits to directly, described in the module's section above. The gear deletion and bulk edit pools run one job at a time and hold 5.
| Subscriber id | Event | Runs without durable jobs | Repaired by | On dead-letter |
|---|---|---|---|---|
activity_thumbnail.generate |
activity.created |
Bus | Hourly missing-thumbnail job | Net fills the gap |
activity_thumbnail.cleanup |
activity.deleted |
Bus | Daily orphan sweep | Net fills the gap |
activity_thumbnail.settings_changed |
server_settings.tile_settings_changed |
Bus | Nothing: repeat the settings save | Nothing |
activity_geocoding.reverse_geocode |
activity.created |
Bus | Hourly location backfill | Net fills the gap |
activity_file_storage.cleanup |
activity.deleted |
Bus | Daily orphan sweep | Net fills the gap |
activity_media.cleanup |
activity.deleted |
Bus | Daily orphan sweep | Net fills the gap |
activity.notify_created |
activity.created |
Bus | Nothing, by design | Nothing |
followers.notify_requested |
follower.requested |
Bus | Nothing, by design | Nothing |
followers.notify_accepted |
follower.accepted |
Bus | Nothing, by design | Nothing |
followers.clear_request_notification |
follower.request_removed |
Bus | Nothing, by design | Nothing |
auth.send_sign_up_verification_email |
auth.sign_up_verification_requested |
Bus | The user signs up again, or an administrator helps | Nothing |
auth.send_admin_invitation_email |
auth.admin_invitation_requested |
Bus | The administrator sets a password for the account | Nothing |
auth.send_email_change_verification_email |
auth.email_change_verification_requested |
Bus | The user resends the confirmation from Settings, or an administrator helps | Nothing |
auth.send_email_change_requested_email |
auth.email_change_requested |
Bus | Nothing: the notice is a courtesy, and the change stays unconfirmed | Nothing |
auth.send_email_changed_email |
auth.email_changed |
Bus | Nothing: the notice is a courtesy | Nothing |
auth.send_email_change_address_taken_email |
auth.email_change_address_taken |
Bus | Nothing: the notice is a courtesy | Nothing |
auth.send_sign_up_approval_request_email |
auth.sign_up_approval_requested |
Bus | Administrators also see the pending user | Nothing |
auth.notify_admin_sign_up_approval |
auth.sign_up_approval_requested |
Bus | Administrators also see the pending user | Nothing |
auth.send_sign_up_approved_email |
auth.sign_up_approved |
Bus | Nothing: the email is a courtesy | Nothing |
auth.send_sign_up_existing_account_email |
auth.sign_up_existing_account |
Bus | Nothing: the email is a courtesy | Nothing |
auth.send_password_reset_email |
auth.password_reset_requested |
Bus | The user requests another reset | Nothing |
auth.revoke_idp_session_tokens |
auth.sessions_revoked |
Bus | The 15-minute revocation sweep | Net fills the gap |
auth.refresh_idp_tokens |
auth.idp_token_refresh_requested |
Bus | Every session refresh queues it again | Nothing |
websocket.close_sockets |
auth.sessions_revoked |
Bus | Nothing: a socket ends at its 15-minute lifetime | Nothing |
auth.notify_idp_linked |
auth.idp_linked |
Bus | Nothing, by design | Nothing |
auth.notify_account_security_changed |
auth.account_security_changed |
Bus | Nothing, by design | Nothing |
auth.send_account_security_changed_email |
auth.account_security_changed |
Bus | Nothing: the change is stored and audited | Nothing |
auth.send_idp_linked_email |
auth.idp_linked |
Bus | Nothing: the change is stored and audited | Nothing |
auth.notify_password_reset_completed |
auth.password_reset_completed |
Bus | Nothing, by design | Nothing |
auth.send_password_reset_completed_email |
auth.password_reset_completed |
Bus | Nothing: the reset is stored and audited | Nothing |
auth.notify_account_locked |
auth.account_locked |
Bus | Nothing, by design | Nothing |
auth.send_account_locked_email |
auth.account_locked |
Bus | Nothing: the lock is audited | Nothing |
auth.notify_mfa_locked |
auth.mfa_locked |
Bus | Nothing, by design | Nothing |
auth.send_mfa_locked_email |
auth.mfa_locked |
Bus | Nothing: the lock is audited | Nothing |
auth.notify_step_up_session_ended |
auth.step_up_session_ended |
Bus | Nothing, by design | Nothing |
auth.send_step_up_session_ended_email |
auth.step_up_session_ended |
Bus | Nothing: the session end is audited | Nothing |
auth.notify_step_up_locked |
auth.step_up_locked |
Bus | Nothing, by design | Nothing |
auth.send_step_up_locked_email |
auth.step_up_locked |
Bus | Nothing: the lock is audited | Nothing |
auth.notify_new_device_sign_in |
auth.new_device_sign_in |
Bus | Nothing, by design | Nothing |
auth.send_new_device_sign_in_email |
auth.new_device_sign_in |
Bus | Nothing: the sign-in is audited | Nothing |
auth.notify_backup_code_used |
auth.backup_code_used |
Bus | Nothing, by design | Nothing |
auth.send_backup_code_used_email |
auth.backup_code_used |
Bus | Nothing: the use is audited | Nothing |
auth.notify_refresh_reuse |
auth.refresh_reuse_detected |
Bus | Nothing, by design | Nothing |
auth.send_refresh_reuse_email |
auth.refresh_reuse_detected |
Bus | Nothing: the reuse is audited | Nothing |
users.notify_admin_user_promoted |
users.admin_promoted |
Bus | Nothing, by design | Nothing |
users.send_admin_promoted_email |
users.admin_promoted |
Bus | Nothing: the promotion is stored and audited | Nothing |
users.profile_import_requested |
users.profile_import_requested |
Bus, then the profile transfer pool | The user starts again | Import state closed as failed |
users.profile_export_requested |
users.profile_export_requested |
Bus, then the profile transfer pool | The user starts again | Export state closed as failed |
activity_ingestion.uploaded_file |
activity.file_uploaded |
Ingestion command pool | The user uploads again | Job marked failed |
activity_ingestion.refresh_requested |
activity.refresh_requested |
Ingestion command pool | The hourly refresh fan-out, or the user | Job marked failed |
activity_ingestion.bulk_import |
activity.bulk_import_continue |
Ingestion command pool | Continues itself in bounded passes and resumes from its saved position. The stale-job reaper closes one that stops progressing | Job marked failed |
garmin.sync_requested |
garmin.sync_requested |
Garmin sync pool | The 4-hourly health pull, or the user | Sync state marked failed |
strava.sync_requested |
strava.sync_requested |
Strava sync pool | The user queues it again | Sync state marked failed |
gears.deletion_requested |
gears.deletion_requested |
Gear deletion pool | The user deletes the gear again | Deletion state marked failed |
activities.visibility_edit_requested |
activities.visibility_edit_requested |
Bulk edit pool | The user edits again | Edit state marked failed |
users.deletion_requested |
users.deletion_requested |
Account deletion pool | The hourly stalled-deletion sweep | Deletion state marked failed |
Three things follow from the table:
- A handler that has a repair job is derived work. Losing one costs time, not data.
- A handler that repairs nothing is either a command the user can repeat or a transient signal such as a notification. Each of these says why in its module's
subscriber_registry.py, and a test fails if a durable subscriber declares neither a repair job nor a reason. - The ingestion, provider sync, gear deletion, bulk edit and account deletion commands have no bus subscriber. Without durable jobs the service hands them to its own bounded pool instead, which is why a full pool answers the request with
503.
The realtime hint to the browser is not a job. It is delivered over the WebSocket and, with several processes, over a Redis channel; see Realtime updates.
Reading the table in practice
| You run | Expect |
|---|---|
| One container, defaults | Derived work follows within seconds. A restart can drop queued thumbnails and locations, and the hourly jobs fill them in. Commands that were queued are lost on restart, and their users retry |
| A few web workers, local | Same, with shared state in Redis so limits and live updates work across workers |
| Durable jobs on one host | Retries, a dead-letter list you can inspect and no loss of an event between save and publish. A worker restart costs a delay, not the work |
| Distributed | The above on several hosts, with locks so each scheduled job runs once and a bucket so any replica can serve any file |
If something did not appear, look in this order: the logs for the module's warning, the job tables on the durable path, and then wait for the next run of the module's scheduled net.