Skip to content

Background work and failure handling

Endurain does some work after a request has returned: map thumbnails, reverse geocoding, notifications, emails, imports and provider syncs. This page explains how that work is delivered, what happens when it fails or is dropped, and how each module behaves on a single process versus a distributed deployment.

For the settings that choose the mode, see Scaling and background jobs. For how to write a subscriber, see Events and subscribers.

Two kinds of background work

The distinction explains most of the behaviour below.

Kind Example If it is lost
Derived work reacts to something that already happened and can be recomputed from the database A thumbnail, a location, deleting a file after an activity is deleted A scheduled job later finds what is missing and repairs it
A command is something a user asked for An import, an upload, a provider sync, an email Nothing repairs it. Its result is recorded on a job or state record, a failure is shown to the user, and the user starts it again

Every piece of derived work has a reconciliation net, a scheduled job that repairs lost work. Commands deliberately have none; their module says why in subscriber_registry.py, and the reasons are summarised in the table per module below.

How an event is delivered

An event is a fact such as activity.created. The module that causes it publishes it in the same step as the change. What happens next depends on JOBS_ENABLED and the event bus.

flowchart TD
    P[Service changes data and publishes an event] --> J{JOBS_ENABLED?}
    J -- yes: durable --> O[Event written to the outbox in the same transaction]
    O --> R[Relay creates one job per subscriber]
    R --> W[A worker claims each job with a lease and runs it]
    W -- fails --> B[Retry with backoff]
    B --> W
    B -- attempts used up --> DL[Dead letter: kept for review]
    W -- succeeds --> D[Done]
    J -- no: best effort --> C[Commit first, then publish]
    C --> E{Event bus}
    E -- memory, one process --> Q[Queue on a small in-process pool]
    E -- Redis --> RB[Consumer thread of each process]
    Q --> H[Handler runs once. A failure is logged and dropped]
    RB --> H
    H --> N[Scheduled reconciliation repairs what was lost]
    DL --> N

Two points are easy to miss:

  • Durable: the outbox row is committed with the data, so an event cannot be lost between "the activity was saved" and "something was told". If the transaction rolls back, the event goes with it.
  • Best effort: the event is published after the commit. If the process dies in between, or a handler fails, nothing retries it. A handler that raises is logged and swallowed so that it can never break the request that caused the event.

Handlers must therefore be idempotent (delivery is at least once on the durable path) and must not assume they ran (best effort can lose them).

What the durable path guarantees

Behaviour Value
Attempts before a job is dead-lettered JOBS_MAX_ATTEMPTS, 5
Wait before a retry JOBS_BACKOFF_BASE_SECONDS (5 s), doubling each attempt, up to JOBS_BACKOFF_MAX_SECONDS (1 hour), randomised to 50 to 100 % of that delay
A job claimed by a worker that goes silent Requeued once its lease (JOBS_LEASE_SECONDS, 300 s) lapses. A running worker renews it every 15 s. The reaper runs every minute
A single job that runs too long Abandoned after 30 minutes and handed to the reaper, even if its worker is healthy
A dead-lettered job Kept for review. Never pruned automatically
Completed jobs and relayed outbox rows Pruned after JOBS_RETENTION_DAYS, 90 days
stateDiagram-v2
    [*] --> pending: Relay creates the job
    pending --> claimed: Worker claims it with a lease
    claimed --> completed: Handler returns
    claimed --> pending: Handler raises, retry later with backoff
    claimed --> pending: Lease lapsed, worker presumed dead
    pending --> dead_letter: Attempts used up
    claimed --> dead_letter: Last attempt failed
    completed --> [*]
    dead_letter --> [*]

What happens when a job dead-letters

For derived work, nothing more: the reconciliation net will fill the gap. For a command a user is waiting on, the module registers an on_dead_letter callback that closes the user-visible state, so the user sees "failed" instead of "processing" until a timeout. Those are:

  • activity uploads, provider refreshes and bulk imports (the job is marked failed),
  • Garmin Connect and Strava syncs (the sync state becomes failed, so another can be queued),
  • profile imports and exports, gear deletions, bulk visibility edits and account deletions (the state becomes failed).

A scheduled pass that runs every minute delivers the callbacks, so one runs also when the job died without raising, for example a worker killed mid-job, whose own error handling never ran.

The same event, two deployments

The delivery flowchart above shows the choices. This is what actually runs, step by step, when an activity is stored, side by side. The settings that select each column are in Scaling and background jobs, and the topologies are drawn in How events are delivered.

sequenceDiagram
    autonumber
    participant API as API process
    participant DB as PostgreSQL
    participant Q as Queue or relay
    participant W as Thumbnail handler
    participant S as Scheduler

    rect rgb(235, 245, 255)
    Note over API,S: Local: JOBS_ENABLED=false, one process
    API->>DB: Commit the activity
    API->>Q: Publish activity.created on the in-process bus
    Q->>W: Queue it on the 2-thread pool, 500 waiting at most
    W->>DB: Render and record the thumbnail
    Note over Q,W: If the queue is full, the handler is skipped and a warning is logged
    S->>DB: Hourly: find activities with no thumbnail
    S->>W: Render the missing ones
    end

    rect rgb(240, 255, 240)
    Note over API,S: Distributed: JOBS_ENABLED=true, several replicas and workers
    API->>DB: Commit the activity and the outbox row together
    Q->>DB: Relay, every 5 s: turn the outbox row into one job per subscriber
    W->>DB: Claim the job with a lease, renewed every 15 s
    W->>DB: Render and record the thumbnail
    Note over Q,W: A failure retries with backoff, then dead-letters after 5 attempts
    S->>DB: Reaper, every minute: requeue jobs whose lease lapsed
    S->>DB: Hourly net, under a lock: only one replica runs it
    end

The processes involved differ too:

flowchart LR
    subgraph L[Local, one container]
      LA[API process] --- LP[In-process pools: subscribers, ingestion, Garmin sync, profile transfer]
      LA --- LS[Scheduler, no lock needed]
    end
    subgraph D[Distributed]
      DA1[API replica 1]
      DA2[API replica 2]
      DW1[Worker 1]
      DW2[Worker 2]
      PG[(PostgreSQL: outbox and jobs)]
      RD[(Redis: state, events, realtime)]
      LK{{Coordination lock}}
      DA1 & DA2 -- publish --> PG
      DA1 & DA2 -- relay every 5 s, reap every minute --> PG
      DW1 & DW2 -- claim jobs, SKIP LOCKED --> PG
      DA1 & DA2 & DW1 & DW2 --- RD
      DA1 & DA2 -- scheduled jobs --> LK
    end

What each part does:

Part Runs where Job
Outbox PostgreSQL Holds an event, committed in the same transaction as the change that caused it
Relay Every API process, every 5 s Fans an outbox row out into one job per subscriber. Safe on several replicas: it coordinates with SELECT ... FOR UPDATE SKIP LOCKED and the fan-out is idempotent
Worker The API process (JOBS_RUN_IN_PROCESS_WORKER=true) or a dedicated APP_ROLE=worker container Claims jobs with a lease and runs the handlers. Workers compete for jobs, so adding workers adds throughput
Reaper Every API process, every minute Requeues jobs whose lease lapsed, and abandons a job that has run for 30 minutes
Scheduler Every API process Runs the scheduled jobs. Jobs with a lock run on one replica per interval
Event log PostgreSQL One row per event and subscriber for diagnostics, pruned after EVENT_LOG_RETENTION_DAYS

Workers do not run the scheduler, the relay or the reaper, so with JOBS_RUN_IN_PROCESS_WORKER=false at least one API process must stay up for jobs to be relayed. Workers do not apply database migrations either; one started before the API waits until the schema is current before it claims jobs.

When something is dropped

Situation What happens How it is repaired How you see it
The in-process subscriber queue is full (500 waiting) The subscriber is skipped The reconciliation net of that module WARNING Dropped an event subscriber: the subscriber pool is full
A best-effort handler raises Logged and ignored. No retry The reconciliation net, or the user retries a command ERROR with the traceback
The process stops with work queued on an in-process pool The queued work is lost Derived work: the nets. Commands: the user retries Nothing is logged for work that never started
A bounded local pool for a command is full (uploads, imports, syncs, Garmin Connect and Strava links, profile import and export, gear and account deletion, bulk edit) The request is refused before anything is staged, with 503 and a message to try again shortly The client retries 503 Too many <name> jobs in progress; try again shortly
The pool fills between accepting the request and starting the job The job is closed as failed instead of left pending The user retries Logged, and the job shows failed
A durable job fails every attempt Dead-lettered, and the user-visible state is closed A net for derived work, or the user retries Row in jasil_processing_jobs with status dead_letter
Redis is down while a realtime hint is sent The hint is lost The client refetches (every minute, on reconnect, when the menu opens) WARNING Dropped a realtime hint for key ...: publish failed
A notification's handler is dropped The notification is not created, and not recreated later None: notifications have no net WARNING for the dropped subscriber

Notifications are the one derived item without a repair, on purpose: the activity or follow row is the source of truth and the notification is a transient signal.

Scheduled jobs

Recurring jobs are declared by each module and run by the scheduler of the API process. Most hold a named lock, so only one replica runs them.

Job Every Lock What it does
Generate missing thumbnails 60 min yes Net for thumbnail generation
Remove orphaned thumbnails 24 h yes Net for thumbnail cleanup
Backfill missing locations 60 min yes Net for reverse geocoding
Remove orphaned retained activity files 24 h yes Net for file cleanup
Remove orphaned activity photos 24 h yes Net for photo cleanup
Remove activity deletion records past their retention 24 h yes Retention
Queue a provider refresh for every user with Strava or Garmin linked 60 min yes Fan-out, one job per user
Queue again provider refreshes whose rate limit cleared 5 min yes Releases deferred refreshes
Fail ingestion jobs that stopped progressing 60 min yes Closes jobs stuck in processing
Prune ingestion job records 24 h yes Retention
Remove staged ingestion blobs nothing reads 24 h yes Cleanup
Remove failed bulk-import files past JOBS_RETENTION_DAYS 24 h yes Retention
Pull the last day of Garmin Connect health data 4 h yes One sync per linked user
Delete the Strava data of users who revoked access on Strava 60 min yes Deletes it once the 7-day grace period has passed without a relink
Remove staged profile imports and exports 6 h each yes Cleanup
Queue again account deletions that never finished 60 min yes Net for account deletion
Remove user photos no user names 24 h yes Cleanup
Remove gear and gear component deletion records past their retention 24 h each yes Retention
Remove health record deletion records past their retention 24 h yes Retention
Purge read notifications past retention 24 h yes Retention
Authentication clean-ups (reset and sign-up tokens, OAuth states, sessions, rotated tokens) 1 to 60 min yes Delete what has expired; an email change's rows stay 7 days, for its "this wasn't me" link
Delete unverified sign-ups whose verification link expired 60 min yes Deletes the account row; each one is audited as users.user.purge
Revoke identity provider tokens of ended sessions 15 min yes Net for identity provider token revocation
Remove identity provider icons no provider names 24 h yes Cleanup
Prune known devices unseen for 400 days 24 h yes Retention, in batches
Delete API keys revoked or expired more than 30 days ago 60 min yes Retention, in batches
Refresh trusted-proxy hostnames 1 min no Keeps TRUSTED_PROXIES names current
Prune event log, outbox and job bookkeeping 24 h yes Retention
Maintain the audit trail 5 min yes Creates the audit partitions ahead of time, seals checkpoints and applies the audit retention

A job with a lock uses the platform's coordination lock. With the default noop:// lock on one process it always runs. With several processes the default is postgres-advisory://, and a replica that does not get the lock skips that run and logs it at DEBUG. A replica also skips a run when another replica already completed that job in the same interval, so a locked job runs once per interval across replicas. A job without a lock runs on every scheduler process, so it is written to be safe to repeat.

Strava's hourly activity pull is the provider refresh fan-out above; its only scheduled job of its own is the deletion of revoked data.

Startup also queues a catch-up run of the thumbnail, location, provider refresh and Garmin health jobs, so a restart does not wait a full interval.

Module by module

Each section says what triggers the work, what happens locally and when distributed, and what is lost if it is dropped. "Distributed" here means DEPLOYMENT_PROFILE=distributed, which requires JOBS_ENABLED=true, Redis, S3 and a shared lock. A multi-worker local deployment sits between the two: the shared state moves to Redis but the other rules are the local ones unless you also enable durable jobs.

Activity thumbnails

Triggers activity.created (render), activity.deleted (remove), and a change of the tile server (re-render everything, when that option is on)
Local The handler runs on the 2-thread subscriber pool, after the request has returned. A map needs tiles, so a slow tile server delays only thumbnails
Distributed One durable job per subscriber on a worker, with retries. Thumbnails are written to the bucket
If dropped The hourly net renders any activity that still has no thumbnail. The daily sweep removes thumbnails whose activity is gone. The tile-change re-render has no net: it is an admin command, so repeat the settings save

Reverse geocoding

Trigger activity.created
Local Runs on the subscriber pool, throttled by the geocoding rate limit. A lookup that would wait more than a few seconds for its turn, or that the service refuses or fails, is left to the backfill
Distributed Durable job with retries. A lookup that would wait more than a few seconds for its turn, or that the service refuses or fails, runs again when the service allows, without using up a retry. The rate limit holds for every replica and worker together
If dropped The hourly backfill fills in activities that still have no location, in batches that respect the rate limit

Retained files and activity photos

Trigger activity.deleted
Local The handler deletes the blobs on the subscriber pool
Distributed Durable job
If dropped A daily sweep lists the storage area and removes blobs whose activity no longer exists

Notifications

Triggers activity.created, follower requested, accepted and removed
Local Created on the subscriber pool. A hint then goes to the browser in the same process
Distributed Durable job. The hint travels over the Redis channel to whichever replica holds your socket
If dropped The notification is not created and not recreated. The creation is idempotent, so a redelivered event does not duplicate it

Emails: sign-up and password reset

Triggers A sign-up verification, an invitation to choose a password for an account an administrator created, an approval request to administrators, an approved notice, a password reset request, an email-change confirmation, a notice to the previous address after an email change
Local The email is sent from the subscriber pool. A failed send is logged and not retried
Distributed Durable job. A failed send is retried with backoff, which is why email retries need JOBS_ENABLED=true
If dropped Nothing repairs it. A lost reset email is recovered by requesting another, a lost verification email by signing up again or an administrator, a lost email-change confirmation by resending it from Settings. The user and token rows stay the source of truth. A password reset request answers the same for every address and only stages the event; the subscriber decides (account state, the five-minute cooldown, the audit row), so a dropped event leaves no cooldown or audit row behind

Activity uploads, provider refreshes and bulk imports

These are commands. Their state lives on an ingestion job row, which the client polls.

Local After the job row is committed, the work runs on a 2-worker pool that holds up to 100 jobs. A full pool refuses the request with 503 before a job row exists
Distributed The job row and the outbox event commit together and a worker runs the job. A bulk import runs in bounded passes (a time and file budget) and continues itself as the same job, so large imports interleave with other work. A crash resumes from the saved position
Fan-out Every hour one refresh job is queued per user who linked Strava or Garmin Connect. A user whose previous refresh is still running is skipped
After a restart Local with one web worker: at startup every open upload and bulk import is submitted to the pool again (a bulk import continues after its saved position), and a folder import that was still being packed is failed. Provider refreshes are failed instead and the next hourly fan-out queues them again
If dropped A job that stops progressing is failed by the hourly reaper (after 24 hours without progress). A dead-lettered job is closed as failed. The user uploads or starts the import again. Re-running is safe: duplicate detection skips what was already imported

Garmin Connect

Linking The login runs in the background on a pool of 4. The attempt's status (queued, running, succeeded or failed, with mfa_required while the login waits for a code) lives in the shared state, so any replica can answer the status poll. The MFA code is passed through the same shared state. A second attempt while one runs is refused with 409
Gear and health syncs Local: a 2-worker pool holding up to 50. Distributed: a durable job. Only one sync of a kind per user at a time (409 otherwise)
Recurring Every 4 hours, one health sync per linked user, under a lock, so replicas do not repeat it
If dropped A failed sync is marked failed so another can be queued. A dead-lettered sync is closed the same way. The recurring pull covers the last day again

Profile import and export

These are commands with a state record the client polls (an import's state lives 4 hours, an export's 2 hours).

Local Run on a dedicated pool of 1 worker holding 3 jobs, apart from the subscriber pool, because an archive can take minutes. A full pool refuses the request with 503 before the archive is staged
Distributed Durable job. The staged archive is in the bucket, so any worker can take it
If dropped A job the pool could not take is closed as failed and its archive removed. Unused staged archives are swept every 6 hours. The user starts again

Account deletion

This is a command with a state record that an administrator polls (the state lives 2 hours).

Local Runs on a dedicated pool of 1 worker holding 5 deletions, in one pass to the end. A full pool refuses the request with 503
Distributed Durable job. It runs in bounded passes (a third of the job lease, at least 30 seconds) and continues itself as the same job until the activities are gone, then erases the photo and the user. Between passes the deletion stays running and a reactivation still cancels it
If dropped A dead-lettered deletion is closed as failed. The hourly sweep queues a stalled one again, up to 5 times, then an administrator deletes the account again

Server settings

A change to the tile server publishes server_settings.tile_settings_changed in the same transaction as the save, and the thumbnail module reacts. The settings are cached for a few seconds in each process, and the content security policy for a new tile domain reaches the other replicas within about 30 seconds.

Followers and activities

These modules publish events and subscribe to none. activity.created, activity.updated and activity.deleted and the follower events are consumed by the modules above.

Realtime hints

Delivered best effort over the WebSocket, and over a Redis channel when several processes run. See Realtime updates.

A revoked session (sign-out, revocation, password change or reset, a deactivated or deleted account, a detected refresh-token reuse) publishes auth.sessions_revoked when its transaction commits. A bus handler in the websocket module turns it into a control message on the realtime channel, and every API replica closes the sockets of that session with close code 4401. Nothing repairs a lost close: the socket ends with its 15-minute connection lifetime.

Subscriber catalog

Every durable job is one subscriber reacting to one event. The subscriber id is what you see in jasil_processing_jobs.subscriber_id and jasil_event_log.handler_name, and it is stable across refactors, so you can filter on it. See Troubleshooting for the queries, and Events and subscribers for how an id is chosen.

"Runs without durable jobs" says where the handler runs when JOBS_ENABLED=false. A bus handler is queued on the 2-thread subscriber pool and a failure is dropped. A command pool is a bounded pool that the request submits to directly, described in the module's section above. The gear deletion and bulk edit pools run one job at a time and hold 5.

Subscriber id Event Runs without durable jobs Repaired by On dead-letter
activity_thumbnail.generate activity.created Bus Hourly missing-thumbnail job Net fills the gap
activity_thumbnail.cleanup activity.deleted Bus Daily orphan sweep Net fills the gap
activity_thumbnail.settings_changed server_settings.tile_settings_changed Bus Nothing: repeat the settings save Nothing
activity_geocoding.reverse_geocode activity.created Bus Hourly location backfill Net fills the gap
activity_file_storage.cleanup activity.deleted Bus Daily orphan sweep Net fills the gap
activity_media.cleanup activity.deleted Bus Daily orphan sweep Net fills the gap
activity.notify_created activity.created Bus Nothing, by design Nothing
followers.notify_requested follower.requested Bus Nothing, by design Nothing
followers.notify_accepted follower.accepted Bus Nothing, by design Nothing
followers.clear_request_notification follower.request_removed Bus Nothing, by design Nothing
auth.send_sign_up_verification_email auth.sign_up_verification_requested Bus The user signs up again, or an administrator helps Nothing
auth.send_admin_invitation_email auth.admin_invitation_requested Bus The administrator sets a password for the account Nothing
auth.send_email_change_verification_email auth.email_change_verification_requested Bus The user resends the confirmation from Settings, or an administrator helps Nothing
auth.send_email_change_requested_email auth.email_change_requested Bus Nothing: the notice is a courtesy, and the change stays unconfirmed Nothing
auth.send_email_changed_email auth.email_changed Bus Nothing: the notice is a courtesy Nothing
auth.send_email_change_address_taken_email auth.email_change_address_taken Bus Nothing: the notice is a courtesy Nothing
auth.send_sign_up_approval_request_email auth.sign_up_approval_requested Bus Administrators also see the pending user Nothing
auth.notify_admin_sign_up_approval auth.sign_up_approval_requested Bus Administrators also see the pending user Nothing
auth.send_sign_up_approved_email auth.sign_up_approved Bus Nothing: the email is a courtesy Nothing
auth.send_sign_up_existing_account_email auth.sign_up_existing_account Bus Nothing: the email is a courtesy Nothing
auth.send_password_reset_email auth.password_reset_requested Bus The user requests another reset Nothing
auth.revoke_idp_session_tokens auth.sessions_revoked Bus The 15-minute revocation sweep Net fills the gap
auth.refresh_idp_tokens auth.idp_token_refresh_requested Bus Every session refresh queues it again Nothing
websocket.close_sockets auth.sessions_revoked Bus Nothing: a socket ends at its 15-minute lifetime Nothing
auth.notify_idp_linked auth.idp_linked Bus Nothing, by design Nothing
auth.notify_account_security_changed auth.account_security_changed Bus Nothing, by design Nothing
auth.send_account_security_changed_email auth.account_security_changed Bus Nothing: the change is stored and audited Nothing
auth.send_idp_linked_email auth.idp_linked Bus Nothing: the change is stored and audited Nothing
auth.notify_password_reset_completed auth.password_reset_completed Bus Nothing, by design Nothing
auth.send_password_reset_completed_email auth.password_reset_completed Bus Nothing: the reset is stored and audited Nothing
auth.notify_account_locked auth.account_locked Bus Nothing, by design Nothing
auth.send_account_locked_email auth.account_locked Bus Nothing: the lock is audited Nothing
auth.notify_mfa_locked auth.mfa_locked Bus Nothing, by design Nothing
auth.send_mfa_locked_email auth.mfa_locked Bus Nothing: the lock is audited Nothing
auth.notify_step_up_session_ended auth.step_up_session_ended Bus Nothing, by design Nothing
auth.send_step_up_session_ended_email auth.step_up_session_ended Bus Nothing: the session end is audited Nothing
auth.notify_step_up_locked auth.step_up_locked Bus Nothing, by design Nothing
auth.send_step_up_locked_email auth.step_up_locked Bus Nothing: the lock is audited Nothing
auth.notify_new_device_sign_in auth.new_device_sign_in Bus Nothing, by design Nothing
auth.send_new_device_sign_in_email auth.new_device_sign_in Bus Nothing: the sign-in is audited Nothing
auth.notify_backup_code_used auth.backup_code_used Bus Nothing, by design Nothing
auth.send_backup_code_used_email auth.backup_code_used Bus Nothing: the use is audited Nothing
auth.notify_refresh_reuse auth.refresh_reuse_detected Bus Nothing, by design Nothing
auth.send_refresh_reuse_email auth.refresh_reuse_detected Bus Nothing: the reuse is audited Nothing
users.notify_admin_user_promoted users.admin_promoted Bus Nothing, by design Nothing
users.send_admin_promoted_email users.admin_promoted Bus Nothing: the promotion is stored and audited Nothing
users.profile_import_requested users.profile_import_requested Bus, then the profile transfer pool The user starts again Import state closed as failed
users.profile_export_requested users.profile_export_requested Bus, then the profile transfer pool The user starts again Export state closed as failed
activity_ingestion.uploaded_file activity.file_uploaded Ingestion command pool The user uploads again Job marked failed
activity_ingestion.refresh_requested activity.refresh_requested Ingestion command pool The hourly refresh fan-out, or the user Job marked failed
activity_ingestion.bulk_import activity.bulk_import_continue Ingestion command pool Continues itself in bounded passes and resumes from its saved position. The stale-job reaper closes one that stops progressing Job marked failed
garmin.sync_requested garmin.sync_requested Garmin sync pool The 4-hourly health pull, or the user Sync state marked failed
strava.sync_requested strava.sync_requested Strava sync pool The user queues it again Sync state marked failed
gears.deletion_requested gears.deletion_requested Gear deletion pool The user deletes the gear again Deletion state marked failed
activities.visibility_edit_requested activities.visibility_edit_requested Bulk edit pool The user edits again Edit state marked failed
users.deletion_requested users.deletion_requested Account deletion pool The hourly stalled-deletion sweep Deletion state marked failed

Three things follow from the table:

  • A handler that has a repair job is derived work. Losing one costs time, not data.
  • A handler that repairs nothing is either a command the user can repeat or a transient signal such as a notification. Each of these says why in its module's subscriber_registry.py, and a test fails if a durable subscriber declares neither a repair job nor a reason.
  • The ingestion, provider sync, gear deletion, bulk edit and account deletion commands have no bus subscriber. Without durable jobs the service hands them to its own bounded pool instead, which is why a full pool answers the request with 503.

The realtime hint to the browser is not a job. It is delivered over the WebSocket and, with several processes, over a Redis channel; see Realtime updates.

Reading the table in practice

You run Expect
One container, defaults Derived work follows within seconds. A restart can drop queued thumbnails and locations, and the hourly jobs fill them in. Commands that were queued are lost on restart, and their users retry
A few web workers, local Same, with shared state in Redis so limits and live updates work across workers
Durable jobs on one host Retries, a dead-letter list you can inspect and no loss of an event between save and publish. A worker restart costs a delay, not the work
Distributed The above on several hosts, with locks so each scheduled job runs once and a bucket so any replica can serve any file

If something did not appear, look in this order: the logs for the module's warning, the job tables on the durable path, and then wait for the next run of the module's scheduled net.