Task Scheduler

The Task Scheduler lets you run Tasks automatically on a recurring cron-based schedule, without requiring a manual trigger from the GUI or REST API. The Task Scheduler consists of two components working together:

• Scheduler loop — checks all enabled Schedules every 60 seconds (configurable). When a Schedule’s cron expression fires, the assigned Tasks are added to a job queue.
• Worker loop — drains the job queue every 3 seconds (configurable), executing one Task at a time. This single-threaded design prevents concurrent writes to the shared staging table.

Both loops run embedded in the API process by default and can be disabled or moved to a separate Docker service.

Note

A Task in Katalogue is the definition of a piece of work, while a Job is an executed instance of a task.

Managing Schedules

Schedules are managed in the Schedules view (/schedules), available to admin users.

Schedule Properties

Property	Description
Name	Unique name for the schedule
Description	Optional description
Cron Expression	Standard 5-part cron expression (e.g. 0 2 * * *)
Tasks	One or more Tasks assigned to this schedule
Enabled	Toggle to pause a schedule without deleting it

The human-readable translation of the cron expression (e.g. “At 02:00 AM, every day”) is shown below the field while editing and as a column in the Schedules table.

Cron Expression Format

Katalogue supports both 5-field (standard) and 6-field (with seconds) expressions.

5-field (standard):

┌─── minute (0–59)
│ ┌─── hour (0–23)
│ │ ┌─── day of month (1–31)
│ │ │ ┌─── month (1–12)
│ │ │ │ ┌─── day of week (0–7, 0 and 7 = Sunday)
│ │ │ │ │
* * * * *

6-field (with seconds):

┌─── second (0–59)
│ ┌─── minute (0–59)
│ │ ┌─── hour (0–23)
│ │ │ ┌─── day of month (1–31)
│ │ │ │ ┌─── month (1–12)
│ │ │ │ │ ┌─── day of week (0–7, 0 and 7 = Sunday)
│ │ │ │ │ │
* * * * * *

Examples

Expression	Meaning
0 2 * * *	Every day at 02:00
*/15 * * * *	Every 15 minutes
0 6 * * 1	Every Monday at 06:00
0 0 1 * *	First day of every month at midnight
0 30 2 * * *	Every day at 02:30:00 (6-field with seconds)
*/30 * * * * *	Every 30 seconds

See crontab.guru for more help when constructing cron expressions.

Skip Logic

The scheduler skips enqueueing a Task if it is already running or queued. This prevents duplicate executions and concurrent writes to the staging table. The skipped task will be picked up at the next cron tick.

Queue Ordering

When multiple Tasks are in the queue, the worker picks the next one using this order:

1. Execution method — manually triggered (or externally triggered via REST API) Tasks always run before scheduler-triggered Tasks, regardless of priority. A user-initiated run is never held behind scheduled work.
2. Task priority (task_priority field, higher number = higher priority, default 100) — ranks Tasks within the same execution method. Set per Task in the Task edit form.
3. Arrival time — the oldest queue entry runs first (FIFO).
4. Insertion order — when multiple Tasks are enqueued at the same instant (e.g. a scheduler tick that fires several Tasks at once), they are processed in the order they were inserted into the queue. The scheduler inserts Tasks alphabetically by task name (in ascending order), so within a single tick the effective order is alphabetical.

For example: a scheduled Task with priority 200 waits behind a manually triggered Task with priority 100.

Cancelling a Queued Task

When a Task has been enqueued by the scheduler (status shows as Queued in the Tasks view), it can be cancelled before the worker picks it up:

1. Select the Task in the Tasks view.
2. Click Cancel.

This removes the Task from the queue. The next scheduler tick may re-enqueue it if the cron expression fires again.

Auto-Disable on Repeated Failures

To prevent a persistently broken Task from repeatedly occupying the scheduler and blocking other work, Katalogue will automatically disable a Task after 5 consecutive scheduled failures.

This feature can be controlled by SCHEDULER_AUTO_DISABLE_TASK_FAILURE_COUNT. Set this to 0 or a negative value to allow unlimited failures without auto-disabling.

Behaviour

• The failure counter increments only when a Task fails and was triggered by the scheduler. Manually triggered runs do not affect the counter.
• The counter resets to 0 on any successful completion, regardless of how the Task was triggered.
• When the counter reaches the configured threshold, the Task is set to Disabled and will no longer be enqueued by the scheduler.
• If email notifications are configured, a notification is sent to the system email address(es) when a Task is auto-disabled. See Email Notifications for setup details.

Re-enabling a Task

After resolving the underlying issue:

1. Go to the Tasks view.
2. Select the Task and click Edit.
3. Toggle Status back to Enabled.

Re-enabling the task resets the failure counter to 0, so the Task will not be immediately re-disabled on its next failure.

The tooltip on the Disabled status pill — both in the Tasks list and on the Task detail page — shows the number of consecutive failures that triggered the auto-disable.

Configuration

The scheduler and worker are controlled by the following Config Parameter Reference:

Parameter	Default	Description
SCHEDULER_IS_ENABLED	true	Enable/disable the scheduler loop
WORKER_IS_ENABLED	true	Enable/disable the worker loop
SCHEDULER_POLL_INTERVAL_SECONDS	60	How often the scheduler checks for due schedules
WORKER_POLL_INTERVAL_SECONDS	3	How often the worker polls the job queue
SCHEDULER_AUTO_DISABLE_TASK_FAILURE_COUNT	5	Number of consecutive scheduled failures before a Task is automatically disabled. Set to 0 or a negative value to disable this feature.

Docker Deployment

The Task Scheduler can be run embedded in the api service or as a separate worker service.

The services use the same Dockerfile (services/api/Dockerfile), which uses a multi-stage build with three named targets. Choose the target that matches your deployment:

Target	Dockerfile Contents	Use when
embedded	API + all connector system deps	Default — scheduler and worker run inside the API container
worker	Connector system deps only, no HTTP server	Dedicated background worker container
api	HTTP API only, no connector system deps	API container in a split deployment where a separate worker handles all connector calls

Embedded (Default)

Run the scheduler and worker loop embedded in the api service. This is the simplest mode with no need to maintain a separate worker service. However, this means that the worker and api share resources, which might cause performance issues in the GUI if heavy jobs like datasource sync tasks are frequently run during high-traffic hours.

This deployment mode is the default, no extra configuration needed. The included docker-compose.yml uses the embedded target by default and starts both loops automatically:

docker compose up

Dedicated Worker Container

For resource isolation, the worker can run as a separate container.

The included docker-compose.yml defines the worker service behind a Docker Compose profile, so it only starts when explicitly activated.

When using a dedicated worker, switch the API to the lean api build target and disable both loops to avoid double-processing:

API_BUILD_TARGET=api SCHEDULER_IS_ENABLED=false WORKER_IS_ENABLED=false docker compose --profile worker up --build

--profile worker starts the worker container (built from the worker target, which includes all connector system deps). API_BUILD_TARGET=api switches the api container to the lean build that omits connector system deps. SCHEDULER_IS_ENABLED=false and WORKER_IS_ENABLED=false hand off scheduling and execution to the worker.

Deployment Mode Reference

Scenario	API env	Worker container
Embedded (default)	SCHEDULER=true, WORKER=true	—
Dedicated worker container	SCHEDULER=false, WORKER=false	SCHEDULER=true, WORKER=true
Scheduler in API, worker separate	SCHEDULER=true, WORKER=false	SCHEDULER=false, WORKER=true
No scheduler (fully external)	SCHEDULER=false, WORKER=true	—

Danger

There must be exactly one worker loop enabled at a time since the Katalogue architecture is single-threaded.

It is technically possible to setup multiple worker containers or run both a dedicated worker container and the worker embedded in the api service, but running Tasks in parallell will cause conflicts and might result in data corruption or data loss.

Having no worker loop enabled means that Tasks will be stuck in the queued state forever and never run.

Using External Schedulers

It is possible to use an external scheduler such as Airflow or Kubernetes CronJobs to trigger Tasks via the REST API. The principle is to trigger tasks, get a related job_id in response and use that to poll the job endpoint to follow progress of the job.

These are the most relevant endpoints, see the REST API reference for details:

Method	Endpoint	Description
GET	task/all	List all tasks
POST	task/run	Run one or more tasks by including {"task_ids": [1, 2, 3, ...]} in the request body. Returns an array of task_id and job_id mappings: {runs: [{"task_id": 1, "job_id": 1}]}
POST	task/abort	Abort one or more tasks by including {"task_ids": [1, 2, 3, ...]} in the request body)
GET	job/:job_id	Get a single job by ID. Poll regularly as long as status is “queued” or “running” until the job completes.

External schedulers does not really affect the scheduler deployment mode. An external scheduler will only trigger Tasks. The Katalogue worker loop is still required to actually run the Job. The worker loop might be running in either embedded mode or as a separate service.

The Katalogue scheduler loop might be completely disabled, but doing so means that all tasks (internal maintenance tasks, Microsoft Entra Id sync tasks etc) must be controlled by the external scheduler. The recommendation is to keep the scheduler loop enabled and use it for internal tasks, while the external scheduler controls datasource sync Tasks.

Direct Database Access

Caution

Direct writes to the schedule table bypass the API and will not automatically recompute schedule_next_run_timestamp. This column is the authoritative value the scheduler uses to determine when a schedule is due — it is set on create and kept current by the API on every update that changes the cron expression, timezone, or enabled state.

If you insert or update a schedule row directly in the database and set schedule_is_enabled = true, you must also set schedule_next_run_timestamp to the correct next occurrence:

• A NULL value is self-healed: the scheduler computes and stores the correct future time on its next tick without enqueueing any tasks.
• A stale value in the past is indistinguishable from a legitimately-due schedule and will trigger a one-time immediate fire before self-correcting.