Background jobs in Pergola

By default, a Pergola Component is started once and takes care of its own runtime cycle itself. Often Components run as daemons, i.e. supposed to run forever. An example of a daemon Component is a webserver that runs permanently to answer client requests.

But for background work, e.g. data updates, model trainings, nightly computations, etc., Pergola offers scheduled Components. You can define a scheduled Component by adding the scheduled property to your pergola.yaml, e.g. like this:

  scheduled: "0 2 * * *"

It takes either a CRON-expression or a Pergola keyword directive as an argument.

Scheduling with CRON

With CRON expressions, you can schedule your Component to run regularly at a certain time, down to a certain minute. Some examples:

  scheduled: "27 * * * *"

--> Once every hour at the 27th minute, that is at 00:27, 01:27, 02:27 ...

  scheduled: "15 12 * * *"

--> Once every day at 12:15

  scheduled: "33 */2 * * *"

--> Every second hour at the 33rd minute, that is 00:33, 02:33, 04:33 ...

  scheduled: "49 9,12,15 * * *"

--> Every day at 09:49, 12:49 and 15:49

There is a lot more you can do with CRON, e.g. weekday specific behaviour. Please refer to the cron expression or other publicly available resources regarding CRON.

For testing your cron expression, you can use: cronitor

note

All timings are in UTC.

Scheduling with @release

With Pergola you can also schedule a Component to run once every Release (aka at deployment) with the special tag @release:

  scheduled: "@release"

note

Pergola will start a Component marked as @release immediately as for any other non-scheduled Component. The only difference is that such Component will run only once at deployment time, till completion.

In case of a failure, it will be retried up to 3x times with an exponential back-off delay (10s, 20s, 40s).