Skip to main content

Project Manifest

A Project Manifest is a file that is managed as part of your source code. It defines the entire application stack of your Project so it can be automatically built, deployed, and operated on the Pergola platform.

The Manifest file is placed within the root folder of your source tree and can be either a YAML or a JSON file. For YAML, allowed filenames are pergola.yaml or pergola.yml. For JSON, it is pergola.json.

tip

You can validate your Manifest file locally, before even committing to Git.

Example Manifest file

version: v1

components:
- name: web-api
docker:
file: Dockerfile
build-context: . # default is .
build-args:
- name: ARG1
value: MY_VALUE1
- name: ARG2
value: MY_VALUE2
env:
# static env variable
- name: API_VERSION
value: v1
# variable from required config
- name: PUBLIC_FQDN_ROOT
config-ref: public_hostname
# variable from optional config with default
- name: LOG_LEVEL
config-ref: loglevel
value: info #default
- name: DB_HOST
component-ref: db
- name: DB_PORT
value: "5432"
- name: DB_USER
config-ref: db.user
- name: DB_PWD
config-ref: db.pwd
files:
# map file from required config
- path: /etc/web-api/config.json
config-ref: web-api-config.json
# map file providing inline (static) content
- path: /var/lib/web-api/static-content.xml
content: |
<?xml version="1.0" encoding="UTF-8"?>
<note>
<to>Tove</to>
<from>Jani</from>
<heading>Reminder</heading>
<body>Don't forget me this weekend!</body>
</note>
scaling:
min: 2
max: 5
ports:
- 8080
- 9091
ingresses:
- host: api
path: /v1
port: 8080

- name: db
docker:
image: postgres:16
ports:
- 5432
env:
- name: POSTGRES_PASSWORD
config-ref: postgres_initial_pwd
resources:
cpu: 500m
memory: 2Gi
storage:
- name: pgdata
path: /var/lib/postgresql/data
size: 50Gi

- name: job-regularly
docker:
file: job1/Dockerfile
command:
- "./run.sh"
args:
- "--db=$DB_HOST:$DB_PORT"
- "crunch"
env:
- name: DB_HOST
component-ref: db
- name: DB_PORT
value: "5432"
scheduled: "0 2 * * *"
identity: "data-processor"

- name: job-once
docker:
file: job2/Dockerfile
scheduled: "@release"

Version

Every Manifest file needs a version. Currently, only v1 is supported. Future releases of Pergola will support different flavours and capabilities of the Manifest file, distinguished by unique version numbers.

version: v1

Components

A Project needs at least one Component. A Component is the smallest unit which is built and runs on the Pergola platform. A typical Project usually consists of multiple Components, e.g. a dashboard, a database, scheduled job(s), etc., which all together form your application stack.

Per Component, you describe how to build and run it. The minimum required fields are:

  • name of your Component
  • docker that points to a Dockerfile or to an existing container image
components:
- name: dashboard
docker:
file: Dockerfile

The name must be unique within the Manifest. Allowed are lowercase letters (a-z), numbers (0-9) and hyphens (-), starts with a letter and cannot end with a hyphen.

Renaming Components

As each Component is identified by its unique name, renaming it consequently results in a "new" Component. Technically, any previously deployed instances of that Component with the old name will be shut down and new ones created under the new name.

This also has an impact on persistent data. Any Storage attached to a Component is exclusively owned by that Component, thus a "new" Component always starts with fresh data. However, any storage which was attached to the "old" Component will be retained and automatically reclaimed as soon as that Component is added back. This ensures that persistent data is not lost if you change the code branch or use different versions with a changed set of Components.

Docker

Within the docker field you either define the path to the Dockerfile to be built:

docker:
file: Dockerfile

whereas, file can also be a relative path to a specific Dockerfile within your code tree:

docker:
file: path/to/some_other_docker.file

Or, you reference an existing container image from a publicly available repository:

docker:
image: python:3.10.0-alpine

or fully qualified pointing to a specific container registry:

docker:
image: public.ecr.aws/pergola/postgres:16

Build context

The build-context is a relative path within your source tree that refers to the files and directories that will be available during build. Anything not included in the build context won’t be accessible to commands in your Dockerfile.

docker:
file: Dockerfile
build-context: path/to/context

In the example above, the Dockerfile is located at the root of your source tree and will see only the files under the path/to/context.

For example, a COPY file.txt /some/path/ instruction within the Dockerfile above will copy the path/to/context/file.txt from your source tree to /some/path/file.txt within the final container image.

Build arguments

build-args is a list of key-value pairs that are used as build-time variables:

docker:
file: Dockerfile
build-args:
- name: CLUSTER_ENABLED
value: "true"
- name: BUILD_FLAGS
value: "-z -o --raft"

Any variable defined here should be also declared (and used) in the referenced Dockerfile via the ARG instruction. These variables only exist during the build process and do not persist in the final image.

info

Only strings can be passed as values. For values like boolean or numbers make sure they are quoted as strings.

For example, this is wrong:

docker:
build-args:
- name: A_NUMBER
value: 1234
- name: A_BOOL
value: true

This is correct:

docker:
build-args:
- name: A_NUMBER
value: "1234"
- name: A_BOOL
value: "true"

Command and runtime arguments

If you want to override your Dockerfile's start command (ENTRYPOINT) and/or arguments (CMD), you can use the command and args fields in the Manifest:

command:
- "./run.sh"
args:
- "--db=$DB_HOST:$DB_PORT"
- "crunch"
  • command field specifies the actual command run by your Component
  • args field specifies the arguments passed to the command

When you specify command and/or args, these rules apply:

  • only strings are allowed; if you have a single number or a boolean (i.e. as args, or in very strange cases even a command), make sure they are quoted as strings, e.g. "1234" or "true" respectively
  • if you supply a command, regardless with or without args, any ENTRYPOINT or CMD defined in your Dockerfile are ignored; only the supplied command, and if provided the args, will start your Component
  • if you supply args only, any CMD defined in your Dockerfile is ignored; your Component will start with the ENTRYPOINT (if defined in your Dockerfile) and the args defined in your Manifest
  • if you do not supply command or args, the defaults (ENTRYPOINT and/or CMD) defined in your Dockerfile are used to start your Component

Examples:

ManifestDockerfileActual command run
command: ["/cmdM"]
args: ["foo", "bar"]
ENTRYPOINT ["/cmdD"]
CMD ["zoo", "boo"]
/cmdM foo bar
command: ["/cmdM"]ENTRYPOINT ["/cmdD"]
CMD ["zoo", "boo"]
/cmdM
args: ["foo", "bar"]ENTRYPOINT ["/cmdD"]
CMD ["zoo", "boo"]
/cmdD foo bar
<nothing specified>ENTRYPOINT ["/cmdD"]
CMD ["zoo", "boo"]
/cmdD zoo boo

Runtime user and group

If your Component runs as a specific user (UID) or optionally requires a specific group (GID) at runtime, you can specify them within the Manifest under user.

info

Only UID (and optionally with colon separated the GID) is allowed. You cannot provide the user name or the group name, even if they are known within the built Docker image. You have to provide the ID in digits.

Run as user 1001:

user: 1001

The Component's main process will run as user 1001. Any Storage attached will be read/write accessible by the same user 1001.

Or, as user 1001 and group 3003:

user: 1001:3003

The Component's main process will run as user 1001, and the user will be member of the group 3003. Any Storage attached will be read/write accessible by all members of the group 3003.

This requires that your Docker image supports running as a specific user, or maybe even enforces a specific UID/GID. Either check the documentation of your base image or specify a runtime user within your Dockerfile.

info

If you do not specify a user, neither in your Dockerfile nor in your Manifest, your Component will most probably run as root.

Running containers as non-root is a security best practice and is highly recommended. If not absolutely necessary, ensure your Dockerfile requires a non-root user at runtime, or choose a non-root / unprivileged base image as a starting point. Then use the same user (UID and optionally the GID) in your Manifest.

When you provide a user within your Manifest, Pergola will automatically apply the most appropriate security context to your Component.

Environment variables

Use env to provide environment variables to your Component at runtime.

Each environment variable has a name and a value. Only strings are allowed, for both. In case you have a number or a boolean as value, make sure they are quoted as strings, e.g. "1234" or "true" respectively.

Environment variables you provide here will override variables that are defined in the Dockerfile or the container image of your Component.

Literal values

A simple environment variable could be:

env:
- name: API_VERSION
value: v1

Configuration driven values

In most cases you don't want to hardcode the value but rather use a different one per Stage, e.g. log_level=debug on DEV, but log_level=info on LIVE environments. In order to achive that, you reference a Configuration entry via the config-ref which will be resolved to the actual value at Release based on the Stage you deploy to:

env:
- name: PUBLIC_FQDN_ROOT
config-ref: public_hostname

In the example above the value of PUBLIC_FQDN_ROOT will be resolved to a fully qualified domain name or host based on the configuration entry public_hostname, which is most probably different per Stage. For further details, see Configuration Management.

Any config-ref defined here must be resolvable at Release, means the Configuration selected must satisfy all referenced entries here, otherwise the Release will be rejected.

If you have a config-ref which might not be served by a Configuration entry (is optional), you can define a default value:

env:
- name: LOG_LEVEL
config-ref: loglevel
value: info

The LOG_LEVEL above will be either set by a Configuration, or will default to info if no Configuration entry defined.

Value derived from a Component's name

If you need a reference to another Component within the same Manifest, you can use component-ref and store its actual value (which is the runtime hostname of that Component) in an environment variable:

components:
- name: web-api
env:
- name: DB_HOST
component-ref: db
- name: db
docker:
image: postgres:16

The web-api above will have the hostname of the Component db stored in its environment variable DB_HOST. This requires that the referenced Component (db) has at least one port exposed.

Referencing other environment variables

Environment variables within same Component may reference each other, however ordering is important. Variables making use of others defined in the same Component must come later in the list. Similarly, avoid circular references.

env:
- name: REDIS_HOST_REF
component-ref: redis
- name: REDIS_CONNECT_URI
value: "redis://$(REDIS_HOST_REF):6379"

In the example above, the variable REDIS_HOST_REF is first resolved to the runtime hostname of the redis Component. Then the REDIS_CONNECT_URI references that hostname to construct the actual connection URI to Redis.

Files

Files can be mapped to desired paths within a Component, e.g. to serve environment specific configuration files, secrets or even binary files, as required at runtime.

Served from Configuration

A file can be served from a Configuration entry via the config-ref and mapped to its desired path within the Component:

files:
- path: /etc/web-api/config.json
config-ref: web-api-config.json

For further details, see Configuration Management.

Static content

It is also possible to provide a file inlined as a static content:

files:
- path: /var/lib/web-api/static-content.xml
content: |
<?xml version="1.0" encoding="UTF-8"?>
<note>
<to>Tove</to>
<from>Jani</from>
<heading>Reminder</heading>
<body>Don't forget me this weekend!</body>
</note>

The content must be a string. In YAML you can also use multiline strings. In JSON you have to escape special characters and newlines accordingly.

In case you need a binary content, you can base64 encode its content and put it here. The Component is then responsible for decoding it back into its original binary format at runtime before using it.

Storage

A Storage is a volume attached to your Component. Data on a persistent storage will survive Component restarts and is also retained if you suspend the complete Stage. Whereas, data on an ephemeral storage is lost as soon as its Component is gone.

A storage is always exclusively owned by a Component, more precisely by an instance of a Component. If you are running multiple instances of your Component, see Scaling below, each of them will have its own volume attached. They are not shared across Components or instances of a Component.

Each storage requires:

  • a unique name
  • a path where to mount the volume within the Component
  • and a desired minimum size for the volume

With the optional attribute type you can choose between a persistent and an ephemeral storage:

  • standard: persistent disk storage for data that is retained between releases and restarts, e.g. for databases
  • temporary: ephemeral disk storage, e.g. for local caching; may be still around after a restart, but not guaranteed
  • memory: ephemeral in-memory (RAM) storage, e.g. for shared memory (like /dev/shm); definitely gone after a restart

If type is not specified, standard is the default. You cannot change the type of a storage once released.

storage:
# persistent storage for Postgres
- name: pgdata
path: /var/lib/postgresql/data
size: 50Gi
# more shared memory, more parallel queries
- name: dshm
path: /dev/shm
size: 2Gi
type: memory

The name must be unique within the Component. Allowed are lowercase letters (a-z), numbers (0-9) and hyphens (-), starts with a letter and cannot end with a hyphen.

The size is measured in bytes. You can express volume size using one of these quantity suffixes: M (mega), G (giga), T (tera), or in binary: Mi (mebi), Gi (gibi), Ti (tebi).

Examples

1G means 1 gigabyte which equals to 1000 megabytes, or 1000000 kilobytes.

1Gi means 1 gibibyte which equals to 1024 mebibytes, or 1048576 kibibytes.

See binary units for further details.

caution

The suffixes are case-sensitive, i.e. 1000M means 1000 megabytes, while 1000m would mean 1000 millibytes which equals to one byte.

For persistent storages (i.e. standard), you cannot shrink the size of an existing storage. If the actual size is larger than the one defined within the Manifest, e.g. due to previous deployments with a different volume size, the actual size takes precedence.

Ports

In order to provide access to your Component within your Project, you can define its ports to be exposed:

ports:
- 8080
- 9091

For further details, see the Linking Components tutorial.

Resources

For each Component, you can optionally specify how much resources it needs at runtime. The resources you can specify are cpu and/or memory:

resources:
cpu: 500m
memory: 2Gi

CPU

You can express cpu in total units, like "1", or in millis, like 100m. 1 CPU is equivalent to one virtual core. Whereas 100m is a fraction of a CPU and is equivalent to 0.1 (10% of a virtual core).

For total units, like 2 CPUs, make sure the value is quoted as string:

resources:
cpu: "2"
info

If you do not specify CPU requirements, your Component will run on a best-effort basis regarding CPU usage. Means, it can still utilize as much CPU as it needs and as available on the infrastructure, but it might be ousted by other Components with explicit CPU requirements when the underlying compute node is under CPU pressure.

Memory

memory is measured in bytes. You can express memory using one of these quantity suffixes: k (kilo), M (mega), G (giga), or in binary: Ki (kibi), Mi (mebi), Gi (gibi).

Examples

1G means 1 gigabyte which equals to 1000 megabytes, or 1000000 kilobytes.

1Gi means 1 gibibyte which equals to 1024 mebibytes, or 1048576 kibibytes.

See binary units for further details.

caution

The suffixes are case-sensitive, i.e. 1000M means 1000 megabytes, while 1000m would mean 1000 millibytes which equals to one byte.

info

If you do not specify memory requirements, your Component will run on a best-effort basis regarding memory usage. Means, it can still utilize as much memory as it needs and as available on the infrastructure, but it might be ousted by other Components with explicit memory requirements when the underlying compute node is under memory pressure.

Scaling

If you need more than one instance of your Component running you can define scaling boundaries. Depending on the usage of infrastructure resources it will dynamically scale within given boundaries. Given that, the scaling is only reliable if you also have defined proper resources.

scaling:
min: 2
max: 5

min represents the minimum amount of instances of your Component running for sure, and max is the maximum amount of instances which will be running when needed. The values are integers and must not be quoted.

Ingresses

An ingress exposes your Component to the web, via http/https. Pergola automatically provides a unique URL for each ingress of your Component which you can share with your users, or use in other (external) applications to access your Component's web API.

ingresses:
- host: api
path: /v1
port: 8080

host defines the host prefix for the URL to be generated. This field is mandatory.

path is optional and defaults to /. Defining a path can be useful if you want to distinquish different API versions, like /v1 vs. /v2, or when you need separate access points within your app, e.g. /dashboard vs. /admin, or when your Component just simply accepts incoming requests on a specific path only.

Uniqueness

The combination of host and path within a Manifest must be unique.

port defines the internal (target) port of your Component as defined under ports. This field is optional, if your Component exposes one port only. If your Component exposes multiple ports, then the specific one to be served via ingress must be declared here.

For further details, see the Web Component tutorial.

Scheduling

You can schedule Components, so they do not run all the time (daemon) but are executed on a schedule and are shut down once they run to completion.

Regularly

To run a Component on a regular basis, you provide the scheduled field with the desired configuration as a cron expression:

scheduled: "0 2 * * *"
note

All timings are in UTC.

For further details, see the Scheduled Components tutorial.

Once per Release

Components can be also scheduled to run for each Release:

scheduled: "@release"

This will trigger the Component to be executed exactly once, whenever there is a new Release. This is useful for example for database alterations, for notifying external workflows, etc.

For further details, see the Scheduled Components tutorial.

Automatic retries

When a scheduled Component fails, it will be automatically retried up to 3 times (default setting), so in total it will run max. 4 times before it is considered as failed.

You can specify the maximum amount of retries or disable it completely with the max-retries attribute:

# retry up to 5 times
- name: job-regularly
scheduled: "0 2 * * *"
max-retries: 5

# disable retries, run exactly once only
- name: job-once
scheduled: "@release"
max-retries: 0

Identity

An identity enables access from within a Pergola Component to cloud resources (e.g. Google BigQuery or AWS S3) via cloud provider’s native IAM entities (e.g. a GCP Service Account or an AWS IAM Role) at runtime.

identity: "data-processor"

The identity defined here is first of all a logical name. It does not grant access to cloud resources on its own. Only when you link this identity with an actual native cloud IAM at configuration time, your Component will inherit the cloud IAM permissions at next Release.

For further details, see Identity management.