Pipelines

Workers and pipelines

Due to cache and filesystem configuration, pipelines run on dedicated workers. Depending on the assignment type, Buddy can delegate the pipeline to the worker with the lowest load at the time of the first run, match the worker by pipeline tags, or allow the user to choose the worker manually.

Assigning a pipeline to a new worker erases the pipeline's cache. As a result, the first execution of the pipeline after it's assigned to a new worker can take longer than subsequent runs.

Filesystem, static files and cache

A pipeline is fixed to a specific worker together with its cache and filesystem. This means that whenever you switch a pipeline to another worker, you lose the cache.

The exception are static files uploaded to the filesystem. The static files are stored in the main instance of Buddy On-Premises and are automatically downloaded on the pipeline's first execution on the new worker.

Pipeline assignment types

  1. Fixed – the pipeline is assigned to a specific worker and always runs on that worker. If the worker version is lower than the main instance of Buddy or the worker does not respond (is offline), the pipeline execution will fail.

  2. Dynamic (lowest load and matching tags) – in this case, the pipeline will be automatically reassigned before the execution if one of the following conditions occurs:

    a. the current worker is not available (is offline, is in different version, etc.)
    b. the average load on the current node exceeds 100% per CPU core over the last 5 minutes
    c. the worker's tags do not match the pipeline's tags

Rules for tag matching

  • If the pipeline doesn't have any tags > Buddy looks up all unlocked workers with no tags, and selects the one with the lowest load
  • If the pipeline has tags > Buddy looks up all unlocked workers with at least one matching tag

If no workers are found for both options, an error will appear.

Adding a new pipeline

If there's more than one worker available, a worker assignment setting will appear when adding a new pipeline:

Worker assignment optionsWorker assignment options

There are three options available:

  1. Assign pipeline to worker with lowest load and no tags (default) – Buddy will assign the pipeline to the worker with the lowest load at the time of the first pipeline run. If no workers match the conditions, an error will appear. This option is only available when adding a new pipeline

  2. Assign pipeline to worker with lowest load and matching tags – Buddy will assign the pipeline to the worker with matching tags. If there are more than one, workers with matching tags, it will select the one with the lowest load at the time of the first pipeline run. If no workers match the conditions, an error will appear.

You can set the tags for the worker in the worker details.

  1. Select the worker manually from the list

If you can't see the worker assignment option, your workers are probably locked and not available.

Updating pipelines

Once the pipeline is added, you can change the worker assignment mode in the pipeline settings near the bottom.