Workers

Workers

  • Workers are used to distribute the load of multiple pipelines you run.
  • The server where Buddy On-Premises is first installed is the default worker of the instance.
  • You can install more workers on other hosts for extra performance.

Requirements

ResourceRequirement
CPU2.6 GHz
Memory4 GB
Root disk30 GB

Workers require a Linux-type server (we recommend Ubuntu 18.04) with Docker and docker-compose installed to run.

The workers and the main Buddy instance must be able to connect to each other. You can configure the IP address and ports of the worker during the installation.

Installation

You can't install a new worker on the server that runs the main Buddy On-Premises instance. This instance is a fully operational worker. Install new workers on different, compatible servers.

  1. Sign in to your Buddy On-Premises instance as admin.
  2. Go to the On-Premises Admin panel and switch to the Workers tab.
  3. Click Add new worker and copy the generated installation command.
  4. Launch a terminal on the server where you want to install the worker and run the command you copied. The command installs and runs the worker, and registers it in the main instance.

You can also generate the installation command using the CLI on the server with the main instance by running sudo buddy install-worker. This option allows for additional configuration.

Non-interactive Installation

You can automate creating and launching workers. Create an executable script that runs the installation in the non-interactive mode.

Installation commands created using buddy install-worker are valid for 30 minutes. When this time passes, you must generate a new command to install a worker.

Follow these steps:

  1. On the server with the main instance, generate the installation command and token and save it to an executable file:
buddy --yes install-worker > install_worker.sh
$
  1. Upload install_worker.sh to the worker server.
  2. Set up permissions and run the installation script:
chmod +x run.sh
./install_worker.sh
$$

Updating workers

Updating workers follows the same process as updating the main Buddy On-Premises instance. Sign in to the server that hosts the worker and start by running this command:

sudo buddy update
$

The version of the worker and the main instance must be the same. If the versions are different, you can't assign new pipelines to the worker, and all pipelines assigned previously will fail with the "Wrong version" status.

Backing up workers

Backing up workers follows the same process as backing up the main Buddy On-Premises instance.

Managing workers

You can view the list of connected workers in the Workers tab of the On-Premises Admin panel:

Workers tabWorkers tab

Each worker has a status: - Testing – the main instance is running a connection test with the worker. - Error – unable to connect to the worker. Check the worker status and/or firewall settings. - Wrong version – the working is running, but its version is different from the main machine. Upgrade both the worker and the main instance to the same version. - Connected – the worker is running and ready to use.

Locking workers

You can lock a worker and change its name in the edit options. Locked workers cannot be assigned to new pipelines.

Locking workersLocking workers

Workers and pipelines

Due to cache and filesystem configuration, pipelines run on a dedicated workers. You can assign a pipeline to a specific worker in the Pipeline settings. If you don't assign a pipeline manually, Buddy delegates it to the worker with the lowest load at the time of the first execution.

Assigning a pipeline to a new worker erases the pipeline's cache. As a result, the first execution of the pipeline after it's assigned to a new worker can take longer than subsequent runs.

Assigning YAML-configured pipelines to workers

To define the worker on which the pipeline should be launched in YAML, add the worker field to the pipeline configuration:

Defining workers in pipelineDefining workers in pipeline

You can view the list of connected workers in the Workers tab of the On-Premises Admin panel:

Workers tabWorkers tab

Load-based pipeline reassignment

By default, when you assign a pipeline to a specific worker, all executions of that pipeline are handled exclusively by the selected worker.

If you want to facilitate better workload distribution among workers, you can configure Buddy to reassign pipelines to other workers if the current worker is under excessive load. When the pipeline is reassigned to a different worker, that worker becomes the new default worker for that pipeline. If the worker experiences excessive load again, Buddy will move the pipeline to a different worker again.

Buddy transfers the pipeline to a less busy worker if the average load on the current node exceeds 100% per CPU core over the last 5 minutes. For example, on a worker with 16 cores, the load must exceed 16.

When Buddy moves a pipeline to a different worker, the pipeline loses its cache and runs from scratch.

Configuration through CLI

  1. Sign in to the machine where Buddy On-Premises is installed.
  2. Run this command:

    sudo buddy configure
    $
  3. Select option 16. Pipelines worker assignment. This starts the wizard and shows the current status of the feature.
  4. Press the O key and hit Enter to enable. Press F and hit Enter to disable.
  5. Restart the application. Press Y and hit Enter.

Configuration through the UI

  1. Sign in to your Buddy On-Premises instance as admin.
  2. Go to the On-Premises Admin panel and switch to the Performance tab.
  3. Find the Resources per pipeline run section.
  4. Tick the box under Pipeline distribution policy to enable the feature.
  5. Click Save changes.

Configuration through UIConfiguration through UI

Offline workers

If a worker is offline or in a different version than the main instance, all pipeline executions end with an error. It is also impossible to browse the filesystems and view execution logs of the pipelines assigned to that worker.

You can remove an inactive worker from the worker list. When you remove a worker, pipelines assigned to that worker will be automatically assigned to a differebt worker on the next execution.

Filesystem, static files and cache

A pipeline is fixed to a specific worker together with its cache and filesystem. This means that whenever you switch a pipeline to another worker, you lose the cache.

The exception are static files uploaded to the filesystem. The static files are stored in the main instance of Buddy On-Premises and are automatically downloaded on the pipeline's first execution on the new worker.

Solving connection problems

To work properly, the worker and the main instance must be establish a two-way connection. This requires the IP addresses and ports used for connecting to be configured properly.

By default, the worker connects to the public IP address of the standalone (the address that you use to open Buddy in your browser). The default ports are 1090 and 1091.

If the connection between a worker and the main instance doesn't work, adjust the settings of the worker. Run sudo buddy configure on the worker server and adjust the relevant options.

You can also remove the worker and create an installation command with updated settings using the wizard, and install it again. To do that, run this command on the server with the main instance:

sudo buddy install-worker
$