Concurrency & parallelism

Running tests and deployments in parallel is crucial for high-performing companies that cannot afford long waiting times. In Buddy, the number of pipelines and actions that can be run simultaneously is specified by the number of runners in the workspace plan. 1 runner means that only 1 pipeline can be run at a given time. 2 runners means that you can run 2 pipelines at the same time, or 1 pipeline with 2 parallelized actions, etc. etc.

Pipelines

Pipeline concurrency defines how many pipelines can run at the same time. When the number of triggered pipelines is greater that the number of runners in your plan, the subsequent executions get enqueued and wait for all previously triggered pipelines to finish.

Image loading...

Actions

To speed up tasks like testing and multi-server deployment, actions can be run in parallel. As with the pipelines, the number of actions that can be run simultaneously depends on the chosen plan. For example, if your plan includes 4 runners, it means you have the capability to run either 4 actions in one pipeline, or 2 pipelines with 2 parallel actions concurrently.

Image loading...

Use case: Parallel testing

In this example we'll create a pipeline which splits test files into three packages and runs them in parallel:

Image loading...

Configuration

1. Create a new pipeline in your project.

2. Choose the Split Test Files action from the action list:

   Image loading...

3. Select the number of groups (packages) into which you want to split the tests. In this example, we’ll use 3 packages.

4. Choose if you want to sort them alphabetically or by file size.

5. Provide the path to the directory with test files. You can do it in two ways:

   a. Select one file with the paths to split (each path in a separate line)
   b. Use glob patterns:

   • * – matches any sequence of characters (excluding path separators)
   • ** – matches any sequence of characters (including path separators)
   • ? – matches any single character (excluding path separators)
   • [abc] – matches any character (excluding path separators) against characters in brackets
   • {foo,bar,...} – matches a sequence of characters for alternatives in the braces

6. Once done, save the changes:

   Image loading...

7. The action creates environment variables required to run the tests in parallel: $BUDDY_SPLIT_1, $BUDDY_SPLIT_2, $BUDDY_SPLIT_3, etc. etc. Once the pipeline is executed, you can see the variables in the logs:

   Image loading...

8. Next, you need to prepare the environment for the tests (install dependencies etc.) in the build action:

   Image loading...

9. The last part is adding an action that will run the tests using the proper variable from the Split Test Files action:

   Image loading...