How to scale Buddy on AWS with Terraform

In this guide, we show how to create a pipeline that automatically scales Buddy self-hosted by adding or removing extra runners in AWS infrastructure using Terraform.

Warning

The best idea is to use the guide as a reference for configuring your own scaling process, as each company is different and has individual requirements.

About runner variables

The guide is based on runner variables, a new type of env vars that describe the behavior of Buddy runners. The variables are generated at the beginning of every pipeline run and are available for all actions in the pipeline. You can use them to calculate how many runners you need to serve your workload, send current load to CloudWatch, Datadog, or New Relic, or notify your team on a Telegram channel.

Success

In this example, we use the variables in a script which updates Terraform configuration and launches (or shuts down) EC2 instances in AWS.

Name	Value
BUDDY_RUNNERS The JSON with the information about all installed runners	`{"runners":[{"name":"Primary","address":"build-server","status":"RUNNING","load":0.56,"free_slots":4,"tag":"NOT_TAGGED","locked":false}],"tags":[{"name":"NOT_TAGGED","avg_load":0.56,"free_slots":4,"runners_quantity":1}]}`
BUDDY_RUNNERS_ADDRESS_${TAG} The list of IP addresses of runners tagged with `${TAG}`	`192.168.4.11`
BUDDY_RUNNERS_ADDRESS_NOT_TAGGED The list of IP addresses of untagged runners	`build-server`
BUDDY_RUNNERS_AVG_LOAD_${TAG} The average load from the last minute on all runners tagged with `${TAG}`	`0.69`
BUDDY_RUNNERS_AVG_LOAD_NOT_TAGGED The average load from the last minute on all untagged runners	`2.03`
BUDDY_RUNNERS_CONCURRENT_SLOTS The total number of pipeline slots across all runners	`4`
BUDDY_RUNNERS_COUNT_${TAG} The number of runners tagged with `${TAG}`	`1`
BUDDY_RUNNERS_COUNT_NOT_TAGGED The number of untagged runners	`1`
BUDDY_RUNNERS_FREE_SLOTS_${TAG} The number of free pipeline slots on runners tagged with `${TAG}`	`4`
BUDDY_RUNNERS_FREE_SLOTS_NOT_TAGGED The number of free pipeline slots on all untagged runners	`4`

Hint

Click here for full list of Buddy environment variables.

Part 1: Project configuration

1. Fork repository

Success

The first step is forking https://github.com/buddy/runners-scale.

The repository contains two things:

Runner calculator (action.yml) – a custom action which calculates the load in your on-premises network. The output of the action can be sent as a variable in a notification when the load exceeds the limit. You can also use it in a script that will add or remove runners when necessary (see below).
Scaling pipeline (/example_pipeline) – a preconfigured pipeline which uses Terraform to launch or shut down EC2 instances basing on the calculator's output. The pipeline contains example data that can be swapped to make it work with your infrastructure.
The folder holds the following files:
- buddy.yml – the definition of the pipeline
- scale.sh – the script which updates Terraform configuration
- main.tf and vars.tf – the Terraform configuration files
- install.tmpl.sh – the template of the script which installs Buddy on the new machine

2. Synchronize project

Create a new project in Buddy, select your Git provider, and add the forked repository. Buddy will automatically create the pipeline from the buddy.yml definition:

Image loading...

Warning

The files must be stored in the root directory or the pipeline configuration will not be detected.

Hint

If the pipeline still doesn't appear, check the Activity for parsing logs. You may also want to clear the values of Terraform variables from the Scale runners action.

Also, the calculator will be permanently added to the action list:

Image loading...

Part 2: Pipeline overview

The pipeline's configuration is stored in buddy.yml. In this part of the guide, we are going to trace the file from top to down and show how Buddy parses the input data to create the pipeline in the application.

1. Pipeline settings

The first part describes the pipeline behavior: trigger mode, assigned branch, and time interval between runs. We can see it runs on schedule every 5 minutes for the branch main:

yaml
- pipeline: Runner scaler
  on: SCHEDULE
  delay: 5
  start_date: "2023-01-01T00:00:00Z"
  refs:
  default: 
    - "refs/heads/master"

Warning

Make sure to update the date accordingly – the pipeline will not start for dates set to past.

In the application, pipeline runtime details are available in the Settings tab:

Image loading...

Tip

You can update pipeline configuration directly in the application. When done, click 'Generate YAML' and use it to update the code in the buddy.yml file.

2. Runner calculator

Going down, we approach actions. The first one is the runner calculator:

yaml
  actions:
    - action: Runner Calculator
      type: CUSTOM
      custom_type: Runners_Scale:latest
      inputs:
        RUNNER_TAG: ""
        RUNNER_SLOTS: "2"
        MAX_RUNNERS: "2"
        MIN_FREE_SLOTS: "1"

Hint

The action is described in detail in the README file.

Input

The inputs describe the configuration of the runners, and the number of desired free slots.

RUNNER_TAG – the tag for which the action calculates the runners. Leave empty if you want to scale untagged runners.
RUNNER_SLOTS – the number of concurrent slots per runner, i.e. how many pipelines or actions can be run at the same time. Leave empty to fetch the value from your license settings.
MAX_RUNNERS – the maximum number of runners that can be launched in your on-premises infrastructure.
MIN_FREE_SLOTS – the desired number of free slots for the given tag. Used to calculate whether to add or remove runners for the given tag.

Hint

Learn how tagging works in Buddy.

In this example:

the tag field is empty which means it refers to untagged runners only
each runner in the instance has 2 concurrency slots
the maximum number of runners running in the instance is 2
1 slot should always be available on every runner

Output

The input data is passed to calc.sh which generates three variables:

RUNNER_TAG – the runner tag for which the action was run
RUNNER_SLOTS – the number of concurrent slots per runner
RUNNERS – the optimized number of runners that can be passed to scaling scripts

3. Terraform

The second action is Terraform, which physically adds runners to the instance. The action runs the scale.sh script which updates the configuration of Terraform using the output from the runner calculator.

Image loading...

The Backend tab describes the AWS bucket storing the config:

Image loading...

yaml
    - action: Scale Runners
      type: TERRAFORM
      version: latest
      execute_commands:
        - chmod +x scale.sh
        - ./scale.sh
      integration_hash: "[AWS INTEGRATION HASH]"
      variables:
        - key: AWS_REGION
          value: "eu-central-1"
        - key: AWS_AZ
          value: "eu-central-1c"
        - key: INSTANCE_PRIVATE_KEY
          value: "[PRIVATE KEY TO CONNECT TO RUNNER]"
        - key: INSTANCE_PUBLIC_KEY
          value: "[PUBLIC KEY TO CONNECT TO RUNNER]"
        - key: INSTANCE_TYPE
          value: "t3.medium"
        - key: INSTANCE_AMI_ID
          value: "ami-06ce824c157700cd2"
        - key: INSTANCE_VOLUME_SIZE
          value: "50"
        - key: INSTANCE_VOLUME_THROUGHPUT
          value: "125"
        - key: INSTANCE_VOLUME_IOPS
          value: "3000"
        - key: STANDALONE_HOST
          value: "172.31.15.212"
        - key: STANDALONE_TOKEN
          value: "[ON-PREMISES RUNNER TOKEN]"
        - key: BACKEND_BUCKET
          value: "[BUCKET ON S3 TO SAVE TERRAFORM STATE]"
        - key: BACKEND_KEY
          value: "runners.tfstate"

Terraform variables

At the bottom we can see the variables that need to be filled in order to work:

Name	Description
`$AWS_REGION`	The AWS region in which the runners are hosted
`$AWS_AZ`	The availability zone in the AWS region
`$INSTANCE_PRIVATE_KEY`	The private SSH key used on the runner
`$INSTANCE_PUBLIC_KEY`	The public SSH key used on the runner
`$INSTANCE_TYPE`	The type of the instance on which the runner is hosted
`$INSTANCE_AMI_ID`	The ID of the AMI from which the runner is launched on the instance (e.g. Ubuntu in Ohio region)
`$INSTANCE_VOLUME_SIZE`	The size of the disk in the instance in GB
`$INSTANCE_VOLUME_THROUGHPUT`	The speed of the disk in the instance in MB/s
`$INSTANCE_VOLUME_IOPS`	The number of input operations per second set on the disk in the instance
`$STANDALONE_HOST`	The IP address of the primary runner (host) in the instance
`$STANDALONE_TOKEN`	The runner token generated on the primary runner (host) in the instance
`$BACKEND_BUCKET`	The S3 bucket on AWS in which Terraform keeps its current state
`$BACKEND_KEY`	The name of the file in the S3 bucket with the current Terraform state

Part 3: AWS configuration

This section describes installing Buddy on AWS EC2, creating a bucket for Terraform configuration, and generating SSH keys.

1. Installing Buddy on AWS

Hint

Follow this guide to install Buddy on AWS EC2.

Image loading...

Success

Things to note down

Public IPv4 address (required for $STANDALONE_HOST)
AMI ID (required for $INSTANCE_AMI_ID)
Region (required for $AWS_REGION)
Availability Zone (required for $AWS_AZ)
Instance type (required for $INSTANCE_TYPE)
Disk size (required for $INSTANCE_VOLUME_SIZE; can be left at default value)
Disk speed (required for $INSTANCE_VOLUME_THROUGHPUT; can be left at default value)
Disk IOPS (required for $INSTANCE_VOLUME_IOPS; can be left at default value)
Runner authorization token (required for $STANDALONE_TOKEN)

2. Creating S3 bucket for Terraform

The bucket is required to store Terraform configuration.

Image loading...

Requirements

Must be in the same region as Buddy on the EC2 instance.
Must have permissions so that the instance can use it.

Success

Things to note down

The name of the bucket (required for $BACKEND_BUCKET)
The name of the TF file (required for $BACKEND_KEY; can be left at default value)

3. Adding AWS integration

Now we need to integrate Buddy with AWS so that it can connect with AWS S3.

Image loading...

Success

Thing to note down

The ID of the integration (required for integration_hash)

4. Configuring SSH runner keys

The keys are required to authenticate in your AWS runners. To generate a new pair of keys for runners, run:

bash
ssh-keygen -b 2048 -t rsa$

Success

Things to note down

The contents of id_buddy_runner (required for $INSTANCE_PRIVATE_KEY)
The contents of id_buddy_runner.pub (required for $INSTANCE_PUBLIC_KEY)

Part 4: Filling Terraform variables

With everything prepared, you can now fill the variables in the buddy.yml file from the /example_pipeline directory. Follow the template and make sure the formatting is correct:

yaml
      variables:
        - key: AWS_REGION
          value: "eu-central-1"
        - key: AWS_AZ
          value: "eu-central-1c"
        - key: INSTANCE_PRIVATE_KEY
          value: "[PRIVATE KEY TO CONNECT TO RUNNER]"
        - key: INSTANCE_PUBLIC_KEY
          value: "[PUBLIC KEY TO CONNECT TO RUNNER]"
        - key: INSTANCE_TYPE
          value: "t3.medium"
        - key: INSTANCE_AMI_ID
          value: "ami-06ce824c157700cd2"
        - key: INSTANCE_VOLUME_SIZE
          value: "50"
        - key: INSTANCE_VOLUME_THROUGHPUT
          value: "125"
        - key: INSTANCE_VOLUME_IOPS
          value: "3000"
        - key: STANDALONE_HOST
          value: "172.31.15.212"
        - key: STANDALONE_TOKEN
          value: "[ON-PREMISES RUNNER TOKEN]"
        - key: BACKEND_BUCKET
          value: "[BUCKET ON S3 TO SAVE TERRAFORM STATE]"
        - key: BACKEND_KEY
          value: "runners.tfstate"

From now on, whenever you're short on horsepower, the pipeline will automatically create a new runner in your infrastructure:

Image loading...

Jarek Dylewski

Customer Support

A journalist and an SEO specialist trying to find himself in the unforgiving world of coders. Gamer, a non-fiction literature fan and obsessive carnivore. Jarek uses his talents to convert the programming lingo into a cohesive and approachable narration.

How to scale Buddy on AWS with Terraform

About runner variables

Part 1: Project configuration

1. Fork repository

2. Synchronize project

Part 2: Pipeline overview

1. Pipeline settings

2. Runner calculator

Input

Output

3. Terraform

Part 3: AWS configuration

1. Installing Buddy on AWS

2. Creating S3 bucket for Terraform

3. Adding AWS integration

4. Configuring SSH runner keys

Part 4: Filling Terraform variables

Start for free

Self hosted

Cloud