How to scale Buddy on AWS with Terraform

In this guide, we show how to create a pipeline that automatically scales Buddy on-premises by adding or removing extra workers in AWS infrastructure using Terraform.

About worker variables

The guide is based on worker variables, a new type of env vars that describe the behavior of Buddy workers. The variables are generated at the beginning of every pipeline run and are available for all actions in the pipeline. You can use them to calculate how many workers you need to serve your workload, send current load to CloudWatch, Datadog, or New Relic, or notify your team on a Telegram channel.

Part 1: Project configuration

1. Fork repository

The repository contains two things:

1. Worker calculator (action.yml) – a custom action which calculates the load in your on-premises network. The output of the action can be sent as a variable in a notification when the load exceeds the limit. You can also use it in a script that will add or remove workers when necessary (see below).

2. Scaling pipeline (/example_pipeline) – a preconfigured pipeline which uses Terraform to launch or shut down EC2 instances basing on the calculator's output. The pipeline contains example data that can be swapped to make it work with your infrastructure.

   The folder holds the following files:

   • buddy.yml – the definition of the pipeline
   • scale.sh – the script which updates Terraform configuration
   • main.tf and vars.tf – the Terraform configuration files
   • install.tmpl.sh – the template of the script which installs Buddy on the new machine

2. Synchronize project

Create a new project in Buddy, select your Git provider, and add the forked repository. Buddy will automatically create the pipeline from the buddy.yml definition:

Also, the calculator will be permanently added to the action list:

Part 2: Pipeline overview

The pipeline's configuration is stored in buddy.yml. In this part of the guide, we are going to trace the file from top to down and show how Buddy parses the input data to create the pipeline in the application.

1. Pipeline settings

The first part describes the pipeline behavior: trigger mode, assigned branch, and time interval between runs. We can see it runs on schedule every 5 minutes for the branch main:

- pipeline: Worker scaler
  on: SCHEDULE
  delay: 5
  start_date: "2023-01-01T00:00:00Z"
  refs:
  default: 
    - "refs/heads/master"

In the application, pipeline runtime details are available in the Settings tab:

2. Worker calculator

Going down, we approach actions. The first one is the worker calculator:

  actions:
    - action: Worker Calculator
      type: CUSTOM
      custom_type: Workers_Scale:latest
      inputs:
        WORKER_TAG: ""
        WORKER_SLOTS: "2"
        MAX_WORKERS: "2"
        MIN_FREE_SLOTS: "1"

Input

The inputs describe the configuration of the workers, and the number of desired free slots.

• WORKER_TAG – the tag for which the action calculates the workers. Leave empty if you want to scale untagged workers.
• WORKER_SLOTS – the number of concurrent slots per worker, i.e. how many pipelines or actions can be run at the same time. Leave empty to fetch the value from your license settings.
• MAX_WORKERS – the maximum number of workers that can be launched in your on-premises infrastructure.
• MIN_FREE_SLOTS – the desired number of free slots for the given tag. Used to calculate whether to add or remove workers for the given tag.

In this example:

• the tag field is empty which means it refers to untagged workers only
• each worker in the instance has 2 concurrency slots
• the maximum number of workers running in the instance is 2
• 1 slot should always be available on every worker

Output

The input data is passed to calc.sh which generates three variables:

• WORKER_TAG – the worker tag for which the action was run
• WORKER_SLOTS – the number of concurrent slots per worker
• WORKERS – the optimized number of workers that can be passed to scaling scripts

3. Terraform

The second action is Terraform, which physically adds workers to the instance. The action runs the scale.sh script which updates the configuration of Terraform using the output from the worker calculator.

The Backend tab describes the AWS bucket storing the config:

    - action: Scale Workers
      type: TERRAFORM
      version: latest
      execute_commands:
        - chmod +x scale.sh
        - ./scale.sh
      integration_hash: "[AWS INTEGRATION HASH]"
      variables:
        - key: AWS_REGION
          value: "eu-central-1"
        - key: AWS_AZ
          value: "eu-central-1c"
        - key: INSTANCE_PRIVATE_KEY
          value: "[PRIVATE KEY TO CONNECT TO WORKER]"
        - key: INSTANCE_PUBLIC_KEY
          value: "[PUBLIC KEY TO CONNECT TO WORKER]"
        - key: INSTANCE_TYPE
          value: "t3.medium"
        - key: INSTANCE_AMI_ID
          value: "ami-06ce824c157700cd2"
        - key: INSTANCE_VOLUME_SIZE
          value: "50"
        - key: INSTANCE_VOLUME_THROUGHPUT
          value: "125"
        - key: INSTANCE_VOLUME_IOPS
          value: "3000"
        - key: STANDALONE_HOST
          value: "172.31.15.212"
        - key: STANDALONE_TOKEN
          value: "[ON-PREMISES WORKER TOKEN]"
        - key: BACKEND_BUCKET
          value: "[BUCKET ON S3 TO SAVE TERRAFORM STATE]"
        - key: BACKEND_KEY
          value: "workers.tfstate"

Terraform variables

At the bottom we can see the variables that need to be filled in order to work:

Part 3: AWS configuration

This section describes installing Buddy on AWS EC2, creating a bucket for Terraform configuration, and generating SSH keys.

1. Installing Buddy on AWS

2. Creating S3 bucket for Terraform

The bucket is required to store Terraform configuration.

Requirements

1. Must be in the same region as Buddy on the EC2 instance.
2. Must have permissions so that the instance can use it.

3. Adding AWS integration

Now we need to integrate Buddy with AWS so that it can connect with AWS S3.

4. Configuring SSH worker keys

The keys are required to authenticate in your AWS workers. To generate a new pair of keys for workers, run:

ssh-keygen -b 2048 -t rsa
$

Part 4: Filling Terraform variables

With everything prepared, you can now fill the variables in the buddy.yml file from the /example_pipeline directory. Follow the template and make sure the formatting is correct:

      variables:
        - key: AWS_REGION
          value: "eu-central-1"
        - key: AWS_AZ
          value: "eu-central-1c"
        - key: INSTANCE_PRIVATE_KEY
          value: "[PRIVATE KEY TO CONNECT TO WORKER]"
        - key: INSTANCE_PUBLIC_KEY
          value: "[PUBLIC KEY TO CONNECT TO WORKER]"
        - key: INSTANCE_TYPE
          value: "t3.medium"
        - key: INSTANCE_AMI_ID
          value: "ami-06ce824c157700cd2"
        - key: INSTANCE_VOLUME_SIZE
          value: "50"
        - key: INSTANCE_VOLUME_THROUGHPUT
          value: "125"
        - key: INSTANCE_VOLUME_IOPS
          value: "3000"
        - key: STANDALONE_HOST
          value: "172.31.15.212"
        - key: STANDALONE_TOKEN
          value: "[ON-PREMISES WORKER TOKEN]"
        - key: BACKEND_BUCKET
          value: "[BUCKET ON S3 TO SAVE TERRAFORM STATE]"
        - key: BACKEND_KEY
          value: "workers.tfstate"

From now on, whenever you're short on horsepower, the pipeline will automatically create a new worker in your infrastructure: