How to scale Buddy on AWS with Terraform
April 21, 2023
In this guide, we show how to create a pipeline that automatically scales Buddy on-premises by adding or removing extra workers in AWS infrastructure using Terraform.
About worker variables
The guide is based on worker variables, a new type of env vars that describe the behavior of Buddy workers. The variables are generated at the beginning of every pipeline run and are available for all actions in the pipeline. You can use them to calculate how many workers you need to serve your workload, send current load to CloudWatch, Datadog, or New Relic, or notify your team on a Telegram channel.
The JSON with the information about all installed workers
The list of IP addresses of workers tagged with
The list of IP addresses of untagged workers
The average load from the last minute on all workers tagged with
The average load from the last minute on all untagged workers
The total number of pipeline slots across all workers
The number of workers tagged with
The number of untagged workers
The number of free pipeline slots on workers tagged with
The number of free pipeline slots on all untagged workers
Part 1: Project configuration
1. Fork repository
The repository contains two things:
Worker calculator (
action.yml) – a custom action which calculates the load in your on-premises network. The output of the action can be sent as a variable in a notification when the load exceeds the limit. You can also use it in a script that will add or remove workers when necessary (see below).
Scaling pipeline (
/example_pipeline) – a preconfigured pipeline which uses Terraform to launch or shut down EC2 instances basing on the calculator's output. The pipeline contains example data that can be swapped to make it work with your infrastructure.
The folder holds the following files:
buddy.yml– the definition of the pipeline
scale.sh– the script which updates Terraform configuration
vars.tf– the Terraform configuration files
install.tmpl.sh– the template of the script which installs Buddy on the new machine
2. Synchronize project
Create a new project in Buddy, select your Git provider, and add the forked repository. Buddy will automatically create the pipeline from the
Also, the calculator will be permanently added to the action list:
Part 2: Pipeline overview
The pipeline's configuration is stored in
buddy.yml. In this part of the guide, we are going to trace the file from top to down and show how Buddy parses the input data to create the pipeline in the application.
1. Pipeline settings
The first part describes the pipeline behavior: trigger mode, assigned branch, and time interval between runs. We can see it runs on schedule every 5 minutes for the branch
- pipeline: Worker scaler on: SCHEDULE delay: 5 start_date: "2023-01-01T00:00:00Z" refs: default: - "refs/heads/master"
In the application, pipeline runtime details are available in the Settings tab:
2. Worker calculator
Going down, we approach
actions. The first one is the worker calculator:
actions: - action: Worker Calculator type: CUSTOM custom_type: Workers_Scale:latest inputs: WORKER_TAG: "" WORKER_SLOTS: "2" MAX_WORKERS: "2" MIN_FREE_SLOTS: "1"
inputs describe the configuration of the workers, and the number of desired free slots.
WORKER_TAG– the tag for which the action calculates the workers. Leave empty if you want to scale untagged workers.
WORKER_SLOTS– the number of concurrent slots per worker, i.e. how many pipelines or actions can be run at the same time. Leave empty to fetch the value from your license settings.
MAX_WORKERS– the maximum number of workers that can be launched in your on-premises infrastructure.
MIN_FREE_SLOTS– the desired number of free slots for the given tag. Used to calculate whether to add or remove workers for the given tag.
In this example:
- the tag field is empty which means it refers to untagged workers only
- each worker in the instance has 2 concurrency slots
- the maximum number of workers running in the instance is 2
- 1 slot should always be available on every worker
The input data is passed to
calc.sh which generates three variables:
WORKER_TAG– the worker tag for which the action was run
WORKER_SLOTS– the number of concurrent slots per worker
WORKERS– the optimized number of workers that can be passed to scaling scripts
The second action is Terraform, which physically adds workers to the instance. The action runs the
scale.sh script which updates the configuration of Terraform using the output from the worker calculator.
The Backend tab describes the AWS bucket storing the config:
- action: Scale Workers type: TERRAFORM version: latest execute_commands: - chmod +x scale.sh - ./scale.sh integration_hash: "[AWS INTEGRATION HASH]" variables: - key: AWS_REGION value: "eu-central-1" - key: AWS_AZ value: "eu-central-1c" - key: INSTANCE_PRIVATE_KEY value: "[PRIVATE KEY TO CONNECT TO WORKER]" - key: INSTANCE_PUBLIC_KEY value: "[PUBLIC KEY TO CONNECT TO WORKER]" - key: INSTANCE_TYPE value: "t3.medium" - key: INSTANCE_AMI_ID value: "ami-06ce824c157700cd2" - key: INSTANCE_VOLUME_SIZE value: "50" - key: INSTANCE_VOLUME_THROUGHPUT value: "125" - key: INSTANCE_VOLUME_IOPS value: "3000" - key: STANDALONE_HOST value: "172.31.15.212" - key: STANDALONE_TOKEN value: "[ON-PREMISES WORKER TOKEN]" - key: BACKEND_BUCKET value: "[BUCKET ON S3 TO SAVE TERRAFORM STATE]" - key: BACKEND_KEY value: "workers.tfstate"
At the bottom we can see the variables that need to be filled in order to work:
|The AWS region in which the workers are hosted|
|The availability zone in the AWS region|
|The private SSH key used on the worker|
|The public SSH key used on the worker|
|The type of the instance on which the worker is hosted|
|The ID of the AMI from which the worker is launched on the instance (e.g. Ubuntu in Ohio region)|
|The size of the disk in the instance in GB|
|The speed of the disk in the instance in MB/s|
|The number of input operations per second set on the disk in the instance|
|The IP address of the primary worker (host) in the instance|
|The worker token generated on the primary worker (host) in the instance|
|The S3 bucket on AWS in which Terraform keeps its current state|
|The name of the file in the S3 bucket with the current Terraform state|
Part 3: AWS configuration
This section describes installing Buddy on AWS EC2, creating a bucket for Terraform configuration, and generating SSH keys.
1. Installing Buddy on AWS
Things to note down
- Public IPv4 address (required for
- AMI ID (required for
- Region (required for
- Availability Zone (required for
- Instance type (required for
- Disk size (required for
$INSTANCE_VOLUME_SIZE; can be left at default value)
- Disk speed (required for
$INSTANCE_VOLUME_THROUGHPUT; can be left at default value)
- Disk IOPS (required for
$INSTANCE_VOLUME_IOPS; can be left at default value)
- Worker authorization token (required for
2. Creating S3 bucket for Terraform
The bucket is required to store Terraform configuration.
- Must be in the same region as Buddy on the EC2 instance.
- Must have permissions so that the instance can use it.
Things to note down
- The name of the bucket (required for
- The name of the TF file (required for
$BACKEND_KEY; can be left at default value)
3. Adding AWS integration
Now we need to integrate Buddy with AWS so that it can connect with AWS S3.
Thing to note down
- The ID of the integration (required for
4. Configuring SSH worker keys
The keys are required to authenticate in your AWS workers. To generate a new pair of keys for workers, run:
ssh-keygen -b 2048 -t rsa$
Things to note down
- The contents of
- The contents of
Part 4: Filling Terraform variables
With everything prepared, you can now fill the variables in the
buddy.yml file from the
/example_pipeline directory. Follow the template and make sure the formatting is correct:
variables: - key: AWS_REGION value: "eu-central-1" - key: AWS_AZ value: "eu-central-1c" - key: INSTANCE_PRIVATE_KEY value: "[PRIVATE KEY TO CONNECT TO WORKER]" - key: INSTANCE_PUBLIC_KEY value: "[PUBLIC KEY TO CONNECT TO WORKER]" - key: INSTANCE_TYPE value: "t3.medium" - key: INSTANCE_AMI_ID value: "ami-06ce824c157700cd2" - key: INSTANCE_VOLUME_SIZE value: "50" - key: INSTANCE_VOLUME_THROUGHPUT value: "125" - key: INSTANCE_VOLUME_IOPS value: "3000" - key: STANDALONE_HOST value: "172.31.15.212" - key: STANDALONE_TOKEN value: "[ON-PREMISES WORKER TOKEN]" - key: BACKEND_BUCKET value: "[BUCKET ON S3 TO SAVE TERRAFORM STATE]" - key: BACKEND_KEY value: "workers.tfstate"
From now on, whenever you're short on horsepower, the pipeline will automatically create a new worker in your infrastructure:
Customer Success Manager
With Buddy even the most complicated CI/CD workflows take minutes to create
Sign up for Buddy CI/CDStart a free trial