Announcing: New Build Servers & Infrastructure Upgrade

Before we started working on Buddy in 2015, we made an analysis to indicate the user needs and the best niche for our application.

We worked out 3 selling points that were to become the foundations of Buddy:

• speed and stability (essential in Continuous Integration and Delivery)
• ease of configuration/integration with other tools
• outstanding UI/UX

Since development of Buddy is 100% customer driven, we did change our approach to some things in the past two years. However, we have never stepped an inch away from any of these policies. Such approach allowed us to grow and win recognition of web and software developers.

At the moment over 8000 companies use Buddy in their everyday work. On average, we run over 3.000.000 builds per month, with 7% of monthly grow. Every second 37 pipelines are being executed.

Why we decided to upgrade the infrastructure

Until now we used only the infrastructure provided by SingleHop: dedicated high-performing servers with 32 cores, 128GB RAM and SSD RAID 10 hard drives.

However, with growing number of users we started to observe some problems:

• The main was the number of disk operations run on servers (disk I/O utilization). Although RAM and average CPU usage rarely exceeded 25%, disk utilization kept showing 80% of usage which in effect produced I/O waits.
• The other problem was the flexibility: the ability to add another node to our build cluster and to reconfigure the nodes if needed (eg. resize disk)

To maintain the speed and stability, in June we decided to transfer the build servers to AWS infrastructure. The operation required multiple changes in the application and at the same time had to be transparent to our users. In the second half of August, the servers were launched for production, with the whole migration process coming to an end this week.

During that time some of our users may experience longer execution times. The situation is temporary and results from the following things:

• when the pipeline is executed, the pipeline data is being migrated to the new infrastructure (only on the first run)
• deployment from the filesystem is now taking longer because it's being proxied through the old server (so that our users have time to whitelist the new IP's). This will last for the next 3 weeks until we fully switch to the new servers.

How it looks now

At the moment the core application and databases still use the SingleHop infrastructure – it's been working very well for this part of the system and there's no need to change it. Build servers, on the other hand, have been fully moved to new AWS nodes: c4.4xlarge (16 cores & 30GB or RAM) with EBS (SSD) hard drives in RAID 10. The servers may not look as powerful as those on SingleHop, but launching them in a higher number allowed us to fine-tune the build balance and significantly decrease execution time.

For some of our clients, the speed of builds have doubled! Here's what we received from one of our users (cheers Mark):

Image loading...

Hope you're excited about the upgrade like the whole Buddy team! Let us know if you felt any change in performance!

Current list of IP's

Like we mentioned above, we're going to proxy the deployment through the old servers for the next 3 weeks. This means you need to add the new IP's to your firewall's whitelist.

To whitelist all our IP's, just run

default
iptables -A INPUT -p tcp -s workers.buddy.works -j ACCEPT

The command will resolve the hostname and add all current IP's to the firewall.

If your firewall doesn't support adding DNS source, here's the full list of IP's that you can add manually:

default
91.200.38.2
108.163.203.146
184.154.12.42
184.154.193.210
---NEW IP's---
13.59.138.22
18.220.79.31
18.221.94.43
18.220.233.117
52.14.34.6
52.15.76.37