How to master Drupal deployment cycle with a proper DevOps process

Hello, everyone, welcome to today's webinar. Today we are going to talk a bit about how to master Drupal deployment cycle with a proper DevOps process. As you already know, I'm more of a WordPress guy. That's why I invited a whole team of Drupal specialists to help me, guys. Tell us a bit about yourself and about about Cyber-Duck.

Cool. Hi, everyone. So yeah, we're from Cyber-Duck. We're an award winning digital transformation agency based in the UK. And we've been growing organically for the last 17 years now, working mostly with Drupal and Laravel to deliver software as a service application and CMS solutions for clients, mostly, mostly in the UK, both in the private and public sector. And, yeah, it's been it's been amazing. So I'm Savannah writer, the co founder and chief delivery officer, I come from a technical background and gone through through the stages of deploying website manually via FTP to kind of building the whole dev team and growing the agency with a lot of experts, which brought me along today, then can gems or to you.

Hi, I'm, I'm Duncan Worrell I've been with Cyber-Duck for about almost three years. I'm a Drupal technical lead. So I'm the person who has to rely on my colleague here, James to set up pipelines for me and Buddy, so I can deploy things easily. So I'm very reliant on DevOps. So I'm here on our supporting capacity to try and answer any specific Drupal questions anyone might have over two days.

Thanks, Duncan. You make it sound so easy, Duncan, but I see it the other way around. You see, I see that I'm here to support you to get your code into production. So that's your team synergy. Yeah. So hello, everyone. My name is James. I'm the technical lead for DevOps at Cyber-Duck. spend my day to day work, engaged in engineering effort and helping DevOps seem to support our software engineering functions to deliver code into production faster.

And my name is Maciek Palmowski. I work here at Buddy as a WordPress ambassador and the regular webinar host. So before we'll get started, if you like our webinars, do not forget to press the subscribe button on YouTube face to this, you will be always informed about some new upcoming webinars, and we are really doing quite a few of them lately. And also, if you have any questions regarding Drupal, DevOps, or whatever our discussion will, will lead us today. Don't hesitate to use either YouTube, LinkedIn or Facebook commands to ask them we'll try to respond to as many as possible in the end during the q&a session. Okay, so let me start with the first question. Why Drupal? I mean, looking at the market shirt. Drupal has about 1.3% and looking at the numbers WordPress Shopify week square, Squarespace and Joomla. So why did you pick Drupal?

Yeah, I can start on the I guess the more like the business side thinking about it from from the end users or the other clients and then my colleague can take on the tech and environments but yeah, look, we decided the time was several years ago to start with Drupal. We tried actually a bit on WordPress as well and other of the shell CMS is but as the agency groove realised that a lot of the projects were becoming more complex, and sometimes the clients requirements are a bit vague, where, if it's just a corporate website, it's might start small, saying, Yeah, we just want a few pages and explain about us having an online presence. But that was fine. 1015 years ago, nowadays, websites are much more interactive. There's much more connectivity with third party systems. A lot of the marketing team rely on marketing automation tools. And a lot of the more technical team wants to connect with CRMs and ERPs. And there's, there's been obviously a lot of advances in the digital space. And those complex requirements needs an enterprise ready platform. And we were just pushing the limits of WordPress felt like that as we got bigger clients. Some issues around the security and regular updates being WordPress being the most popular platform in the world is also probably the most attacked and invulnerable. So some of our corporate clients or even government bodies quickly moved away from that. And Drupal is the platform of choice in the UK, specifically the UK Government pushing a lot on the digital everything on UX and agile and on the technology side, promoting open source very heavily and and Drupal is one of the most used platform. So the answers a lot of the business requirements, it complies with a lot of the top level and corporate requirements. So we felt that was the best platform and it's proven us right in the last few years, we've grown exponentially and having a lot of Drupal projects coming in. So that's from the business side.

From from a technical point of view, I I was running a business at the time, and I had to decide on on a CMS and I looked at several services, including WordPress, and Joomla, and some others. And I settled personally on choosing Drupal. It seemed to fit my use case, I needed to create lots of different types of content. And Drupal seemed to do that out of the box. I think with with Drupal and WordPress is a conversation we could talk about probably for hours. WordPress is easier for sure. It's certainly easier for someone off the street to pick up and use Drupal is quite hard. But that that, that means it's also more powerful, I believe. So you end up in a situation where pretty much the Sylvan says you could start a website with WordPress and and there's been plenty of surveys done this that as you become more proficient in WordPress, become more and more frustrated with it. And the opposite is true with Drupal. The opposite is true. Because when you first start using it, you're you bang your head against the wall when things are difficult. But the more you use it, the more you love it. And Drupal recognise that that Drupal 10 comes out in December this year. And one of their big, big pushes is to make it easier out of the box. And that's very, very squarely aimed at the WordPress space at the moment. So for me, group, Drupal is a more powerful platform, arguably. But on with that you you have a very good community that sits behind you is obviously very obviously open source, which a lot of UK government contracts or mandate, they you have to use an open source solution. And it's also very scalable, very performant. And very secure. And I know these are all buzzwords, but again, demonstrating demonstrates the coming say that were true. You know, I every tonight there will be a security update for Drupal if they found anything. And they have a whole team of ethical hackers constantly trying to break Drupal. And that's not just Drupal core, it's it's Drupal contributed modules. So I like the fact that it's so dynamic. I like the fact that it's complicated, because that complexity gives me the potential to create some really, really good websites for our class. So that's that's why I like it.

Fantastic it's a really good point that Duncan makes about the complexity, I think that we you know, certainly in terms of engineering, many of us have experienced this paradigm whereby, you know, there are tools out there that make the simple things simple, but then when you try when you try to apply them to the complex challenges, they actually make it more complex. It's kind of you know, that that kind of classic paradigm really, that there are tools you know, that make simple jobs simple but make more complex jobs as soon as you want to move outside the kind of confines of the functionality you're into kind of uncharted territory. Whereas, you know, certainly from the work, I experienced the teams that Cyber-Duck doing, it seems like, you know, the Drupal space, we have a good a good framework to, to build functionality according to, you know, the Drupal community's ideas, but we also have plenty of scope to build our own, you know, tools and processes outside of that, where necessary, and that includes things like partnerships, you know, partnering with companies like Acquia, to provide cloud services, and the kind of benefits that they provide, we can talk about that a little bit later.

Okay, so, like, our topic says, we will be talking a bit about DevOps processes. So why do we even even need DevOps? I mean, apart from using this really nice buzzword because DevOps is so cool, right?

Yeah, so I think I can talk to this one. Still, when can tell us why he thinks it's important to so we can hear both sides. So yeah, so So my background is mostly an engineering really software engineering, and infrastructure. And so I kind of, you know, have more of a kind of traditional engineering point of view on this, but, but also, you know, understand the history. So, you know, say, five or 10 years ago, the kind of the traditional paradigms in, in this industry, in software engineering are that there are two independent and distinct teams, which are working towards common business objectives, but quite often have different priorities. And those two teams, as the name implies, would have been development and operations. Right. So the job of development is to build functionality build software, move quickly produce features, add value to the business, that's one side of the coin is the development. The other side of the coin is the operations. The people in operations, their responsibilities are, keep systems running, maintain stability, you know, change control processes that are very rigid or beneficial to them. And so these two competing forces have often had a kind of siloed approach to their common goals, developers produce software quickly, and want to get it into production in front of the business users quickly, to deliver value quickly. operations teams want to maintain stability, so they can be more resistant to change, and less willing to push regular changes, because of the processes that are involved in doing that. Quite often manual rigorous reviews. So this is where DevOps comes in. It's the amalgamation that coming together of two different kinds of effort really, the coming together of development and operations developers and operations people really in a in a kind of, in a culture of DevOps. So the key function of any DevOps team, any DevOps engineer is to empower their colleagues to deliver functionality to the business as quickly as possible. In Cyber-Duck, for example, we have a DevOps engineering team who work with every team in the business, front end developers, back end developers, you know, design, user experience, QA, these teams all working together to deliver change quickly. And the main way we do that is through culture, is by embracing the culture of DevOps, which is a kind of spin off of agile and agile principle, to deliver value quickly and to the business. And you know, the tools and processes that support that are pretty kind of ubiquitous across across different disciplines. So, you know, if that means, for example, automation, we can apply that automation to our back end development, to our front end development, or testing to our server deployments. All these areas can benefit from the DevOps culture, without needing to be a DevOps team, so to speak themselves, we imbibe the culture and the business and then everyone has the value of it.

I really liked that you are talking about DevOps as a culture inside of the company, rather than what some tried to say that those are tools. Those are specific roles. Now it's all about about the culture about the way how we work. So yeah, I fully agree with you.

I think you you quite often find that it's a really interesting point, you quite often find that in the teams who see DevOps as a as a as only a specific job role. They tend to actually be not really doing DevOps, they're just doing ops, right. Yeah, exactly. Exactly. It's like we took operations, we took these rigid structures. We applied some cool tooling to it and we're calling it DevOps. but it doesn't work in reality unless you have the culture and, you know, to be perfectly transparent, you know, one of the main challenges with the culture is, is delivering that culture to every part of the business. It's you need complete buy in on every level, you need buy in, from stakeholders, from directors, from people in positions of management, and in engineering, in customer relations in new business, you need buy in completely throughout. Because if you don't have it, and what you end up with, is you end up trying to shoehorn the DevOps process at the last part of the process. And then it just doesn't, it just doesn't work. There's frustrations throughout. So you know, my experience is that the best DevOps businesses embrace that culture and invite it into that into the core of their being into their strategies, business processes, their identities, and they live it and breathe it. They don't just say you go do DevOps, right? Yeah, doesn't really work like that. You could call that person an automation engineer, perhaps, but it's not DevOps.

Yeah. What about the business, the business side of DevOps?

Yeah, for me, main reason is DevOps is the ability to scale and scale kind of safely or confidently. Because yeah, we all started, I guess, as kind of web developers in our bedroom, kind of deploying our first websites. And when you are alone, it's fine. You have you work on your own code, you then can explain can push to FTP, and like, you've managed all of that you have to control everything and you do the front end, the back end, you create your hosting account, and you have to do your DNS things. And then that's fine when you're on your own. But as soon as you have a bigger project, or you start building your team, or can have multiple developers start working together, it's not it's not scalable. So DevOps allows you to minimise risk and increase efficiency, minimising the risk by automating some of the steps. So yeah, deploying code by FTP overwriting each other up that it's prone to many errors. And I'm sure we've all done them where you forgot to deploy a file or you don't lie, the right changes to the right config item or something missing somewhere in your pipeline. And so reducing human errors through automation is one of the main business benefits. And the other one is maximising efficiency. Because with the rise of cloud technologies, we don't have to wait two weeks to deploy dedicated servers in Iraq, or you don't have to wait for the next release. Like some some clients still have like, a yearly release schedule where if you want something deployed in your app, you need to wait another six months for the next deployment cycle or something. So again, the efficiency of applying those that automation and the automated testing all of the benefits of DevOps, bring confidence to the team, and just the developers, but also the management, that things can move much quicker. And you can deploy testing environment with almost every day and the deployment to production is one click and so many benefits that the efficiency just go through the roof. And that's why we're using it on every single project.

So if we could sum up, what are the benefits for each group for developers, for clients, for managers? How would you say,

Let me talk about developers. I haven't spoken for ages. I think for developers, we've covered much of it, it's repeatable processes. As a Drupal developer, we want to develop a Drupal solution. Normally, we'd be doing that on our local systems, our Mac's or whatever we're using. So but to get that stuff deployed, we don't want the hassle. We don't want that. Okay, how do I do deployment, but what we want to do is is, is have a pipeline set up where we can click a button, and it does it, it's a repeatable pattern that we know is the same every single time. And we know we can go into testing so we can do our QA cycle. We also want that same process to be in place when we deploy to our staging environment. So we can perform user acceptance testing and clicking the same button with the same pipeline gives us a great amount of confidence that we're going to deploy exactly the same thing in exactly the same way. So we can test it. And also, of course, have the final deployment into production when we go live. So it gives us a repeatable pattern. Something that theme Silva mentioned about being able to spin up new instances quickly, that that hasn't been around for very long. You know, I guess we're talking about cloud instances. So I need a new service seminar. I don't want to do that really. I can lead up to DevOps, and then they can repeat the or effectively clone a pipeline for us. And we know this is going to work exactly the same as every other pipeline we deploy. So so forth for devs it allows us to concentrate external development. But it gives us a huge amount of confidence that when we do deploy stuff, that it's going to every single instance in exactly the same way. So that's the biggest benefit, I think from a development point of view.

And businesses had before the increase the frequency of release, I guess, would be the main, the main benefits, like were mostly still need to wait for the developers to do their part. But in terms of releasing features to product or to an application, any system really, it's, it's the guarantee that there is an automated deployment process, there is a roll back step I've been involved in. And again, all of that takes time to build over time. It's not, it's not, it's not one tool that will do all of that. And so that James was mentioning this Agile methodology and this notion of continual improvement, so you can always improve your process with from the dev and the ops side as well. So that is just part of this agile culture, agile methodology to continually improve the process and make it more efficient, more secure, more scalable, and bring more confidence to the business.

And what about clients? What are the biggest benefits from

For clients, that's a good good one as well. I would say lower costs overall, even though it's a bit of a higher investment is kind of traditional roles. Additional tools that you might need to subscribe to, to quit see at the beginning is a bit of a Yeah, the initial phases requires a bit of upfront investment. But from again, I've been we've been doing it for what four or five years now, benefits, you just kind of get are amazing. And you make whatever it invested the original investment was you make that up in the first couple of months, probably in your first few sprints, it already pays off, because yeah, it saves time, the developer can focus on building actual features and not having to worry or repeat deployments that are failing. So yeah, again, higher efficiency, lower cost and more confidence,

Lower risk, isn't it, we're talking about lower risk risks to the, if you're reducing your costs, increasing your efficient efficiencies, you're reducing your risk of the business running out of money for your projects, also, the risk of having a deployment go wrong. That's the worst thing, you make a deployment goes wrong, it just doesn't happen anymore, we've got a large amount of confidence, particularly as we're deploying with the same pipeline across different instances, we've tested it on two or three different places. So it reduces the risk for live deployments. And that's reducing risk is always good, it's one of those things, you don't notice until you lose it. You know, having a pipeline that works every single time, you can be a little bit ofay about it. DevOps is easy, we just hit the button and it deploys those those pipelines have been developed over, you know, years of experience. So good DevOps team knows how to set up a pipeline well knows how to create the server correct in the first place, knows how to put all those all those things that start with CF in front of Things Cloud free, cloud front. So on Cloud Foundry, happens on all begin with all those things in place, so you don't have to worry about them. And that decreases the risk to the business which ultimately saves money. And I guess we're talking about isn't it making things more efficient.

And we've ever had, we interview the client actually last year about the DevOps process that we applied and why they selected us to run it for them a global kind of corporate website. And he's saying as it was more from from the IT side, but client side, that trust is earned with confidence. Unfortunately, in being in it, he was struggling to connect with his marketing team was kind of owning the product, because was delays in releases. There was a discrepancy between environments, so they were testing something in UHD. And that was fine. But on production, it stopped working, or it's not working, behaving the same way or some of the jobs or the config were different. And then every time it couldn't really answer you have good answers for the marketing team saying like, yeah, it was just it did fail sometimes and complete lack of confidence. And, and yeah, everything was was taking longer, it was difficult to add new features, and it's just very overall very inefficient. With DevOps that applied throughout the whole lifecycle and rebuilding kind of migrating to AWS and building a proper pipeline. And applying the agile methodology now they're they're in a much better position and him being in it is almost not involved anymore. The marketing team runs it and they deploy when they give us the go ahead for the deployment, it's happening without it involvement. So very happy clients.

Yeah, I can, I can understand that. Okay, so I think now it's time for the part that most of us are waiting for. So how do you deploy Drupal using body? Let me share your screen already, James, because I know, now the scene is yours, go for it.

Thanks. So, as Duncan said, quite rightly, you know, when we're running deployments to web sites for clients in Cyber-Duck, it's usually actually the development team that running those deployments. It's not it's not the DevOps engineering team. You know, the, the whole purpose of DevOps, as I said earlier, is to empower business users to deliver, you know, quicker. So, you know, we like to build automations as much as possible and use those automations to allow our developers to release their own code. I think everyone's made really good points about kind of, you know, standardisation, automation, or the benefits they provide. I think it can't be understated, how beneficial is the single click of a button. through automation, we can enable a consistent release process by anyone, you know, the kind of the days of centralising that knowledge and that control in a silo. And all the risks that come along with that. They're they're kind of long run with DevOps, certainly in Cyber-Duck. So, yeah, what I'm showing you today is, is a website, it's a website for the Commonwealth Secretariat. Obviously, very well known public institution Cyber-Duck, built this website, for the Commonwealth Secretariat as a project, it's obviously based on Drupal. Hence being used as a demonstration for today. The tooling that we use for this is for the DevOps aspects, this is hosted within AWS. So we're using many of the AWS well known services like easy to to provide us with Compute and load balancers. We're using rds, for example, to provide MySQL database service. And we're using many of the other supporting services to like the usual ones, s3 for your file storage, and I am for your identity and access management. So how does Buddy tie into all this? This is the question, well, if you take a look at our pipelines in Buddy, this is the user interface through which developers can deploy their code. Within Cyber-Duck, we've got a few different environments for this particular client. And if we take one of them, for example, we can see that we've got a history of the different runs that were executed against this environment. This is the testing environment for this client. Building this within Buddy, for us is a simple case of using the interface that we see here. And quite often copying the existing examples we have for other clients. If we look at the steps that are actually required to build and launch a site, a Drupal site onto AWS, it's really straightforward. And there's lots of scope to add additional steps and functionalities as required. So in this case, this is the most basic example, which is each of the steps being executed sequentially in order. So first of all, we're performing our composure steps. So that's, you know, instantiate composure, you know, instal our dependencies, run config checks, etc. All of those things are executed within the step here. I can click through to it, and I can get more information about that step. You can see, for example, that this manual steps that we're taking here, and something that's interesting to note is that all of these steps are being executed within containers. So we know that we can change the container that this is running within and tailor the specific step to the functionality. In this case, we're actually using our generic PHP container that we built within Cyber-Duck with many of our common dependencies to run our composer instals. So first step is composer. The next step is we execute our front end build, we're installing node packages, running builds, you know, building our assets, which are ultimately going to be deployed to to AWS. From there, we're configuring AWS. So this means within within Drupal, we've obviously got different config files. And the conflict files vary depending on the environment. So if we're running locally, we have local development credentials. If we're running in production, we have production credentials. And in this case, we're swapping out the config files within the artefact that we're generating to ensure that the application will run in production that things like database connection strings are in the right place URLs are set correctly, and so on and so forth. As this is a testing site, we have some HTTP Basic auth in front of it. This is just one of the steps that we take to protect the non production environments from web crawlers who would otherwise index it and you'd have your testing site appear on Google, not a particularly sophisticated security mechanism or anything. It's a public site after all, just a means to keep it out of search engines in a simple, consistent way that we can use across projects because it's in our pipeline. The main two steps which perform the majority of the heavy lifting are actually templates, which we use directly from Buddy. We're using technology within AWS called Elastic Beanstalk. Elastic Beanstalk allows us to provision compute environments and have great benefits like auto scaling, automatic security updates, automatic monitoring, automatic rollbacks, of problematic releases, all these kinds of quite, you know, enterprise grade, sophisticated compute features of cloud environments, can come out of the box in Buddy when we deploy through their pipeline. Once we've deployed the code, we monitor it, a key part of the process is, in the cases of our clients, we have fault tolerant, highly available cloud infrastructures, what that means, really, is that everything has multiple, redundant copies. So if you take the kind of traditional service paradigm, which is what we'd call compute, that's got redundancies, we're running multiple different Compute Engines across multiple Amazon availability zones. So there's redundancy both geographically and also within the technology. And it also allows us to scale to meet demand as required, we can add additional servers additional compute capacity to meet increased traffic demand, particularly important for a public site. So this is our monitoring step monitors the release of our code as it's gone into AWS. And then in this case, the Commonwealth is using the Cloudflare service, which is a very well known kind of all in one networking service company who are providing, amongst other things a web application firewall for security, content delivery network for performance, and a number of other kinds of routing and auditing based services for the site. We want to clear the cache from that, obviously, after we've run a deployment, things usually change, we want users to get the latest version of the site. So again, we use the built in Buddy integration to achieve that. And then within Cyber-Duck, we try to encourage communication, especially with our automated tools. So whenever a deployments run, we're popping a notification into our Slack channel, just to let the developers the engineering team know that a deployments been executed. And we give the output of that whether it's been successful, or there's a challenge that they need to review. This is all information that's fed back every time the deployments run. It's worth saying that these pipelines that we're executing here, whether they're production, staging, testing, or even a CI pipeline, a continuous integration pipeline, which would be run every time an event is triggered. These can all happen either manually or in an automated fashion. So if, for example, you had a testing environment, and you wanted to deliver deliver value very quickly, you might configure this pipeline through the settings, so that it will run on any action that your developers take. So if a developer adds a feature, that's bleeding edge, and they want to get it to testing as quickly as possible, we can use buddy's built in functionality to make this pipeline run as soon as a change is committed to the to the version control technology to get. So the developer makes that change. They committed to get they save it, they push it and then straight away the pipeline is moving that into the testing environment to get it in front of internal stakeholders or testers for example. What does this actually look like when you're running it? Well, I mean, we can we can run it, it's really simple. You click run, you scroll to the bottom, you run. It's running. Right. That's, that's the deployment. So the process, James, sorry.

Let me just ask you one question, because this is, this is one question that came in our comments. And it's really about the thing that you said, Does the monitoring, deploy monitoring tools or just monitor if the deployment went? Okay.

Good question. So, the monitoring is Yeah, So to clarify, the monitoring is not monitored, deploying monitoring tools, it's monitoring the the rolling release that Elastic Beanstalk is carrying out. So that's monitoring to make sure that the current release happens successfully. The monitoring tools is a separate thing again, which just just to clarify the distinction, we're using Buddy to provide the pipeline, which allows us to move code from our development environments into either production or non production environments. The monitoring is monitoring to make sure that that release happens correctly within AWS within the cloud environment, that it doesn't roll back. If it actually has to roll back, you get what we see here. The see the first one has got a red cross against it. We executed a production deployment, there was a challenge, it was rolled back. And as part of that rollback, we saw that the monitoring flag that there was an issue here and an engineers working on that. So yes, that is monitoring the deployment, that's not monitoring the infrastructure. As far as infrastructure monitoring goes, many tools, we can probably talk about that more in a whole nother talk. Yeah, we use a fairly diverse set of tools for monitoring depending on the needs, most of which are, you know, fairly well known in the industry, many of which are specifically within AWS, if we're using AWS. And, you know, yeah, how are they configured? Probably another topic entirely. But we're, we're using infrastructure as code so TerraForm to build infrastructure for clients, which also then feeds into the reproduce, reproducible and cost transferable value that we can deliver.

Okay, yeah. So, at the moment we are waiting for for the deployment. Yeah. Because I always remember that that beanstalk can take a while.

Yeah, so so the way that this is possibly a digression, but I find it quite interesting, the way that Elastic Beanstalk works is you it's a platform as a service, right? We might have heard this term before pas, there are many different platform as a service offerings out there in the in the tech industry. You could you could use Azure, you could use web apps, you could use Elastic Beanstalk on AWS, you could use platforms like SSH, you could, you know, the Cloud Foundry is another common one that we use. There are lots of different choices out there for a platform as a service. But the fundamental principle is that you provide it with a artefact, usually a, you know, a website, and it takes care of deploying that to the infrastructure for you. So it reduces, again, reduces the management overhead required, and increases your time to deliver because you're focusing on building pipelines to deliver content, not on maintaining complex infrastructures, you abstract that infrastructure away and use the automation tools to enable that. So that deployments completed now, deployment to our testing environment is done. Cash has been cleared, cleared, notification in Slack. So that's it, we're done. We've run our deployment. It's as simple as that.

And what happens if something if something fails?

So it depends. It's a complex question with a complex answer. It depends on what step it fails. So if, say the composer instal step fails, the pipeline is configured to just exit at that point, because nothing's actually been changed outside of Buddy. In that case, we can just run it again or correct the problem and debug it. If it's the actual AWS deployment step that fails, then AWS will roll back the version that has been deployed and return us to a safe state. So the idea is that we want to in all cases, fail safe, we want to fail close. So if there's a failure, we roll back with automation. The advantage of that method is that if a failure occurs, which let's face it, you know, things happen, then we always know that we're going to be left with a working functional site, we're not going to be left in a broken state. That behaviour is all configurable within our pipeline. So if we go for example, to our actions, we go to composure, we can define what happens when an error occurs, what do we do? You know, do we carry on do we stop? What are things like timeouts? Do we auto retry? These are all options that we have to configure straight here within the user interface.

Okay, and so another question I have is a bit more related to what other house hosting companies about for from AWS, are you using?

Do you want to talk to this silver or shall I talk to this? Alright, go ahead. Okay. So, historically Cyber-Duck, were using Rackspace, which is a well known kind of UK based consultancy company providing mostly kind of physical hardware but also more recently, cloud services and consultation I think, you know, Rackspace are pivoting now towards cloud as well. But kind of historically, it was more kind of bare metal servers, you know, physical machines and racks. And that worked very well for Cyber-Duck, you know, when the relationship began, and it's worked very well for a long time. But as we've adopted more DevOps principles, those kind of traditional relationships didn't work. So well, we needed direct hands on with the cloud platforms themselves as our internal engineering teams grew. So nowadays, we have a small relationship with Rackspace. But for the most part, we're working with AWS with Microsoft on Azure. We're working with Acquia on their various cloud platforms. And there are a few smaller engagements as well Laravel Forge, for example, where we're doing larval development, but the main kind of three really are AWS, Microsoft, Azure and Acquia. Were with I think, all three right, silver partners with partners, Microsoft,

they on three, so depends on the client's requirements, or what's best suited for a job, basically. So yeah, with the software relationship, some client wanted the, the no code editor of Drupal, to kind of drag and drop landing pages and blocks on pages and the Acquia sites Studio is a great product for that. And and obviously, it's bundled within the, their hosting services, you get one or the other comes with it, or you need the other one when you buy studio. So we kind of bundled that. And yeah, just did have minor tweaks to our deployment process was a question actually, in the chat I saw as well from selling about the Acquia deployments. I know James had prepared some some for that. You explain a bit about the BLT tool and how he's kind of hooked that up? On the Acquia side?

Yeah, I don't, I don't know, whoever was paid to ask that question. It's a great question, because I actually prepared a very small example of that. So yeah, so you know, Acquia is providing us with BLT tool, which is the build and launch tool, you know, accurate as a company are offering many, many services. Now, some are focused around kind of infrastructure and, and cloud computing, some are focused around software consultancy and, and everything in between really. They're providing also some, some pipeline functions. And while in this case, it was actually more beneficial for us to stay for the most part with Buddy for those functions. And we can probably talk more about why we chose to stay with Buddy for those pipeline functions in a minute, but for now, I'll simply say they provide lots of different services. And so for us as a engineering team as development team, it's very important to understand, you know, when you've got that many services at your disposal, which ones actually best meet the needs and the needs of the client. So in this case, we chose to use BLT to effectively build an artefact and push it up to Acquia Cloud where Aqua would then handle the automated deployment of that onto the infrastructure that they're managing. So in this case, we've got a we've got a pipeline, which is running the BLT tool. Effectively, the the first few lines are kind of internal housekeeping. But within a Docker composure, we're basically just running these two commands, which is just a straight up composer instal. And then using BLT using the artefact deploy command specifying the branch we want to deploy to. And that will take care of the deployment for us. I don't have a live demonstration prepared of this one. But the kind of process for BLT in this case is that it will, it will take the artefact that the kind of the code base it will it will sorry, I will take the code base create an artefact from it running optional kind of hooks and scripts, it will create an artefact and then push it up to a remote Git endpoint. And then once it's in that remote Git endpoint and event is triggered, and accurate handles the deployment of that into the environment that we specified. There's lots of other things going on within BLT, it's well worth reading the documentation. We're using BLT more in local development as well to enable some cool features that we don't have with other frameworks. So things like you know, linting on commits and checking of checking commits for kind of quality and some some testing and local platform hooks, all these kinds of things. We're using BLT for locally as well. That's kind of our scope of Buddy.

Yeah, but I don't I mean, at the end of the day Acquia is just another flavour of Drupal. So it's very much part of this discussion. Accurate has a way of doing things pretty much it's very, it's a very opinionated in a technical sense of the word. So when we first started using Acquia, a couple of years back, we didn't suddenly have to look at our DevOps pipelines and say, Oh, how on earth are we going to fit those with how we want to do it, there was some discussion at the time about the best way to do it. But we definitely have a good way of doing it now, which I think Gaines is talked about. So it's, it's a different way of deploying. But the same is true of some other platforms we hosted, we also host our platform, but sh, and also, government UK pass is another hosting solution. I'm personally quite involved with on a daily basis across two or three different projects. So yeah, there's there's, there's always a way we, particularly with Acquia, we wanted to carry on doing some of the things we were doing on other projects. So again, we have a constant approach across many projects. But also use those BLT tools to make best use of the great things Akriti provides. So it was quite a bit of a hybrid approach we used with

it now. Okay, yeah. Tell me, because, James, show us some basics of how you deploy Drupal. But I know that every CMS has some quirks every CMS needs some polishing near the end. So how does it look like with Drupal? Probably Duncan, we'll help

you with Drupal. And then if you read read the documentation, again, why of deploying a Drupal website is you take it back off the copy the code up. And then you have to run a run a number of commands either from your browser or from the command line. Most people here listening who are familiar with Drupal, hopefully I've heard of Drush, which is basically the Drupal command line interface shell interface Drush. So what we what we typically do for deployments for all of our Drupal websites is we have a post installation hook in Buddy, that runs three main commands, and that hopefully most people will be familiar this week. So we've run Drush update dB, which allows us to run all our update hooks with about allows us to tweak the database, delete files, those sorts of functional changes you want to deploy as part of your software update. We also do a couple of configuration inputs. Configuration imports can be configured so that they're different for each environment. So you'd have one for testing, one for staging and one for production. So we'd run that twice. The reason we want it twice is often the first one does most of everything. And then the second one tidies everything up. And then we the fourth command we run is we do a cache rebuild. So yeah, Trash Trash car. And we can build that build those train to the pipeline, which means we end up with what we're after with a one button deployment process. And we can look as James James jumped into the deployment. And what I love, what I love about Buddy, is that user interface is so intuitive, you won't need any training to use Buddy. So you can come into each one of those steps and see what it's doing. You'll see the error messages in there in red, you know that? It's good. We, I did a deployment this morning, a half past seven this morning with a big government project. And, and they they come on to the meetings with us. And they watch us to do the deployment through Buddy. And we go into those things. And we show them Buddy. It gives the client a huge amount of confidence that, that yeah, we that's what we saw last time that that's what what we're expecting to see. And it's like some of the commands just showing a code. But when you're running deployment, and clients can see the progress going through and nothing red. And moving on to the next step. It gives them a large amount of confidence. And we're more than happy to show trans Buddy from a personal point of view, I just love the way Buddy looks. It's it's great. And very occasionally we've we've been you've made a change and we've gone without like that and you can there's a chat icon, bottom corner, I've gone into site. I don't know what you've changed, and they said a few people have said that well said we'll change it back. And lo and behold a couple of weeks later, it's fixed. And that's for me and I know this is a this is a Buddy webinar. But that's it very responsive, just just to make small changes to make our lives easier checks in the face, as I say. So, to answer the question that there is a whole lot of post installation that happens in a Drupal project, we can automate that through by the pipeline's

interesting question from Mark just now then Ken, would you give the client you did this morning the release? Would you give them the trigger to deploy? Confidence in Buddy for them to do it? Yeah,

we could do. We don't, as it happens, we have quite a sophisticated runbook for this client. They're a government organisations we have, we have a lot of checkpoints we go through, but we also share the runbook with them. So they can see us, you know, saying, Yeah, dun, dun, dun, dun, dun, dun, quite near the bottom, it's deployed through Buddy, hit the button. So it's, it's a long...

We do do that for some clients. So we have specific cases where clients approached us for DevOps Services, you know, if they want to help us to use our knowledge to apply to their business to improve their delivery, they've heard about the benefits of DevOps, they'll come to us and say, you know, we'd like you to consult with us on how we can improve our release processes. DevOps culture, will actually build pipelines for them in Buddy, you know, maybe even build infrastructure using infrastructure as code tools like TerraForm. And we'll build the AWS environment for them. And we'll build the Buddy pipeline. And they will perhaps have their own development team, their own software engineers, and will kind of build those tools, educate them on the culture, provide the consultancy, and then kind of hand it over to them. And they then do run their own deployments through the through the tooling. So I think the answer is like What Sal said earlier, right? The best tool for the job, and also the case that each, each organisation is usually different, right? So different people, different companies need and want different things. And so what's really important is the flexibility. This is something that we've not kind of really touched on. But it is very important to us as a company that Cyber-Duck that Buddy allows us to deploy to any platform, you could imagine, really, with any kind of technology, you could imagine. If you can script, if you can automate it, you can use Buddy to deploy it to enable that process to enable the DevOps culture within your business. So for example, if we're deploying a Drupal site to Acquia, or a Laravel, app to AWS, or anything in between, right, we can use Buddy to enable common workflows that allow us to do those things. So we don't have to use Azure DevOps for the Microsoft Azure stuff. We don't have to use code commit on AWS, or pipelines, we don't have to use Aqua pipelines, if we don't want to, we can kind of contain our knowledge and our processes within one location, and then just build out integrations as required. I mean, most of them are out of the box anyway, with Buddy. So

yeah, and if if needed. Also, it's also very easy to change this one block. So for example, you were using AWS, but at some point, you want to switch to Azure, or whatever. So we are just changing one block and the rest of the pipeline stays almost the same. So this is also something something great. And tell me, because we talk about deploying things, but how else do you use Buddy? What else do you automate using it?

I'll jump in really quickly. Some of the some of the things we use it for. Strangely gov.uk Pass has no no ability to create cron jobs. So we have periodically running scripts on our Drupal websites that effectively replace Cron. So that's, that's one thing we use. By default. James has already talked about an a CI CD. We also use it for obviously, running automated testing, which is triggered based on commits, and things like that, so that you can use it for a lot of things. The deployment I was talking about earlier on the slide is a bluegreen deployment. So there's a there's an ability in Buddy to come to a point and pause and say, Are you ready to continue? And at that point, our client goes in does a smoke testing. Yeah, we're happy to swap over. We click a button, it stops two instances over blue green swaps. I never know which ones which blue, green or green blue. Anyway. And again, it's automated. So we know it's worked. It just it just brings us a huge amount of confidence, working on jumping in about anything else James testing, or

we're using a few we're using it in a few different places. I mean, we kind of in some cases, we're almost using it as a kind of drop in for services like AWS lambda where we want serverless compute right? So there are cases where so give you a really solid example and Do we use a very well known off the shelf software as a service product to host our Git repos? I don't think it's super private. We use Bitbucket for many of our projects. And one of the main challenges of using Bitbucket cloud for, for us has been that there's no kind of, you know, being a software as a service product, there's no kind of out of band backups of that, right. So, you know, Atlassian is obviously a huge company. And you know, we trust in them in their software as a service product. But at the same time, we wanted that guarantee that we could have a local offline out of band backup of all our Git repos, you know, if something catastrophic were to happen to Atlassian, and I'm sure it never will, but build redundancies into your systems and all that, we wanted to take an offline copy of our backups. So what we did is actually built a Buddy pipeline, that would execute on a cron using Docker, and built a very small kind of micro service in Laravel in PHP, to enable us to query the Bitbucket API, pull away copies of each of our repos and zip them up, encrypt them, transfer them to an off site backup location. And we do that on a cron, every month within Buddy. So that's a really good example of how we're using it not just for deployments, we're using it not just for continuous integration, not just scheduled tasks on platforms that don't support it, but also to support our business operations. That's almost an IT function, you know, we're kind of, we're empowering every part of the business really using Buddy as a tool. So it's been very versatile for us.

And then there's new been Google Lighthouse released an API recently as well. And we're looking at ways to automate accessibility testing performance testing, because there is an API endpoint, we can write a script and trigger it with, Buddy. So on top of the, the AWS monitoring tool, you could also run a post deployment check on performance or accessibility or pre deployment, probably I would say for for accessibility, like render the code and make sure everything is still compliant. Or you could have those triggers either pre or post deployment to to add additional confidence to the business in other areas, security pen test as well potentially make sure we don't think something was we started exploring as well. So it's something you could run as well to make sure that every deployment you're not opening up a new vulnerabilities. And that's, that's all the future we're looking into. Now,

James mentioned, we use it for our own backups, but it's something you can also you could allow devs to take a backup of service as well, from from Buddy, you, don't you try and keep devs away from the shell and the live system bases and all that. But quite often, you will, you will need a local copy of the production database or certainly a sanitised copy of the local database, we tend to employ a solution where we will clone from production to staging and part of that planning process. We obfuscate the data, but and then without the devs to get very easily to those staging backups, which they think then they can bring locally. And they know they're working on the most recent copy of the database. So all those things can be done in in Buddy. Now the latest it's a tool, which is created for deployments, you can use it for a lot more.

Yeah, I mean, I can tell you that at some point while playing around with IFTT I connected Buddy to Spotify and every time my push to server succeeded plate I have the Tiger I mean after to push us a bit annoying but yeah, I did it I did it work. So when it comes to practical use cases, I think my my was the least useful. But if someone needs to it's it's really possible and

implement that on all our all our Buddy quite nice, James, raise a ticket. Hold on. I'm

just gonna Duncan's access Hold on. With great power comes great responsibility to use a cliche.

Yeah, exactly. But that was true. And said, because you you mentioned a bit about accessibility. And I was thinking, what else are you thinking about implementing in your in your pipelines? Because yes, this accessibility thing, this is something very important. I can tell you already that it's that Buddy like plays really nicely with X CLI. So this is this is the first example and this is also a great way to start testing applications. But what else?

Yeah, it says the main the main thing is security, accessibility and performance. already, because it impacts now SEO impacts kind of most of the business metrics. It's the law for accessibility from all public institutions. There's a thing, there's a lot more improvements we're looking at on the front end as well in terms of compressing things, caching assets. But yeah, there's, again, there's, there's a lot of tools. So we need to find what works best for our process, start, start small, do build a proof of concept, and get management and clients on board. And then we can expand our services on that which inflates build one and reused many times once as Genesis playing once we once we've done it, we can clone it to all our other clients. So being great benefits for everyone.

Yeah, yeah, I mean, I really can't wait, when you will start adding those things that we'll have for sure. Next topic for it for a webinar, right? Evolve, how to evolve your pipeline during during the year or our months. We have some questions from from from the audience that we got before from from the signups. So I think it's time for for for the q&a. So let's start with this one, what is the best deploy process

I would start I would say it's whatever works for you, it really depends on what stage of the journey you're on. If you if you're still deploying manually, then the best one would be one that saves you time or one that gives you more confidence. So you don't have to feel like the next release is going to be the most stressful part of your week, or have to wake up in the middle of the night to fix something. So just start start small automate the most time consuming and the riskiest part of the process or whatever it could be. And, and yeah, that would be that would be the best one for you. But obviously for for our more advanced and corporate or public sector deployment when we've been improving them for for several years, then that's going to be continually improving. So thing that's that's it's hard to say which one will be better with the Lord invest in the future.

Like James said at the beginning, don't DevOps, the term came from the agile methodology. And in Silvan, sort of said the same thing they're effectively is that the best thing to fix is your biggest pain point, or the things that that thing that keeps going wrong the most, or the thing that which you have to struggle around scraps of paper and work out how to do things, automate those things. And I'm a big believer in in constantly changing one thing, and seeing whether it works. And if it didn't work, you back it up. So the best deployment process is probably the next one you're about to write. Because you should always be constantly improving. Agile is not about being nimble, so much. It's about constant iteration of constantly improvement. So the next best deployment process is the one you're gonna write tomorrow.

Yeah, that's that, that's for sure. The next question we have, how to ship database changes using Drupal.

We've sort of already covered that, that's the update DB script, which we can run as part of post installation. So Drupal has a really good mechanism for running post update hooks. And those post update hooks sit at module level and indeed, a core level. So basically, what happens is you can write a script to PHP script with a particular designation. And when you do the deployment, Drupal will look through all those scripts and go run that one that run that one, I've run that one, I've not won that one, I run this one. And it basically gives you a list of all those it's about to run, and then it will run them. So you can use those scripts to do all manner of things all manner of things. A good use case is a a contributed module might change the database schema, and it needs to move some columns around into the new schema. And that will happen automatically when that module updates in Drupal. So you may not even know that change is about to happen. But running the update DB script, just automates that process. And because we're automating that running of the update DB script, as part of the Buddy pipeline, then all those things will happen automatically. And of course, you can write your own custom scripts as well which run as part of that process. If you need to do any, any housekeeping or creating content automatically. You can do that as part of the custom module, custom update. It's, it's barely into Drupal, and it's initiated through the Buddy pipeline.

Understand Yeah, so in short, it depends, right? Yeah, or favourite? It answer for everything. And the next thing, how to break the traditional engineering habit to a DevOps mindset for engineers and management. Yeah, this is this is really something interesting.

That is a few points. I guess, for me looking, I guess from the top would be, as I said, it's it's a bit of an investment. And there might be some, some pushback because it is, it is a change. And maybe James can talk about the cultural element did at the beginning, but it's about bring bringing confidence, sediment, the main thing that the way to break is to, to break that mistrust, I guess, or to change the habits is to show the benefits like because you can have even the tiniest step added to your process that can make a huge difference. And, as I said, or pipeline have been built for for several years, but start small, like show, show them show show your team, how safe and how much better it can be trust by automating the most painful element and, and that's, that could be the beginning of change. I think that's, that's, that's how it started at Cyber-Duck. And that's how we were explaining it to clients. And yeah, it is, it is a change, but you don't have to, you don't have to send the whole company on the on the three weeks training and the whole kind of culture will take months or years to build. So just start small, that's one one, start with the development manager or the marketing team or, and then then it will climb up to the rank and permits the other departments will be my my answer.

Yeah, it's basically spot on it. Just Just to reiterate, really, and re emphasise what you've already said Silver's that, you know, I think the first and most important thing to consider when we're talking about changing, the way that we work is that you need that buy in on every on every part of the organisation. But But perhaps most importantly, you need it from the kind of management and the leadership teams in the business. Because, you know, DevOps, as we said, as we've said, it's becoming a bit of a broken record now, but DevOps is a culture. And so you need to adopt that in every part of your business process. So if we take the traditional, like the agency example, DevOps culture has to be a part, you know, part of the marketing effort, or part of the new business effort as a part of the kind of initial exploration and scoping efforts. And then, you know, many steps down the line, it has to be part of the technical efforts in, you know, kind of release engineering and development, and so on and so forth. And then post release support and refinement. So, you know, if we, if we try to approach this as a thing of kind of, how do I swap from tool a, tool B, you know, we're only ever going to get limited benefit, you know, if you if we're thinking in terms of how do I swap from FTP to Buddy, we've got a small piece of the pie. But we're missing out on the whole cultural change that can really benefit the organisation. If we tackle it as a cultural shift. First, as I say, that requires, perhaps most importantly, buy in from company leadership, because they'll be able to, you know, work with the kind of work and kind of set the agenda for the teams in the business and the kind of strategic plans of the company.

And I think it depends on the organisation as well, it could be, I was thinking like how it happened at CyberArk. That was bottom up. So there's the team on the ground, struggling with deployment and through r&d, and just a general interest in kind of exploring new tech and just changing the next JavaScript framework or the next, the next fancy, fancy language. They we realise that, so that yeah, they started implementing it on one project. And then that came up through the ranks of the company and say, that makes sense and to reach the CTO. So yeah, we should do that more, and then saw the benefits of it. And then clients loved it. And then it became a whole gem centre, it's part of our marketing strategy. And everyone is, is raving about it. So that was bottom up. There might be situation where the new CTO coming in plays straight from the top. And there's been a legacy culture where engineers might just, you know, go trend meals dive like I've been doing it for 10 years. That's, that's right. I can continue doing it and resistant to change, but then the CTO say, now we need to move to the cloud, we need to implement DevOps. And then there might be some resistance but the best teams are the one that embrace that change. And at some developers might disagree and leave but if it's if it's the right decision for the company, Then you'll be left with a good team, wherever everything's aligned and, and the team will learn new things. It's very exciting as well as developer might get bored of doing always the same thing or that's innovation is also very exciting. So that's that could be also from from the top and then going down to the dev team, but depends on the company, right?

Yeah, well, I totally agree. Because every time when I try to explain people, how to start implementing CI/CD, the first thing I always say, and this is something that I always repeat, I emphasise is never ever try to implement everything at once. Because when you start reading about CI/CD, you will find about deployments about unit testing, functional testing this and that. And this is overwhelming, this is scary. And the moment when you try to implement everything at once, when you don't have this culture you don't have. You don't have a habit of writing tests, you don't have a habit of automation, nothing of it, it will just crash and burn. And the only thing that you will waste is some time and the money you'll spend on some applications that should help you out automate this step by step.

Yeah, I agree, I think I think the changing one thing isn't important. It's a good, it's a good way of doing things. If you change two things, and you're processing proofs, you have a pretty good guess that no one of those things work or possibly both of them. If you've only changed one thing in your process gets worse, you know, you need to abandon it. But you might be in situation when you change two things, and it gets better. But if you want one of those things made it worse. So changing one thing and small incremental changes to processes. And this works in life as well as technical delivery is a good way of working out what brings the most benefit. And that is agile at the end of the day, isn't it, we're talking about small incremental changes that bring the biggest value. And, you know, DevOps is a commerce isn't overhead, you have to have that function in your business. And that overhead is only worth it, if it brings value. So you know, you need to know you need to know, need to make sure need to have competence yourself as a business that the DevOps task DevOps changes you're putting in place, or making a big difference to your business. Otherwise, how would you know what progress are a second DevOps?

True, true? And the last question we have is, What would you say is the most common blocker to business DevOps processes?

could be in terms of the kind of deploying that inside the company or business processes? Yeah, it could be either the, the lack of understanding, I guess, where it could get quite technical, as you said, when talking about that. So under the lack of understanding where again, they've been doing it, and why not be very efficient, but they haven't seen better. So that's, that could be one way of resisting that change. Or the other way it could be, it's too overwhelming, because you've just Google DevOps tools or something as this diagram with 150 icons, and you don't even know where you wouldn't know where to start. So,

I mean, when you when you when you Google for DevOps, you see these these arrows that did go infinity. So what?

Yeah, that's that explained the whole cultural thing around kind of the process at all stages and continual improvement that then can explain so I think that would be the most common Blocker with that either lack of understanding of the benefits of it, or it's overwhelming and resistant to change. That doesn't explain earlier just need to start small and show benefits and then can only grow from there.

Yeah, I think that that whole, people are generally scared of change. But you have a situation these days where 510 years ago, you can have one developer doing their coding locally FTP it up to a piece of metal on a rack, somewhere in a note in a in a in a server farm, things have changed. We have much more specialists these days, generally, not just in technology. So I, I lead teams that develop develop Drupal websites, I didn't actually do much development anymore. And I think you because we have the cloud and all these amazing tools like Amazon and all these platforms as a service, that you inevitably end up with people who have to be specialised in those areas. And that is a culture shift, which I guess James has already spoken to. So it's that reluctance to do change. If you don't change, you will just get left behind. So there is a certain amount of specialism you have to buy in, whether that's through recruiting or training, depending on the size, your business or outsourcing. But change is always hard. But but check change has to be for good reason. And convincing the people with the purses to open those purses and to spend them on the right things is, is it's not hard, but it certainly needs evidence and this sort of webinar, hopefully. Take that punch.

Yeah, so. Okay, so this was the last, the last question we have. At the start of the webinar, we launched a poll on YouTube and the here are the results. The question was, is Drupal, your primary CMS and 73% of our viewers said, yes, the triple is their primary CMS. So, so yes, that's, that's, that's really good. And it's also nice. And it's really great that the rest also stayed until the end to listen, something about not the primary CMS but to learn something new to progress. And, yeah, so. So this is, this was really great. And before wrapping everything up. Remember about our Buddy CI/CD group on meetup.com You can follow us there and you will be always informed about upcoming webinars. And like I mentioned earlier, subscribe to our YouTube channel. Thanks to this, you will be able also to get informed about the upcoming webinars and you will have access to all the other things that we already already made about mostly about CI/CD DevOps and but also, we also had a webinar about accessibility. So this is a really great place to learn something new. So, guys, thank you. Thank you very much. I really enjoyed listening. I mean, I will be one of those 26% of people for whose Drupal is not their primary CMS, but I really enjoyed it. I learned a lot. And so like I said, Thank you, thank you very much. And I really hope that at some point, we will do a part two of this right. Definitely. Okay, so, so the last thing we can do is just wave goodbye and thank you everyone for listening. Bye. Thank you. Bye bye.