Vanarama is an award-winning personal and commercial vehicle leasing company.

The Challenge

Vanarama runs its backend workloads on AWS ECS and uses AWS Organizations to manage multiple accounts. Their environments are immutable and provisioned via Terraform. Each workload is made up of multiple applications, databases, caches and other AWS services. A solid IaC implementation, infra pipelines, and robust CI/CD using GitHub Actions and Harness have supported the fast and reliable evolution of their platform. The company’s cost and cloud optimisation capability has evolved on a different path due to a focus on feature development and increasing market capture. The company’s growth was mirrored in their monthly AWS spend and, as a result, Tech Amigos were asked to help them push down on cloud costs and establish some ongoing optimisation strategies.

The Solution

Establish a picture of the cost breakdown

Using AWS Cost Explorer we established some simple cost breakdowns. How much was each workload costing? How did that break down between AWS services? For the most expensive services could we break down the usage type of the headline figure? What was the trend across the past month, 3 months, 6 months and year? What developments in the workloads and architecture had happened recently that could correlate to cost changes?

This step is essential. With this insight, targeted conversations and investigations can start on optimisation strategies that will work best for the customer taking into account their unique circumstances and upcoming plans.

Discount instruments
With discount instruments, to give them their generic name, you save money vs. the on-demand rate by committing to buy a certain amount of the resource for some future period, typically a year or more. For example, AWS offer a 63% saving vs. the on-demand rate for an EC2 m5.large in Europe if you commit for 3 years and pay up-front. That drops to 37% if you commit for 1 year with no up-front payment. These are big savings and almost all companies should take advantage of these discounts in some configuration that fits with their workloads and strategic plans.

For Autorama, this was straightforward as they were running homogenous auto-scaled ECS clusters. We looked at the options surfaced by AWS TrustedAdvisor and choose between general compute or EC2-specific savings plans taking into account what we knew was in the business and technology pipeline for the coming year. We settled on a 1 year, no-upfront EC2 savings plan and went for a conservative coverage of 50%. This decision took into account their plans to decommission some environments in the coming weeks and months.

Rightsizing
Because each environment was fully managed using Terraform it was easy to configure different EC2 instance families and sizes per environment. Infra pipelines plan and apply these changes quickly. For example, performance and traffic in the development environment was much lower than production and therefore the cluster was based on t3.mediums vs. m5.larges.

Development process
TechAmigos also helped Autorama with the transition from a “branch-per-environment” to “trunk-based” development model for all their services. A direct consequence of this was the opportunity to retire the development and pre-prod enviornments. Common patterns across CI/CD for both services and infrastructure and adherence to AWS well-architected advice on workload isolation meant that decommissioning these enviornments was straight forward with the heavly-lifting of completly deleteing infrastructure handled by Terraform.

House keeping
Cost Explorer pointed us in the direction of several areas where there looked to be easy $$$ savings with some simple changes. The first was a never-ending list of AMI images and EBS snapshots being created as a result of a poorly configured Data Lifecycle Manager policy. We replaced this with a clean retention policy setup in AWS Backup for keeping daily, weekly and monthly backups of critical EBS volumes.

There were also some pretty big S3 buckets, although the savings here were smaller. Still, it made sense to Terraform consistent lifecycle policies across the environments with more aggressive expiries in the non-prod environments. Because of Terraform and the CI/CD support in place for infra pipelines, this was a quick and cheap change to make.

Similarly, we reviewed the setup of CloudWatch log retention for all the ECS services. Many of these logs were set to “never expire”. After agreeing on the correct policies for non-prod and prod we were also able to roll this out easily using Terraform.

CloudWatch costs
Using AWS Cost Explorer we were able to drill down on high CloudWatch costs using usage types. This highlighted that 60% of the costs were for data ingestion. This triggered an investigation on the state of application logging in the main workload which uncovered a large amount of superfluous “happy-path” logging that had crept on over many months to help with initial testing and then with bug investigations. This was happening in ECS and also in Lambda. Over the next few sprints, this technical debt was addressed as the approach to logging became more consistent and targeted with logging that was relevant, useful and properly categorised.

Summary

In total we helped Autorama cut their monthly AWS bill by 33%.

There are many different way in which your company can save on cloud costs. Tech Amigos and OptiMigo can help identify and execute the right optimisation strategies for your company. Get in touch at heyamigo@techamigos.co