Tidy Cloud AWS issue #17 - carbon footprint tool, containers, CUE

Hello all,

Welcome to the next issue of the Tidy Cloud AWS bulletin!

This issue of the bulletin has a few notes on the AWS customer carbon footprint tool, about running container-based workloads in AWS, and also a bit on an exciting configuration language project called CUE.

Enjoy!

Carbon footprint tool

One exciting piece of news is that AWS has launched its Customer Carbon Footprint Tool. This is available as an additional report section in the AWS Billing Console section. It is a good starting point to evaluate the environmental impact of your AWS workloads, although there is room for improvement.

I would like to see some more automation and processing options around this, which do not seem to be there in the initial release. This is a usual pattern from AWS though, and hopefully it will get better.

Take a cue

If you are working with various deployments and workloads, chances are that you have various configuration files in place. Chances are also that the format of these configurations files might be YAML, JSON or perhaps TOML. You might also work with JSON Schema or OpenAPI schema definitions.

JSON is a very simple format to handle. It is not always easy to read for humans though and very limited as a configuration format. It was not what it was intended for, after all.

YAML is more readable to humans, but also can trip you up easily, with its indentation-based structure, and some slight esoteric features that may cause problems.

TOML tried to be something in between, more or less - more readable than JSON, easier than YAML.

None of them have any built-in schema handling or validation, though, and not any particular features to handling large-scale configuration management.

I have been bitten several times by failed deployments or executions. Because of a typo, or a number was used where there should be a string, or simply wrong indentation in a YAML file.

Recently I discovered, via the Dagger project, the configuration language CUE (Configure Unify Execute). It is an interesting project, which addresses many concerns with other formats.

I have written an introduction to CUE, to use with validating YAML configurations. Check it out!

Containers in AWS - tools

In AWS, there are a huge number of ways to run containers. For general purpose execution, it sort of boils down a few alternatives though:

  • Elastic Container Service (ECS) - This is AWS’ own container orchestration solution. You can set up ECS clusters to run containers fairly easily and it is fairly well integrated with several other AWS services.
  • Elastic Kubernetes Services (EKS) - This is AWS managed Kubernetes service. Kubernetes is the de facto standard container orchestration solution, if you want to be provider-agnostic. EKS integrates with AWS services, although not as much as ECS does it.
  • App Runner - this is a new kid on the block, which aims to make it very easy to run applications - for some solutions you do not even need to build a container yourself, just point to the source code and App Runner handles the rest. It can use regular containers as well. I would consider it a more modern alternative to Elastic Beanstalk, although it does not have feature parity yet.
  • Lightsail - intended as an easy-to-use alternative to run workloads in AWS, which includes containers. It does not have a dedicated command-line tool, only AWS CLI, so I will leave this one out for now.
  • Elastic Beanstalk - older service that can run containers. Not a bad option per se, but will focus on the more modern alternatives and not cover this one further.

For these, you can work with AWS CloudFormation, Terraform, and also AWS Cloud Development Kit (AWS CDK). Terraform CDK and Pulumi should work as well.

However, what I want to point to in this case are the command-line tools that aim to simplify matters a bit, in particular when using either ECS or EKS.

Run ECS or Kubernetes (EKS)?

Which container orchestration solution to pick? As always, the answer is that it depends. For some simple and specific use cases, App Runner is likely the easiest one. My guess is that it may run ECS under the hood, but you are not really exposed to that directly.

For smaller organisations, and organisations where a team owns application and infrastructure for it, ECS may fit well - at least if you are all in on AWS.

For more complex organisational structures and workloads, Kubernetes may be a better fit. Also, if you want a solution that will work similar to multiple cloud providers, as well as on-premise, then Kubernetes is the only alternative.

It will take more effort to get started properly with Kubernetes though, even though it keeps getting better.

Command-line tools

For App Runner and ECS, AWS provides a neat command-line tool called AWS copilot that simplifies setting up infrastructure to run containers. It has limitations in what kind of container workloads you can run with it, though. It makes it pretty easy to work with and maintain ECS-based (or App Runner) workloads, simpler than using CloudFormation or AWS CDK.

Under the hood, it will generate CloudFormation for the infrastructure it handles. It can even generate CI/CD pipelines for your deployments. It is a good choice, and it is under active development by AWS.

For EKS, AWS does not provide a tool they build themselves, but which is provided by Weaveworks, called eksctl. It is the official command-line tool specific to EKS and makes it pretty easy to handle EKS cluster set-up. It will also generate CloudFormation under the hood for you.

If you use ECS, you also have an option to deploy Docker Compose configurations to ECS, as that is a supported backend solution for ECS. There is a bit of overlap between this and AWS Copilot. If you depend on Docker Compose and Docker Desktop for your work, then this may be an option as well.

If you will run EKS, or Kubernetes, then Rancher Desktop is an interesting alternative for a desktop tool. You will have a local Kubernetes cluster running as part of the Rancher Desktop setup, and you can do the same type of commands as you can do with the docker tools, plus Kubernetes command-line tools.

You are likely better off using these command-line tools than to use CloudFormation directly. If you use AWS CDK for cloud infrastructure, that will probably still be more complex than the command-line tools for many practical use cases, but also more versatile.

Unless you are thrilled with defining all infrastructure as actual code, consider these command-line tools to simplify parts of the setup.

You can find the contents of this bulletin and older ones, and more at Tidy Cloud AWS. You will also find other useful articles around AWS automation and infrastructure-as-software.

Until next time,

/Erik