cloudgnosis.org - How to avoid unexpected AWS costs

If you are a user of Amazon Web Services (AWS) cloud services, you have most likely run into the situation where you have some resources costing money that came as a surprise. They may have been resources someone forgot to turn off once they were done or something that has accumulated more resources over time - or simply cost much more than expected. There is a joke that says that AWS does not charge by usage, but by what you forget to turn off…

Note

Note 2024-01-30:_ This article was moved from Tidy Cloud AWS to here.

One useful way to avoid cost surprises is to create a budget in AWS. This will allow you to set a cost limit and get notified if your spending goes beyond that limit, or is at the risk of doing so.

This article is intentionally kept relatively simple, so you can something that is useful in place and does not require you to access the billing portal (you need permissions to view and modify budgets though).

The bulk of the work is done via a PowerShell script, called Set-SimpleBudget.ps1, plus a supporting script Get-SimpleBudget.ps1. The scripts are in the Github repository cloudgnosis/tidycloud-aws-utilities https://github.com/cloudgnosis/tidycloud-aws-utilities/tree/main/budgets/scripts.

PowerShell is a cross-platform scripting tool that works on macOS, Linux, and Windows, see section How to install PowerShell below on how to set it up.

It uses AWS Tools for PowerShell to access AWS services, see section Introduction to AWS Tools for PowerShell for introduction and Installation of AWS Tools for PowerShell for installation. These sections are at the end of this article.

Set a simple budget

Steps to run the script

There are two (three) steps to create a budget for your resources:

Install the necessary AWS Tools for PowerShell modules (one-time operation)
Set your AWS credentials to use
Run the Set-SimpleBudget.ps1 script to create the budget

Install the necessary AWS Tools for PowerShell modules

With AWS Tools for PowerShell installed, run the command

Install-AWSToolsModule SecurityToken,Budgets

This will install the required modules. This is a one-time operation and is not needed next time.

Set your AWS credentials to use

This is done via the Set-AWSCredential command in PowerShell. If you already have an AWS credentials profile on the computer, just run

Set-AWSCredential -ProfileName yourprofile

replacing yourprofile with the name of your AWS credentials profile. Or you can specify access keys directly

Set-AWSCredential -AccessKey keyid -SecretKey secretaccesskey

This will temporarily save the credentials for the current PowerShell session. As long as you use the same credentials, you do not need to run this again until either the session ends or the credentials expire.

Run Set-SimpleBudget.ps1

This is where you create the budget. The simplest case looks like this:

./Set-SimpleBudget.ps1 -BudgetName MyBudget -Amount 100 -NotificationEmail alertme@example.com`

There are three required options, BudgetName, Amount and NotificationEmail. The BudgetName is the unique name for the budget in the account and Amount is the monthly budget limit in US dollars. NotificationEmail is the email address to send budget alert notifications to. There are two notifications configured by default:

When incurred cost in a month is greater than 80% of monthly budget limit
When the forecasted cost for a month is greater than 100% of the monthly budget limit

By default, the budget is valid for the current month only and all resources in all regions on the AWS account. AWS removes the budget if the end date of the budget has passed. Below You find the description of the additional options for the script.

Getting help with the script

You can run ./Set-SimpleBudget.ps1 -? or Get-Help ./Set-SimpleBudget.ps1 to get a summary of parameters available. Run Get-Help ./Set-SimpleBudget.ps1 -Examples to get example of using the script, or Get-Help ./Set-SimpleBudget.ps1 -Full to get a detailed description and examples.

Setting budget notification thresholds

There are two optional parameters for budget notification thresholds:

ActualPercentageThreshold
The percentage of the budget limit which it will alert if the incurred cost is greater. The default value is 80.

ForecastedPercentageThreshold
The percentage of the budget limit which it will alert if the forecasted cost of the month is greater. The default value is 100.

The first option sets an alert on actual spending when you already have the cost. The forecasted cost is what you are predicted to pay by the end of the month.

Setting period of budget

The budget always starts with the current month in the script. The parameter NumberOfMonths can extend the time limit for the budget to a certain number of months, including the current month. If NumberOfMonths is set to 3 and the script is run in April, then the budget is valid until the end of June.

The budget limit amount is evaluated per month, not for the whole period of the budget.

Restricting which resources to include

The script includes two ways to restrict which resources to include:

By region
The parameter FilterRegion accepts one or more region identities (separated by commas). A region identity is the name of the AWS region in the form eu-west-1, us-east-1.

By tag
The parameters TagKey and TagValue can filter so that only resources with the specified tag name (tag key) and tag value are included. If only TagKey is specified, then it only checks for the presence of the tag name itself.

Note that the tag must have been activated as a cost allocation tag before using it in the budget. This has to be done in the billing service dashboard.

This applies also to AWS-generated tags, so this can be used to set a budget for a specific CloudFormation stack for example, by setting TagName to aws:cloudformation:stack-name and TagValue to the name of the CloudFormation stack.

Use Get-SimpleBudget.ps1

The Get-SimpleBudget.ps1 script is a script that simply returns the budget data for a budget set with the Set-SimpleBudget.ps1 script. It has a single mandatory parameter BudgetName and an optional parameter AddNotificationSummary.

The AWS command Get-BGTBudget kind of does the same thing, but the output does not provide many useful data and does not include the notifications. This script helps with providing a bit more useful data for the budget.

This is an example output from Get-BGTBudget command

BudgetLimit         : Amazon.Budgets.Model.Spend
BudgetName          : ATestBudget
BudgetType          : COST
CalculatedSpend     : Amazon.Budgets.Model.CalculatedSpend
CostFilters         : {[Region, Amazon.Runtime.Internal.Util.AlwaysSendList`1[System.String]], [TagKeyValue,
                    Amazon.Runtime.Internal.Util.AlwaysSendList`1[System.String]]}
CostTypes           : Amazon.Budgets.Model.CostTypes
LastUpdatedTime     : 4/5/2021 4:07:51 PM
PlannedBudgetLimits : {}
TimePeriod          : Amazon.Budgets.Model.TimePeriod
TimeUnit            : MONTHLY

This is the corresponding output from Get-SimpleBudget.ps1 script

BudgetName        : ATestBudget
StartDate         : 4/1/2021 4:07:50 PM
EndDate           : 4/30/2021 4:07:50 PM
BudgetKind        : Monthly cost
BudgetLimitAmount : 80.0
BudgetLimitUnit   : USD
LastUpdatedTime   : 4/5/2021 4:07:51 PM
FilterRegions     : {eu-west-1, eu-north-1}
FilterTags        : Project = SolutionPilot
Notifications     : {@{Operator=GREATER_THAN; CostType=ACTUAL; Threshold=120; ThresholdType=ABSOLUTE_VALUE; SubscriberAddress=alerts@example.com;
                    Description=Actual incurred cost > 120 USD (EMAIL: alerts@example.com)}, @{Operator=GREATER_THAN; CostType=ACTUAL; Threshold=80;
                    ThresholdType=PERCENTAGE; SubscriberAddreess=alerts@example.com; Description=Actual incurred cost > 80 % of monthly budget limit
                    (EMAIL: alerts@example.com)}, @{Operator=GREATER_THAN; CostType=FORECASTED; Threshold=100; ThresholdType=PERCENTAGE;
                    SubscriberAddress=alerts@example.com; Description=Forecasted cost for the month > 100 % of monthly budget limit (EMAIL:
                    alerts@example.com)}}

In addition, it can also include a field with a textual summary of the notifications, which consists of the Description fields of the notifications:

BudgetName          : ATestBudget
StartDate           : 4/1/2021 4:07:50 PM
EndDate             : 4/30/2021 4:07:50 PM
BudgetKind          : Monthly cost
BudgetLimitAmount   : 80.0
BudgetLimitUnit     : USD
LastUpdatedTime     : 4/5/2021 4:07:51 PM
FilterRegions       : {eu-west-1, eu-north-1}
FilterTags          : Project = SolutionPilot
Notifications       : {@{Operator=GREATER_THAN; CostType=ACTUAL; Threshold=80;ThresholdType=PERCENTAGE;
                      SubscriberAddreess=alerts@example.com; Description=Actual incurred cost > 80 % of monthly budget limit
                      (EMAIL: alerts@example.com)}, @{Operator=GREATER_THAN; CostType=FORECASTED; Threshold=100; 
                      ThresholdType=PERCENTAGE; SubscriberAddress=alerts@example.com; Description=Forecasted cost for the 
                      month > 100 % of monthly budget limit (EMAIL: alerts@example.com)}}
NotificationSummary : Actual incurred cost > 80 % of monthly budget limit (EMAIL: alerts@example.com), Forecasted cost for the 
                      month > 100 % of monthly budget limit (EMAIL: alerts@example.com)

AWS permissions

To run the scripts successfully, the following AWS policy statements need to be set, or any policies that include these permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "budgets:ViewBudget",
                "budgets:ModifyBudget"
            ],
            "Resource": "arn:aws:budgets::123456789012:budget/*"
        },
        {
            "Effect": "Allow",
            "Action": "sts:GetCallerIdentity",
            "Resource": "*"
        }
    ]
}

Replace 123456789012 with actual AWS account Id. If only specific name patterns for budgets should be allowed, adjust resource patterns accordingly.

Summary

The Set-SimpleBudget.ps1 and Get-SimpleBudget.ps1 are scripts to facilitate one-off time-limited budgets for AWS resources in an account. Use them as-is to create simple budgets and avoid getting a surprise AWS bill. Or modify them as it fits you.

Script sources

The source for Set-SimpleBudget.ps1: https://github.com/cloudgnosis/tidycloud-aws-utilities/blob/main/budgets/scripts/Set-SimpleBudget.ps1
The source for Get-SimpleBudget.ps1: https://github.com/cloudgnosis/tidycloud-aws-utilities/blob/main/budgets/scripts/Get-SimpleBudget.ps1

Appendix - How to install PowerShell

A starting point for finding information about PowerShell is the Microsoft web page https://aka.ms/powershell. This page contains an overview of PowerShell, documentation, examples, installation instructions, and much more. From there you can get to the installation instruction pages. You can also use the link https://aka.ms/get-powershell to directly go to the installation instructions.

Caution!
By default installation of PowerShell 7, the current version at the time of writing will replace older installations of PowerShell Core, e.g. PowerShell 6.x and earlier versions of PowerShell 7. If you want to retain an older version and install the current version in parallel, please check the instructions for installation via binary archive (.tar.gz file) on the web page above.

Appendix - Introduction to AWS Tools for PowerShell

Amazon Web Services (AWS) provides several tools and frameworks for working with and managing AWS resources. There are tools for many programming languages, multiple command-line tools, as well as their Web-based interface known as AWS Console.

One of these toolsets is targeted specifically towards PowerShell users and its name is AWS Tools for PowerShell. It is an open-source project and the source code is available on Github at https://github.com/aws/aws-tools-for-powershell/. It is available from PowerShell Gallery (https://www.powershellgallery.com) as an add-on modules to PowerShell, and it is available as a zip file download from AWS as well.

There are three different package approaches to choose from. However, the AWS.Tools modular packaging option is the one that is strongly recommended by AWS and which is also geared towards the more modern versions of PowerShell. So this is the option you should choose and the installation procedure is what we describe here in this section.

For information about AWS Tools for PowerShell you can always have a look at AWS own web pages as well, at https://aws.amazon.com/powershell/.

Appendix - Installation of AWS Tools for PowerShell

The AWS Tools for PowerShell consists of several modules. A PowerShell module is a way to package a set of features for a specific area. In our case here there will generally be one module for each AWS service.

For example, if you use S3 there is a corresponding PowerShell module with the name Aws.Tools.S3. As AWS has close to 200 different services, this amounts to quite a few modules. For the most part, you are likely to use a handful of these in your day-to-day work with AWS.

To install the different PowerShell modules, we can simply use the Install-Module command to install each module. However, AWS has provided an installer tool to facilitate the installation of AWS modules. You can use the command Install-AWSToolsModule for module installation and update modules using the command Update-AWSToolsModule.

So first step is to install the installer module:

PS > Install-Module aws.tools.installer

Untrusted repository
You are installing the modules from an untrusted repository. If you trust this repository, change its
InstallationPolicy value by running the Set-PSRepository cmdlet. Are you sure you want to install the modules
from 'PSGallery'?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "N"): y
PS >

We get the question if we trust this repository (PSGallery) since it is not registered as a trusted source. PowerShell will ask us whenever we want to install something from PowerShell Gallery as long as we do not trust it.

I do not want to trust everything from PowerShell Gallery, so I do not want to change the trust policy for it now. If I do not want to get the question when I do the installation I can also add the -Force option, for example, Install-Module aws.tools.installer -Force.

This installs a few helper commands that will help us install any AWS PowerShell modules we need from AWS Tools for PowerShell:

PS > Get-Command | Where-Object -Property Source -value AWS.tools.installer -eq

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Function        Install-AWSToolsModule                             1.0.2.0    AWS.Tools.Installer
Function        Uninstall-AWSToolsModule                           1.0.2.0    AWS.Tools.Installer
Function        Update-AWSToolsModule                              1.0.2.0    AWS.Tools.Installer

The Install-AWSToolsModule is our primary command for installing any AWS modules.