Skip to content

Commit 6e62145

Browse files
cjobermaiercjobermaier
authored
WAF Optimize systems - Manage cloud costs and Tag cloud resources (#1203)
Please go to the `Preview` tab and select the appropriate template: /well-architected-framework/optimize-systems/lifecycle-management/tag-cloud-resources /well-architected-framework/optimize-systems/manage-cost/create-cloud-budgets
2 parents bbca21f + d25a0e4 commit 6e62145

File tree

3 files changed

+330
-0
lines changed

3 files changed

+330
-0
lines changed

‎content/well-architected-framework/data/docs-nav-data.json‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -462,6 +462,19 @@
462462
{
463463
"title": "Decommission resources",
464464
"path": "optimize-systems/lifecycle-management/decommission-infrastructure"
465+
},
466+
{
467+
"title": "Tag cloud resources",
468+
"path": "optimize-systems/lifecycle-management/tag-cloud-resources"
469+
}
470+
]
471+
},
472+
{
473+
"title": "Manage cost",
474+
"routes": [
475+
{
476+
"title": "Create cloud budgets",
477+
"path": "optimize-systems/manage-cost/create-cloud-budgets"
465478
}
466479
]
467480
},

‎content/well-architected-framework/docs/docs/optimize-systems/lifecycle-management/tag-cloud-resources.mdx‎

Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
---
2+
page_title: Tag cloud resources
3+
description: Implement cloud resource tagging best practices with Terraform for AWS, Azure, and GCP. Learn to automate tags, enforce policies, and optimize cost allocation using infrastructure as code.
4+
---
5+
6+
# Tag cloud resources
7+
8+
Managing thousands of cloud resources across regions, environments, and teams is complex. Tags are key-value pairs that help you manage, identify, organize, locate, and filter resources. It is important to have a clear, well-defined cloud resource tagging strategy. You can also use tags to track cost allocation and usage, and automate resource management tasks.
9+
10+
[AWS](https://docs.aws.amazon.com/whitepapers/latest/tagging-best-practices/tagging-best-practices.html), [Azure](https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/ready/azure-best-practices/resource-tagging), and [IBM](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-shared-tagging-resources) maintain best practices and strategies for tagging your cloud resources. Follow these best practices when creating your tagging strategy. You can apply these concepts to other infrastructure providers.
11+
12+
You can implement your tagging strategy using infrastructure as code (IaC) and enforce compliance with policy as code to prevent deploying resources that don't meet your tagging requirements.
13+
14+
When you implement a tagging strategy, you gain the following benefits:
15+
16+
- **Easier resource tracking:** Find resources by environment, owner, or other custom tag. For example, quickly find all servers in the 'dev' environment.
17+
- **Granular cost allocation:** Track costs by project, team, or application.
18+
- **Tag-based resource automation:** Automate resource management tasks based on tags, such as starting or stopping instances.
19+
- **Default resource compliance:** Enforce tagging policies to ensure all resources are tagged correctly.
20+
21+
## Deploy tags using infrastructure as code
22+
23+
Consistent implementation of your tagging strategy helps you track infrastructure costs, manage resources, and ensure compliance. When you use an inconsistent tagging strategy, such as manual tagging, you may end up with resources with incorrect or missing tags.
24+
25+
When you manage your infrastructure with Terraform, you can define tags within your configuration. Terraform will automatically apply these tags to all resources it creates.
26+
27+
The following creates an AWS EC2 instance and adds several tags to the resource:
28+
29+
```hcl
30+
resource "aws_instance" "web_server" {
31+
ami = "ami-0c55b159cbfafe1f0"
32+
instance_type = "t3.micro"
33+
34+
tags = {
35+
Name = "web-server-prod"
36+
Environment = "production"
37+
Owner = "platform-team"
38+
CostCenter = "engineering"
39+
Application = "website"
40+
}
41+
}
42+
43+
```
44+
45+
The AWS and GCP Terraform providers let you add default tags to all resources they create, making it easier to implement a consistent tagging strategy across the resources you manage with Terraform. Default tags ensure that all resources have the minimum required tags but you can override these default tags on a per-resource basis.
46+
47+
The following is an example using default tags:
48+
49+
```hcl
50+
51+
provider "aws" {
52+
profile = "default"
53+
region = "us-east-2"
54+
55+
default_tags {
56+
tags = {
57+
Environment = "Test"
58+
Service = "Payment API"
59+
}
60+
}
61+
}
62+
```
63+
64+
HashiCorp resources:
65+
66+
- AWS provider [resource tagging](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/resource-tagging)
67+
- [AWS default tags](/terraform/tutorials/aws/aws-default-tags)
68+
- [GCP default tags](https://registry.terraform.io/providers/hashicorp/google/latest/docs/guides/provider_reference#default_labels-1)
69+
- Learn how to [configure default tags for AWS resources](/terraform/tutorials/aws/aws-default-tags)
70+
71+
## Enforce tagging strategy
72+
73+
Once you define and implement your tagging strategy using infrastructure as code, you can enforce it to prevent the deployment of resources that do not comply.
74+
75+
### Use the Terraform validation block
76+
77+
You can use the [Terraform validation block](/terraform/language/values/variables#validation) to enforce tagging policies. The validation block allows you to define custom validation rules for input variables. You can use the validation block to ensure that the resources you tag follow your tagging strategy.
78+
79+
The following is an example of a Terraform validation block that ensures the `environment` tag is set to either `dev`, `staging`, or `prod`:
80+
81+
```hcl
82+
variable "environment" {
83+
type = string
84+
description = "Environment name for resource tagging"
85+
86+
validation {
87+
condition = contains(["dev", "staging", "prod"], var.environment)
88+
error_message = "Environment must be one of: dev, staging, prod."
89+
}
90+
}
91+
```
92+
93+
If you create a resource with an invalid `environment` tag, Terraform returns an error and prevents the deployment.
94+
95+
The following tag passes the validation:
96+
97+
```hcl
98+
environment = "prod"
99+
```
100+
101+
The following tag fails the validation due to not meeting the condition of being `dev`, `staging`, or `prod`:
102+
103+
```hcl
104+
environment = "testing"
105+
```
106+
107+
### Use policy as code
108+
109+
For more advanced enforcement of your tagging strategy, you can use policy as code tools such as HashiCorp Sentinel or the Open Policy Agent (OPA) to create policies that enforce tagging rules. You can integrate these policies into your CI/CD pipelines or with HCP Terraform to ensure that all resources comply with your tagging strategy before deployment.
110+
111+
The following is an example of a [`Pass` or `Fail` Sentinel policy](/terraform/tutorials/policy/sentinel-policy#review-your-policy) that ensures that all AWS EC2 instances have a `Name` tag:
112+
113+
```script
114+
import "tfplan/v2" as tfplan
115+
116+
# Get all AWS instances from all modules
117+
ec2_instances = filter tfplan.resource_changes as _, rc {
118+
rc.type is "aws_instance" and
119+
(rc.change.actions contains "create" or rc.change.actions is ["update"])
120+
}
121+
122+
# Mandatory Instance Tags
123+
mandatory_tags = [
124+
"Name",
125+
]
126+
127+
# Rule to enforce "Name" tag on all instances
128+
mandatory_instance_tags = rule {
129+
all ec2_instances as _, instance {
130+
all mandatory_tags as mt {
131+
instance.change.after.tags contains mt
132+
}
133+
}
134+
}
135+
136+
main = rule {
137+
mandatory_instance_tags else true
138+
}
139+
```
140+
141+
You can write similar policies with OPA and HCP Terraform. Refer to the following resources for more information.
142+
143+
HashiCorp resources:
144+
145+
- Read about the [Terraform validation block](/terraform/language/values/variables#validation)
146+
- Write a [Sentinel policy for a Terraform deployment](/terraform/tutorials/policy/sentinel-policy) to ensure that the EC2 instance has a `Name` tag.
147+
- Learn how to [define Open Policy Agent policies for HCP Terraform](/terraform/enterprise/policy-enforcement/define-policies/opa)
148+
- [HCP Terraform policy enforcement overview](/terraform/enterprise/policy-enforcement)
149+
- [Get started with Sentinel](/sentinel/tutorials/get-started)
150+
151+
External resources:
152+
153+
- Use [OPA to write policies](https://www.openpolicyagent.org/docs/terraform) ensuring all resources have tags before you create them.
154+
155+
## Next steps
156+
157+
In this section of Manage cost, you learned how to tag resources using infrastructure as code and enforce tagging policies. Tag resources is part of the Optimize systems pillar.
158+
159+
To learn more about how to manage our resources, visit the following resources:
160+
161+
- [Implement data management policies](/well-architected-framework/optimize-systems/lifecycle-management/data-management)
162+
- [Decommission resources](/well-architected-framework/optimize-systems/lifecycle-management/decommission-infrastructure)

‎content/well-architected-framework/docs/docs/optimize-systems/manage-cost/create-cloud-budgets.mdx‎

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
---
2+
page_title: Create cloud budgets
3+
description: Create cloud budgets and spending alerts using Terraform for AWS, Azure, and GCP. Implement cost monitoring, anomaly detection, and automated notifications with infrastructure as code.
4+
---
5+
6+
# Create cloud budgets
7+
8+
Cloud spending can quickly get out of control without proper oversight and management. According to the [2023 HashiCorp State of Cloud Strategy Survey](https://www.hashicorp.com/en/blog/hashicorp-state-of-cloud-strategy-survey-2023-maturity-drives-operational-efficiency), 94% of respondents experienced avoidable cloud costs. Proactive budget creation, automated alerts, and anomaly detection give you the visibility and control you need to maintain predictable spending and prevent cost overruns before they occur.
9+
10+
Implementing a budget provides you with the following benefits:
11+
12+
- **Visibility into cloud spending:** Understand where your money is going.
13+
- **Proactive cost management:** Take action before costs exceed budgets.
14+
- **Notification of spending anomalies:** Get alerts when spending patterns change.
15+
- **Ability to improve financial planning and forecasting:** Use historical data to make informed budget decisions.
16+
17+
<Note>
18+
19+
The Terraform example in this document uses the `tags` block. Refer to the [Tag cloud resources](/well-architected-framework/optimize-systems/lifecycle-management/tag-cloud-resources) document to learn about implementing a tagging strategy.
20+
21+
</Note>
22+
23+
## Create spending limits and notifications
24+
25+
Most major cloud providers offer native tools to create budgets. These native tools allow you to set budget thresholds, monitor spending, and receive alerts when spending approaches or exceeds defined limits.
26+
27+
You can use Terraform to define and manage cloud budgets across your organization. You can create Terraform modules to create budgets for different teams, projects, or environments. These modules can automatically apply appropriate budget thresholds, alerting mechanisms, and spending limits to new or existing cloud resources.
28+
29+
If you're tracking resources by tags, it is important to have a well-defined tagging strategy to ensure budgets are applied correctly. Terraform can help you enforce tagging policies and ensure that all resources are tagged consistently. Creating infrastructure manually can lead to incorrect or missing tags on resources and result in inaccurate budget tracking.
30+
31+
The following is an example of a Terraform configuration that creates an AWS EC2 budget. This budget tracks EC2 instance costs and sends an alert to test@example.com when the forecasted cost exceeds 100% of the budget. You can set similar budgets and alerts for other cloud providers, such as Azure and GCP.
32+
33+
```hcl
34+
resource "aws_budgets_budget" "ec2" {
35+
name = "budget-ec2-monthly"
36+
budget_type = "COST"
37+
limit_amount = "1200"
38+
limit_unit = "USD"
39+
time_period_end = "2087-06-15_00:00"
40+
time_period_start = "2017-07-01_00:00"
41+
time_unit = "MONTHLY"
42+
43+
cost_filter {
44+
name = "Service"
45+
values = [
46+
"Amazon Elastic Compute Cloud - Compute",
47+
]
48+
}
49+
50+
notification {
51+
comparison_operator = "GREATER_THAN"
52+
threshold = 100
53+
threshold_type = "PERCENTAGE"
54+
notification_type = "FORECASTED"
55+
subscriber_email_addresses = ["test@example.com"]
56+
}
57+
58+
tags = {
59+
Environment = "production"
60+
Team = "engineering"
61+
ManagedBy = "terraform"
62+
}
63+
}
64+
```
65+
66+
Some of the key components in the previous example include:
67+
68+
- **limit_amount:** Defines the monthly spend limit.
69+
- **notification:** Defines the notification criteria, including the recipient email.
70+
- **tags:** Applies tags to the budget resource, not the EC2 instance. Tags allow you to filter and organize budgets in the billing console.
71+
72+
For AWS environments, you can use the `aws_budgets_budget` resource to create budgets that track spending by service, linked account, tag, or other dimensions. You can specify the budget amount, time period, and notification thresholds.
73+
74+
For Azure environments, the `azurerm_consumption_budget_subscription` resource lets you create subscription-level budgets with similar notification capabilities. You can define multiple notification rules that trigger at different spending thresholds.
75+
76+
For Google Cloud Platform, the `google_billing_budget` resource operates at the billing account level, and you can filter by project, service, or label. GCP budgets support both actual and forecasted spending alerts.
77+
78+
HashiCorp resources:
79+
80+
- Learn how to [Tag cloud resources](/well-architected-framework/optimize-systems/lifecycle-management/tag-cloud-resources)
81+
- Terraform resource: [aws_budgets_budget](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/budgets_budget)
82+
- Terraform resource: [azurerm_consumption_budget_subscription](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/consumption_budget_subscription)
83+
- Terraform resource: [google_billing_budget](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/billing_budget)
84+
85+
External resources:
86+
87+
- AWS Budgets: [Getting started with AWS Budgets](https://docs.aws.amazon.com/cost-management/latest/userguide/budgets-managing-costs.html)
88+
- Azure Cost Management and Billing: [Create and manage budgets](https://learn.microsoft.com/en-us/azure/cost-management-billing/costs/tutorial-acm-create-budgets)
89+
- Google Cloud Budgets and alerts: [Creating budgets](https://cloud.google.com/billing/docs/how-to/budgets)
90+
91+
## Detect spending anomalies
92+
93+
Anomaly detection identifies unusual spending patterns rather than absolute thresholds. For example, if your monthly EC2 spending suddenly doubles from $2,000 to $4,000 but remains under your $5,000 budget, a budget alert would not trigger. However, anomaly detection would flag this unusual increase for investigation. Anomaly detection helps you catch issues like misconfigured autoscaling, forgotten resources, or unauthorized usage before they significantly impact costs.
94+
95+
Most cloud providers offer machine learning-based anomaly detection that learns your normal usage patterns and alerts you when spending deviates from the baseline. You can configure anomaly detection with AWS Cost Anomaly Detection and Azure Cost Management using Terraform.
96+
97+
The following is an example Terraform configuration that sets up cost anomaly detection with email alerts in AWS. This cost anomaly detection will detect the previous EC2 scenario.
98+
99+
```hcl
100+
resource "aws_ce_anomaly_monitor" "test" {
101+
name = "AWSServiceMonitor"
102+
monitor_type = "DIMENSIONAL"
103+
monitor_dimension = "SERVICE"
104+
}
105+
106+
resource "aws_ce_anomaly_subscription" "test" {
107+
name = "DAILYSUBSCRIPTION"
108+
frequency = "DAILY"
109+
110+
monitor_arn_list = [
111+
aws_ce_anomaly_monitor.test.arn
112+
]
113+
114+
subscriber {
115+
type = "EMAIL"
116+
address = "abc@example.com"
117+
}
118+
119+
threshold_expression {
120+
dimension {
121+
key = "ANOMALY_TOTAL_IMPACT_ABSOLUTE"
122+
match_options = ["GREATER_THAN_OR_EQUAL"]
123+
values = ["100"]
124+
}
125+
}
126+
}
127+
```
128+
129+
Some of the key components in the previous example include:
130+
131+
- **aws_ce_anomaly_monitor:** Tracks spending patterns across all AWS services including EC2, S3, and Lambda.
132+
- **frequency = "DAILY":** Sends a daily summary of detected anomalies.
133+
- **threshold_expression:** Alerts when the anomaly's financial impact meets or exceeds $100.
134+
135+
HashiCorp resources:
136+
137+
- Terraform resource: [aws_ce_anomaly_subscription](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ce_anomaly_subscription)
138+
- Terraform resource: [azurerm_cost_anomaly_alert](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/cost_anomaly_alert)
139+
140+
External resources:
141+
142+
- [AWS getting started with AWS Cost Anomaly Detection](https://docs.aws.amazon.com/cost-anomaly/latest/userguide/what-is-cost-anomaly.html)
143+
- [Azure identify anomalies and unexpected changes in cost](https://learn.microsoft.com/en-us/azure/cost-management-billing/understand/analyze-unexpected-charges)
144+
- [Google cloud anomaly detection overview](https://cloud.google.com/bigquery/docs/anomaly-detection-overview)
145+
146+
## Next steps
147+
148+
In this section of Manage cost, you learned about creating budgets and alerts to manage and control cloud spending, including creating spending limits with cloud provider budgets and detecting spending anomalies automatically. Create cloud budgets is part of the [Optimize systems](/well-architected-framework/optimize-systems).
149+
150+
To learn more about managing resources with Terraform, view the following resources:
151+
152+
- [Create reusable infrastructure modules](/well-architected-framework/define-and-automate-processes/define/modules)
153+
- [Implement CI/CD](/well-architected-framework/define-and-automate-processes/automate/cicd)
154+
- [Reduce costs with Terraform Cloud ephemeral workspaces](https://www.youtube.com/watch?v=-woCmG8yGdA)
155+
- [Tag cloud resources](/well-architected-framework/optimize-systems/lifecycle-management/tag-cloud-resources)

0 commit comments

Comments
 (0)