Join our community of over 9,300 readers delving into AWS. We highlight real-world best practices through easy-to-understand visualizations and one-pagers. Expect a fresh newsletter edition every two weeks.
Share
AWS Observability Made Easy: Dive Into Cross-Account Monitoring Today!
Published about 1 year agoΒ β’Β 3 min read
β Reading time: 6.3 minutes
π Main Learning: Observability Aggregation with OAM
Ever tried setting up an AWS Landing Zone? If you have, you know it's not easy. AWS recommends using a separate account just for monitoring all your log data.
We're here to introduce the AWS Observability Access Manager (OAM), designed to make this task easier.
Previously, we couldn't use OAM effectively due to a major limitation, but that's changed.
Interested in diving straight into coding? Visit our repo to deploy your setup using Terraform or CDK.
Why Use Multiple Accounts?
Using several AWS accounts helps to isolate applications, manage finances better, and use resources more efficiently.
A multi-account setup could look like this:
Multi-Account Setup
But, it's challenging to oversee operations across these accounts. That's where a dedicated monitoring account comes in, gathering all logs, traces, and metrics in one place.
Aggregating Log Data
Manually aggregating logs across accounts and regions can get complicated. Our own Eduardo Rabelo has an amazing blog post on different strategies for Cross-Account and Cross-Region Log Aggregation.
One common approach is to build a manual solution with Kinesis to ingest all logs. See this architecture:
Log Ingestions with Kinesis
This can be very error-prone and development-intensive.
OAM offers a simpler solution, allowing you to view all application components in one place, analyzing data across accounts.
What is the Observability Access Manager (OAM)?
The Observability Access Manager (OAM) is a tool and set of APIs built by AWS. It helps you set up a dedicated monitoring account in a multi-account environment.
OAM allows you to monitor all the components of your applications from a centralized view.
It creates cross-account observability to search, analyze, and correlate data stored in CloudWatch.
We want to get observability over data such as:
CloudWatch Logs
CloudWatch Metrics
X-Ray Traces
from multiple accounts into one dedicated Monitoring account. This tool is a managed solution of what we wanted to build in the first step.
Sinks Point to the Monitoring Account and Links Send Application Data
β
OAM uses three main components:
Sink: The monitoring account destination.
Link: Connects your application's account to the monitoring account.
Sink Policy: Allows connections from source accounts to the monitoring account.
Multi-Account, Multi-Region Setup
We see three AWS accounts in the previous picture:
Source Account A
Source Account B
Monitoring Account
Imagine both your Source accounts are running two different applications, such as a REST API in A and User Authentication in B.
One user request touches the authentication account and creates logs. It also touches the REST API and creates logs. We need to have one central place to correlate the logs. This is possible with a monitoring account.
OAM helps us with that. You can create a single Sink in us-east-1 in the Monitoring account. You create a Link in the AWS accounts running workloads where logs, metrics, traces, and insights are generated.
The Limit That Made Us Not Working with OAM in the Past
Before, OAM had a one-sink-per-account limit, making it difficult to monitor applications across multiple regions.
Imagine having our account structure again like in the following image:
β
Multi-Region Setup with one Sink limitation
In this setup, we have two regions (us-east-1 and ap-southeast-2). But since we could only create one sink per account we can only aggregate all logs from us-east-1.
This doesn't help us when our actual application is scattered across regions, which it often is.
βSteve Weldon (thanks!) found out that it is actually possible to create multiple sinks per account. That means we can achieve the following setup:
β
Multi Region, Cross-Account OAM Setup
The official quota still states that only one Sink per account is possible, so we're eagerly awaiting some feedback from AWS:
This repository assumes that you use AWS Organizations and know how to handle the trust relationship of the role OrganizationAccountAccessRole.
If you need help with that, please comment or DM us in our Discord server π
After you have all three accounts, you can supply the account IDs after you hit terraform apply.
Logs from Both Accounts
We've deployed a Lambda function in both source accounts to see it in action. We've executed the Lambda a few times to generate some Logs.
β
We see both Log Groups from both accounts (OAM-Source and OAM-Source-2). The logs are arriving in near-real-time. The same thing is happening with Metrics:
β
β
Summary
OAM simplifies setting up cross-account observability, saving you the trouble and cost of manual solutions.
Struggling with CloudWatch? We're working on a CloudWatch Book, aiming for a Q2 release.
Tobi & Sandro
our goal is to simplify AWS & Cloud Learning for everybody. You don't need expensive certifications to build on AWS!
Join our community of over 9,300 readers delving into AWS. We highlight real-world best practices through easy-to-understand visualizations and one-pagers. Expect a fresh newsletter edition every two weeks.
Newsletter Header AWS FOR THE REAL WORLD β±οΈ Reading time: 8 minutes π Main Learning: Migrating from Edgio to CloudFront βοΈ Blog Post π» GitHub Repository Hey Reader ππ½ this newsletter is about π₯ AI π€ We haven't talked too much about AI, Bedrock, MCPs, and agents yet - so we want to change that. Please let us know if this it interests you to build AI on AWS, or if you are much more interested on hands-on fundamentals services. Should we focus on AI Services? Yes, I want to learn to build...
Newsletter Header AWS FOR THE REAL WORLD β±οΈ Reading time: 8 minutes π Main Learning: Migrating from Edgio to CloudFront βοΈ Blog Post π» GitHub Repository Hey Reader ππ½ This newsletter edition is all about saving Lambda costs. I (Sandro is writing this one) was recently involved in saving Lambda costs for a client. So, I thought writing down my thought process was a good idea. Have fun with it! If you need help saving AWS costs or improving your infrastructure, just reply to this email! Now...
β Reading time: 8 minutes π Main Learning: Federated Authentication with Cognito πΎ GitHub Repository βοΈ Read the Full Post Online π Hey Reader ππ½ Federated Authentication lets users sign in to your app using their existing accounts - most prominently Google! This approach eliminates the need to create and remember new credentials (which most users are very happy for! β¨), improving user experience and likely increasing conversion rates. π With OAuth 2.0 and Google as an identity provider, we...