profile

AWS for the Real World

AWS Observability Made Easy: Dive Into Cross-Account Monitoring Today!

Published 2 months agoΒ β€’Β 3 min read

  • βŒ› Reading time: 6.3 minutes
  • πŸŽ“ Main Learning: Observability Aggregation with OAM
  • πŸ‘¨πŸ½β€πŸ’» GitHub Code
  • πŸ“ Blog Post

Hey Reader πŸ‘‹πŸ½

Ever tried setting up an AWS Landing Zone? If you have, you know it's not easy. AWS recommends using a separate account just for monitoring all your log data.

We're here to introduce the AWS Observability Access Manager (OAM), designed to make this task easier.

Previously, we couldn't use OAM effectively due to a major limitation, but that's changed.

Interested in diving straight into coding? Visit our repo to deploy your setup using Terraform or CDK.

Why Use Multiple Accounts?

Using several AWS accounts helps to isolate applications, manage finances better, and use resources more efficiently.

A multi-account setup could look like this:

But, it's challenging to oversee operations across these accounts. That's where a dedicated monitoring account comes in, gathering all logs, traces, and metrics in one place.

Aggregating Log Data

Manually aggregating logs across accounts and regions can get complicated. Our own Eduardo Rabelo has an amazing blog post on different strategies for Cross-Account and Cross-Region Log Aggregation.

One common approach is to build a manual solution with Kinesis to ingest all logs. See this architecture:

This can be very error-prone and development-intensive.

OAM offers a simpler solution, allowing you to view all application components in one place, analyzing data across accounts.

What is the Observability Access Manager (OAM)?

The Observability Access Manager (OAM) is a tool and set of APIs built by AWS. It helps you set up a dedicated monitoring account in a multi-account environment.

OAM allows you to monitor all the components of your applications from a centralized view.

It creates cross-account observability to search, analyze, and correlate data stored in CloudWatch.

We want to get observability over data such as:

  • CloudWatch Logs
  • CloudWatch Metrics
  • X-Ray Traces

from multiple accounts into one dedicated Monitoring account. This tool is a managed solution of what we wanted to build in the first step.

Sinks Point to the Monitoring Account and Links Send Application Data

​

OAM uses three main components:

  • Sink: The monitoring account destination.
  • Link: Connects your application's account to the monitoring account.
  • Sink Policy: Allows connections from source accounts to the monitoring account.

We see three AWS accounts in the previous picture:

  1. Source Account A
  2. Source Account B
  3. Monitoring Account

Imagine both your Source accounts are running two different applications, such as a REST API in A and User Authentication in B.

One user request touches the authentication account and creates logs. It also touches the REST API and creates logs. We need to have one central place to correlate the logs. This is possible with a monitoring account.

OAM helps us with that. You can create a single Sink in us-east-1 in the Monitoring account. You create a Link in the AWS accounts running workloads where logs, metrics, traces, and insights are generated.

The Limit That Made Us Not Working with OAM in the Past

Before, OAM had a one-sink-per-account limit, making it difficult to monitor applications across multiple regions.

Imagine having our account structure again like in the following image:

​

In this setup, we have two regions (us-east-1 and ap-southeast-2). But since we could only create one sink per account we can only aggregate all logs from us-east-1.

This doesn't help us when our actual application is scattered across regions, which it often is.

​Steve Weldon (thanks!) found out that it is actually possible to create multiple sinks per account. That means we can achieve the following setup:

​

The official quota still states that only one Sink per account is possible, so we're eagerly awaiting some feedback from AWS:

​

Deploying Your Setup!

We don't come without code 😎

​Eduardo Rabelo built the whole flow in CDK (SST/ION) and Terraform. Check out the GitHub repository for that.

This repository assumes that you use AWS Organizations and know how to handle the trust relationship of the role OrganizationAccountAccessRole.

If you need help with that, please comment or DM us in our Discord server πŸ˜‰

After you have all three accounts, you can supply the account IDs after you hit terraform apply.

Logs from Both Accounts

We've deployed a Lambda function in both source accounts to see it in action. We've executed the Lambda a few times to generate some Logs.

We see both Log Groups from both accounts (OAM-Source and OAM-Source-2). The logs are arriving in near-real-time. The same thing is happening with Metrics:

​

Summary

OAM simplifies setting up cross-account observability, saving you the trouble and cost of manual solutions.

Struggling with CloudWatch? We're working on a CloudWatch Book, aiming for a Q2 release.

Tobi & Sandro

our goal is to simplify AWS & Cloud Learning for everybody. You don't need expensive certifications to build on AWS!

Dr.-Otto-Bâßner-Weg 7a, Ottobrunn, Bavaria 85521 Β· Unsubscribe Β· Preferences​

AWS for the Real World

by Tobi & Sandro

Join our community of over 8,800 readers delving into AWS. We highlight real-world best practices through easy-to-understand visualizations and one-pagers. Expect a fresh newsletter edition every two weeks.

Read more from AWS for the Real World

βŒ› Reading time: 11.5 minutes πŸŽ“ Main Learning: Host Web Applications on AWS with the CloudFront Hosting Toolkit πŸ‘¨πŸ½πŸ’» GitHub Code πŸ“ Blog Post Hey Reader πŸ‘‹πŸ½ Happy New Week! I (Sandro) will attend the Serverless Days in Milano next week where Jeremy Daley will hold the keynote. I look forward to meeting many of you and the overall AWS community. This week's newsletter is all about hosting your frontend on AWS. AWS launched a new way of deploying your frontend to it: The CloudFront Hosting Toolkit....

10 days agoΒ β€’Β 6 min read

βŒ› Reading time: 7 minutes πŸŽ“ Main Learning: Visual Regression Testing with AWS CloudWatch Synthetics πŸ‘¨πŸ½πŸ’» GitHub Code πŸ“ Blog Post Hey Reader Keeping your website running smoothly is important.It's also important to keep an eye on its visual appearance and that it doesn't change due to an unwanted change. AWS CloudWatch Synthetic Canaries is a tool that helps you check on your websites (and APIs!) to make sure it’s working well and look as expected. In this newsletter, we’re going to show you...

about 1 month agoΒ β€’Β 4 min read

Hi Reader and happy May the 4th ⭐. We want to start the spring by giving you an amazing discount on our book and other resources. For that, we have partnered with four different AWS Creators around the globe. You can use the code AWSFUND30 to get 30% off. The resources are: πŸ“™ AWS Fundamentals - Like this newsletter, it covers the basics of essential AWS Services for real-world applications. πŸ“˜ The DynamoDB Book - Level up your DynamoDB modeling and finally understand Single-Table Design. πŸ—οΈ...

about 2 months agoΒ β€’Β 1 min read
Share this post