The 3 levels of AWS observability


AWS FOR THE REAL WORLD
⏱️
Reading time: 6 minutes
🎯
Main Learning: You cannot skip a level of AWS observability. Structured logs come first, then the AI agent that reads them for you.
📝

Hey Reader 👋🏽

This week I'm bringing back one of our most-read posts: the 3 levels of AWS observability. A lot of you joined after it went out in May, and the topic fits this week perfectly.

Because the real question isn't "do I have logs?" anymore. It's "who actually knows what they mean?" 👇🏽

Sponsored

What if the junior on call knew what your senior knows?

The senior engineer's knowledge (recurrence, root cause, blast radius) as a ranked problem board

In most teams, one senior engineer knows which alert matters. "Ignore that one, it fires every Tuesday." "That's the DB again." When that person is on vacation, everybody else sorts by severity and hopes for the best.

Dynatrace built an open-source app for Kiro Crew, AWS's always-on agent workspace. It puts your production problems on a board, and an agent ranks them by recurrence, root cause and blast radius, with one line on why. "Investigate" opens a chat with the evidence already loaded, and the whole thing is read-only.

You can run it locally on demo data, no account needed.

Sponsored by Dynatrace. These are my own words.

📚 From the Archive

The 3 Levels of AWS Observability

In 2022 I joined a startup with 100 million API requests a month. All on AWS, all logs in CloudWatch. I loved it.

Then the first incident hit. My boss was looking over my shoulder, and I didn't even know which log stream to open. As a self-claimed AWS expert.

Level 1: Log Hell

We had logs, metrics and some traces. All unstructured plain text, spread over different log streams. When a customer reported a missing order, we searched strings and compared timestamps by hand.

You cannot skip a level

Over the years I went through three levels of AWS observability. Each one builds on the one before. Most teams stop at Level 2, and that is the level that makes everything after it possible.

Level 3 is where I stopped opening the console. What does it take to get there, and why does it fail without Level 2?

📰 This Week in AWS

🌐AWS Lambda Web Functions

Run Express or Hono directly on Lambda, no adapter. You pay for active CPU time, not for waiting. Read More →

🔀CloudWatch log routing and chained parsers

Pipelines can now send each log event to its own log group by severity or source. Read More →

🔁Lambda loop detection for EventBridge custom buses

Lambda now stops runaway Lambda/EventBridge loops before they run up your bill. Read More →

That's it for this week.

Where is your team right now: Level 1, 2 or 3? Hit reply and tell me, I read every answer.

See you next week,
Sandro & Tobi

​Sandro Volpicella & Tobias Schmidt from AWS Fundamentals​
​
Cloud Engineers • Fullstack Developers • Educators

Manage your email preferences 📨

AWS for the Real World

We teach AWS for the real world - not for certifications. Join more than 10,500 developers learning how to build real-world applications on AWS.

Read more from AWS for the Real World

⏱️ Reading time: 15 minutes 🎯 Main Learning: A public container on AWS for $7 a month, with no VPC to own. 📝 Blog Post 🤝 Sponsored by AWS. What we built, what it cost and what we think of it are our own. Hey Reader 👋🏽 Your first container on ECS comes with homework: a VPC, subnets in two Availability Zones, a load balancer, security groups, a certificate and a task role. Express Mode creates most of it for you now (App Runner's successor, so ECS abstraction number three), but it all lands in...

⏱️ Reading time: 11 minutes 🎯 Main Learning: Measure NAT traffic first. CI runners rarely need NAT at all. 📝 Blog Post Hey Reader 👋🏽 NAT Gateways are one of those things you set up once and never look at again (probably!) 😅 Until NatGateway-Bytes shows up near the top of the bill, filed under EC2-Other - thanks AWS! I think they've made this on purpose to hide these significant networking related costs somewhere and made it hard to reflect on! The hourly fee is predictable, sure. But the...

AWS FOR THE REAL WORLD ⏱️ Reading time: 10 minutes 🎯 Main Learning: Every convenience layer AWS built on ECS is dead or dying, while ECS itself has not changed since 2014. Build on the primitive, and make the layer above it prove itself first. 📝 Blog Post Hey Reader 👋🏽 Getting started with an AWS service often feels more complicated than it should. Four concepts must be understood before a single container runs. A VPC is required before a single request. So, the fancy CLI that promises to do...