The 3 levels of AWS observability


AWS FOR THE REAL WORLD
⏱️
Reading time: 6 minutes
🎯
Main Learning: You cannot skip a level of AWS observability. Structured logs come first, then the AI agent that reads them for you.
📝

Hey Reader 👋🏽

This week I'm bringing back one of our most-read posts: the 3 levels of AWS observability. A lot of you joined after it went out in May, and the topic fits this week perfectly.

Because the real question isn't "do I have logs?" anymore. It's "who actually knows what they mean?" 👇🏽

Sponsored

What if the junior on call knew what your senior knows?

The senior engineer's knowledge (recurrence, root cause, blast radius) as a ranked problem board

In most teams, one senior engineer knows which alert matters. "Ignore that one, it fires every Tuesday." "That's the DB again." When that person is on vacation, everybody else sorts by severity and hopes for the best.

Dynatrace built an open-source app for Kiro Crew, AWS's always-on agent workspace. It puts your production problems on a board, and an agent ranks them by recurrence, root cause and blast radius, with one line on why. "Investigate" opens a chat with the evidence already loaded, and the whole thing is read-only.

You can run it locally on demo data, no account needed.

Sponsored by Dynatrace. These are my own words.

📚 From the Archive

The 3 Levels of AWS Observability

In 2022 I joined a startup with 100 million API requests a month. All on AWS, all logs in CloudWatch. I loved it.

Then the first incident hit. My boss was looking over my shoulder, and I didn't even know which log stream to open. As a self-claimed AWS expert.

Level 1: Log Hell

We had logs, metrics and some traces. All unstructured plain text, spread over different log streams. When a customer reported a missing order, we searched strings and compared timestamps by hand.

You cannot skip a level

Over the years I went through three levels of AWS observability. Each one builds on the one before. Most teams stop at Level 2, and that is the level that makes everything after it possible.

Level 3 is where I stopped opening the console. What does it take to get there, and why does it fail without Level 2?

📰 This Week in AWS

🌐AWS Lambda Web Functions

Run Express or Hono directly on Lambda, no adapter. You pay for active CPU time, not for waiting. Read More →

🔀CloudWatch log routing and chained parsers

Pipelines can now send each log event to its own log group by severity or source. Read More →

🔁Lambda loop detection for EventBridge custom buses

Lambda now stops runaway Lambda/EventBridge loops before they run up your bill. Read More →

That's it for this week.

Where is your team right now: Level 1, 2 or 3? Hit reply and tell me, I read every answer.

See you next week,
Sandro & Tobi

​Sandro Volpicella & Tobias Schmidt from AWS Fundamentals​
​
Cloud Engineers • Fullstack Developers • Educators

Manage your email preferences 📨

AWS for the Real World

We teach AWS for the real world - not for certifications. Join more than 10,500 developers learning how to build real-world applications on AWS.

Read more from AWS for the Real World

AWS FOR THE REAL WORLD ⏱️ Reading time: 10 minutes 🎯 Main Learning: Every convenience layer AWS built on ECS is dead or dying, while ECS itself has not changed since 2014. Build on the primitive, and make the layer above it prove itself first. 📝 Blog Post Hey Reader 👋🏽 Getting started with an AWS service often feels more complicated than it should. Four concepts must be understood before a single container runs. A VPC is required before a single request. So, the fancy CLI that promises to do...

⏱️ Reading time: 21 minutes 🎯 Main Learning: Same CloudFront logs: 12.5s through Kinesis, S3, Glue and Athena, about 1s through Tinybird's managed ClickHouse. 📝 Blog Post Hey Reader 👋🏽 We've been running Plausible for the analytics on awsfundamentals.com. Good software, no complaints. We still wanted the data to be ours: our retention, our schema, nobody's script in the visitor's browser. So we built the dashboard ourselves. Twice, on two different backends, fed by the same CloudFront logs....

AWS FOR THE REAL WORLD ⏱️ Reading time: 11 minutes 🎯 Main Learning: A Karpenter NodePool is a placement policy, not an instance preference. Pin one instance family, and a Spot shortage moves your fleet across availability zones, where every internal call starts costing $0.01 per GB. 📝 Blog Post Hey Reader 👋🏽 Important off topic things first: Sandro got married! 🎉We were in Munich for it and it was a fantastic day! ☀️Highly recommend a wedding over sprint planning or fighting with AWS...