πŸ“˜ Update #3: Deep Dive into CloudWatch with Our GitHub Tracker Tutorial, Video Included


Hey Reader, amazing that you're interested in creating the CloudWatch Book together with us.

This is the third CloudWatch Update. Coincidentally, Sandro (I'm writing this update) turned 30 last week πŸ₯³

In this update, we'll introduce you to our newly created web application: The GitHub Repository Tracker.

While it is not the fanciest name it, the project will help you a lot in learning & understanding CloudWatch for the real world.

We thought it would be much easier to show it in a video so here we go πŸ“½οΈ

Learning the basics and theories of an AWS service like CloudWatch can be tough. In a real-world scenario, you rarely try to use CloudWatch by just reading through the documentation.

You will always have some application or workload you need to debug, develop, or fix some incident that is currently going on.

This is what our example application is there for. We want to give you a deployable application that shows you a real usage of CloudWatch.

What is the Application about?

As the name suggests the example application lets you track GitHub repositories.

You can add repositories and we will show you some stats such as

  • forks
  • issues
  • stars
  • language
  • avatar
  • last updated date

While this information is not necessarily the most interesting one we want to show you how different AWS services interact and how you can observe applications with CloudWatch.

You will especially learn how to:

  • read, and analyze Logs with Logs Insights
  • hide sensitive data that shouldn't be stored
  • add and analyze custom metrics and create alarms on top of that
  • trace user requests with x-ray with additional data like segments, annotations, and more
  • add feature flags with Evidently
  • do end-to-end testing with Synthetics
  • get analytics data with Real User Monitoring from any client
  • and more!

​

Architecture

Let's go through the architecture of the application.

​

The web application consists of mainly three parts:

  1. Frontend
  2. Backend
  3. Asynchronous Events

Frontend

The front-end code is not the core focus of this application.

​

However, if you build full-stack applications on AWS you will see clients and web applications a lot.

This is why we will also give you full access to the frontend code.

The front end is built with Next.JS and is therefore based on React. For styling, we have used Tailwind CSS.

All files are exported as static files and saved in an S3 bucket. We use CloudFront to serve the front end to all end users.

Backend (Part 1 - Rest API)

The backend gets more interesting.

Everything is built in Infrastructure-as-Code. We use Terraform and CDK.

Why both? It's pretty simple:

  1. Tobi loves Terraform
  2. Sandro loves CDK
  3. Both are widely used in the industry!

We want to give you the best out of both worlds and show you how it's done in a real-world setting! πŸ—οΈ

This pushes back the timeline quite a bit but we are sure it is worth it!

​

​

Let's go through the rest of the architecture.

The main services we use in the backend are:

  • API Gateway REST: Synchronous REST API
  • Lambda: Handling of API requests
  • DynamoDB: Storing of repositories
  • S3: Hosting of static HTML pages
  • CloudFront: CDN for delivering the frontend to every location

Everything is serverless we don't use any virtual machines or containers.

Mainly because we want everybody can deploy our application without any costs.

But we also believe that such applications should be serverless by default in the future (another discussion).

S3 and CloudFront are there to serve the front end to users.

The main part of the backend consists of an API.

We use API Gateway with a Lambda integration to handle all synchronous workloads.

Every time you read, add, or delete a repository you will hit API Gateway.

API Gateway forwards the request to Lambda, and Lambda gets the data from DynamoDB or calls the GitHub API and adds the repository to DynamoDB. This brings us to our external API -> The GitHub API.

For us, it was very important to add an external API call.

We want to show you traces, logs, and network calls within CloudWatch. Adding an external dependency makes it much more interesting.

This is just one part of the backend.

Backend Part 2 - Asynchronous Workloads with Step Functions

The last part of the architecture is our asynchronous workload.

All APIs we have used until this point are synchronous.

Adding, removing, or reading repositories from a database are synchronous operations. Asynchronous operations are the ones that run in the background.

We have implemented one asynchronous schedule that updates all repositories that you have saved. Once every 5 minutes we will update your repositories.

This happens with the following services:

  • EventBridge Schedule: A CRON job that runs every 5 minutes
  • StepFunctions: A step function that does all the processing (updating the repository, sending WebSockets, etc.)
  • API Gateway WebSocket: API Gateway will send all the WebSockets to the client
  • SES: Sends emails to the client
  • Lambda & DynamoDB: Updating the repository and storing it

​

WebSocket API

​

​

Once you open the frontend a WebSocket connection is opened to API Gateway.

The connection ID is saved in DynamoDB.

This is meant to give you an interaction with our asynchronous workloads.

​

Step Function in the Background

The background task checks if there is any update in the repository.

​

The StepFunction iterates through all saved repositories, checks if there is any update. If there is an update you will get notified via a WebSocket directly in the UI.

Additionally, you will receive an email (if you have configured your email). All of that runs in the background and is asynchronous.

CloudWatch Integrations

What has all of that now to do with CloudWatch?

Every part of this application is observable and helps you to understand CloudWatch.

User Traces with X-Ray

Each user request that happens is traceable via X-Ray.

​

Some users experience a bad performance.

Have a look at the service map and identify what is going wrong.

Feature Flagging with Evidently

Modern software development cycles use continuous deployment.

Developers want fast feedback loops and we don't want to wait for month-long releases just to figure out we broke something.

This is where feature flagging comes into play.

Evidently (see the last newsletter) allows you to do exactly that!

You can toggle features.

In our example application, we want to toggle multiple things:

  • showing or hiding extended details of the repositories in the overview, e.g. the number of stars
  • showing or hiding a full-text search bar

To make it even better, you can toggle the feature flags directly from the front end without using the AWS console.

Google Analytics from AWS with RUM

CloudWatch doesn't only help you with debugging and observability, it also helps you understand what your users are doing.

And this for a super little price.

CloudWatch Real User Monitoring is like a little Google Analytics alternative, just much easier to use.

We've integrated it with our GitHub tracker and now we can see various data such as:

  • User clicks
  • First contentful paint
  • Time to interact
  • Errors

...and much more!

All of that within AWS, within your region for a very low price.

Synthetic Monitoring with Canaries

We want to make sure our application and all of its features are working at all times.

With CloudWatch Synthetics, we've integrated continuous monitoring of our application by simulating user behavior and interactions through canaries - configurable scripts that run at specified intervals.

These canaries navigate and interact with the application, e.g. by adding or removing repositories, which allows us to verify its availability and latency, and to ensure that critical workflows are functioning as expected.

By leveraging CloudWatch Synthetics, we gain insights into the performance of our application endpoints and APIs, even when there is no traffic by ourselves.

This proactive approach helps us to detect issues early.

​

The entire book is written around this application

We have built the entire book around this application.

Each CloudWatch feature is integrated to show you its capabilities.

If we say for the real world we want to show it with real-world examples.

We could go on and give you examples for each CloudWatch feature that is integrated as well, such as Logs, Metrics, Live Tail, Logs Insights, Anomaly Detection, etc., but we believe you get the point.

​

Thanks 😊

This newsletter gave an update and an overall overview of our example application. This is the heart of the book and the whole project.

The source code will be available once we launch the book. It is built so that you can deploy it, extend it, and try it out in your environment.

Without paying anything for infrastructure costs.

Thanks for being part of that amazing journey.

If you have any other ideas for additional things to add or if you miss any services please let us know!

We read every reply.

See you in two weeks

Sandro & Tobi ✌🏽


We appreciate your interest in the CloudWatch Book! If you only want to receive the AWS Newsletter and not these updates anymore, please update your preferences here.

Best ✌🏽
Sandro & Tobi

Tobi & Sandro

our goal is to simplify AWS & Cloud Learning for everybody. You don't need expensive certifications to build on AWS!

Dr.-Otto-Bâßner-Weg 7a, Ottobrunn, Bavaria 85521 Β· Unsubscribe Β· Preferences​

AWS for the Real World

Join our community of over 8,800 readers delving into AWS. We highlight real-world best practices through easy-to-understand visualizations and one-pagers. Expect a fresh newsletter edition every two weeks.

Read more from AWS for the Real World

βŒ› Reading time: 14 minutes πŸŽ“ Main Learning: Feature Flags with AWS AppConfig πŸ‘Ύ GitHub Repository ✍️ Read the Full Post Online πŸ”— Hey Reader πŸ‘‹πŸ½ There's no other field where it's so common to have "a small side-project" like in the software industry. Even though it's possible to build things as quickly as ever before due to cloud providers, tools, platforms, and AI, many indie founders (and also large enterprises) tend to fall into the same trap: they tend to build features that users do not...

βŒ› Reading time: 17 minutes πŸŽ“ Main Learning: Observability at Scale with Open-Source πŸ‘Ύ GitHub Repository ✍️ Read the Full Post Online πŸ”— Hey Reader πŸ‘‹πŸ½ Welcome to this edition of the AWS Fundamentals newsletter! In this issue, we're focusing on observability with open-source tools on AWS. As most of you already know, we can use Amazon CloudWatch and X-Ray to monitor our application from every angle. But what if we want to hybrid setup where we run certain parts of our ecosystem outside of AWS?...

βŒ› Reading time: 9 minutes πŸŽ“ Main Learning: Polling or WebSockets: Choosing with Amazon API Gateway πŸ‘Ύ GitHub Repository ✍️ Read the Full Post Online πŸ”— Hey Reader πŸ‘‹πŸ½ What would you use for quick and regular data updates inside your web app? Or let's phrase it another way: how would you choose between Polling and WebSockets? πŸ’­ Understanding the nuances between these two communication methods is important, as they both come with their own advantages, gotchas, and side effects that are not very...