Join our community of over 8,800 readers delving into AWS. We highlight real-world best practices through easy-to-understand visualizations and one-pagers. Expect a fresh newsletter edition every two weeks.
Hope you're having an awesome week! This week's edition is about implementing Step Function Workflows. One major use-case that Step Functions enable is the Wait for Callback pattern.
Code Included π¨π½βπ»
This edition comes with a special bonus - for the first time, we're including aGitHub repository just for you! Clone the repo and deploy it in your AWS Account. β‘
Step Functions Orchestrate Workflows
What are even Step Functions?
Step Functions orchestrate workflows with different AWS Services. They are all about making different AWS Services work together in harmony.
You can make use of direct integrations like:
πΎ Scanning items in DynamoDB
β‘οΈ Invoking Lambda functions
π€ Sending WebSocket Messages
π Getting data from external APIs
You can have dedicated error & retry mechanisms for each step within a Step Function, making it a powerful orchestrator.
Today, we will build such a workflow together.
The Workflow Designer
AWS offers you a great tool to get started with Step Functions: the workflow designer.
It's a bit like playing with digital LEGO on your screen, where you can drag and drop pieces to build your workflow. The workflow designer is a low-code tool. Meaning that you don't need to code your workflow. Instead, you'll design it.
Here is an example from the workflow designer. It is a demo workflow by AWS:
However, in the end, you need to have some sort of code to include it in your Infrastructure as Code (IAC) provider. From the workflow designer, you can export a .json file in the Amazon State Language (ASL).
Or with an IAC framework like the CDK (which we use in the GitHub repository), you can build up your whole Step Function with its own syntax.
We prefer the latter one. It can be quite cumbersome to export a huge .json file into your code base every time you update one option in your state.
To get started with Step Functions we recommend using the workflow designer first. It shows you what step functions are capable of. But if you want to have it in production, translate it into IAC.
Callback Patterns Let You Wait for User Input
Today, we're diving into the magic of the callback pattern with Step Functions, a feature that's like putting your workflow on pause. Think of it as a "Hold, please! π" button for your tasks. The callback pattern lets a part of your workflow wait quietly until it's given the green light to go again.
Picture this: you grab some data from DynamoDB, send it over to a Lambda function, and thenβhere's where the magic happensβyou tell the next Lambda task to hang tight until it gets a special signal, a callback.
This setup is perfect when you need a human interaction, like a manual check or approval, before moving forward. Want to see how we put this together? Check out the Step Function Definition on GitHub and follow along with us.
Real-world Application: Content Moderation
Our use case for implementing the callback pattern is a Content Moderation System. Imagine hosting a blogging platform such as Hashnode or Medium. You don't want to enable everybody to publish anything. This is why you put some safeguards into place.
We've created a streamlined model for this concept. Here's the scenario: when someone submits an article that includes a specific no-go word (yep, "forbidden" is our trigger word), we don't just push it live. Instead, we pause and call in a human β a manual reviewer β to give it the once-over.
If the title includes "forbidden", everything pauses, waiting for a special nod β a task token.
Over in the admin dashboard, there's a new article flagged for review.
Upon approval, the Step Function springs back into action, moving the process forward.
It's a simple but effective way to keep tabs on content.
Diving Into the Architecture
The architecture consists of several parts:
Frontend - a Next.JS/React frontend that handles the submissions and admin dashboard
Lambda API - a REST API based on a Lambda Function URL that handles all business logic
Step Function - a Step Function to handle the moderation workflow
DynamoDB - all workflow executions that need to be reviewed including all URLs
Lambda Approval Receiver - another API to approve or reject the Step Function
This is a birds-eye view. Let's dive into it.
Triggering Step Functions
β
Once the user submits an article we start a new Step Function with the given title.
Content Check: First up, the Step Function scans the title. If the "forbidden" word is present, we'll pause the Step Function.
Requesting a Review: Once paused a Lambda function is executed that sets the workflow into its waiting state. The execution ARN and some other information are saved in DynamoDB.
The Wait for Callback Explained
This waiting game means the Step Function halts until a Task Token is received from the Step Function API. This token is essential; it's the green light our Lambda function needs to proceed.
In the workflow designer, the Lambda Function looks like this:
We've configured the input of the task token (see taskToken.$: $$.Task.Token), and we set the configuration Wait for callback, which pauses the whole workflow.
In CDK the Lambda Task looks like this:
Once a Step Function has the word forbidden in its title the Step Function will halt and look like this:
β
Our Lambda Function Ask For Manual Approval, standing by for a callback, also creates URLs for approving or rejecting the article in question.
These URLs, along with the details of the waiting Step Function, are stored in DynamoDB. The following item is stored in DynamoDB:
With this system in place, retrieving the list of pending reviews and their respective decision-making URLs becomes a breeze.
Decision Time: Approve or Reject?
β
When our Step Function waits, admins jump in. They head to a simple dashboard at the /admin route in our frontend. Here, they see articles waiting for a thumbs up or down, each with its link.
Clicking approve or reject sends their choice straight to a special function. This function tells the Step Function what to do next:
Once this API call is sent the Step Function continues and stores a boolean value for decision in the output of the Lambda function.
β
β
Recap & Resources
Step Functions are a great service in AWS to orchestrate and handle complex workflows. Using the Callback Pattern is one of the best use cases for Step Functions. While the input and output transformations of the data can be quite complex, it is a very good way to build observable workflows.
Key Takeaways:
Step Functions are your workflow orchestrators, ensuring each step of your process harmonizes with the next.
The Workflow Designer is a good way to get started, but you need IAC in the end.
Callback Patterns offer the flexibility to pause your workflows, awaiting human input or external triggers to proceed.
Through our Content Moderation System example, you've seen how these concepts apply in real-world scenarios, ensuring quality and control.
Go ahead, deploy the GitHub Repository to your AWS account, and change a few things. For example:
store the article in a DynamoDB with a direct integration
check the input for more words than just forbidden
send the user a notification based on WebSockets, Emails, or SMS
Do you like these hands-on editions with included & deployable code?
They cost much more time but we think the value readers get out of that is great. Instead of covering one service in all its details like in the first year of this newsletter, we want to do more deep-dives with code included.
Let us know what you think!
See you soon βπ½ Sandro & Tobi β
β
Tobi & Sandro
our goal is to simplify AWS & Cloud Learning for everybody. You don't need expensive certifications to build on AWS!
Join our community of over 8,800 readers delving into AWS. We highlight real-world best practices through easy-to-understand visualizations and one-pagers. Expect a fresh newsletter edition every two weeks.
β Reading time: 14 minutes π Main Learning: Feature Flags with AWS AppConfig πΎ GitHub Repository βοΈ Read the Full Post Online π Hey Reader ππ½ There's no other field where it's so common to have "a small side-project" like in the software industry. Even though it's possible to build things as quickly as ever before due to cloud providers, tools, platforms, and AI, many indie founders (and also large enterprises) tend to fall into the same trap: they tend to build features that users do not...
β Reading time: 17 minutes π Main Learning: Observability at Scale with Open-Source πΎ GitHub Repository βοΈ Read the Full Post Online π Hey Reader ππ½ Welcome to this edition of the AWS Fundamentals newsletter! In this issue, we're focusing on observability with open-source tools on AWS. As most of you already know, we can use Amazon CloudWatch and X-Ray to monitor our application from every angle. But what if we want to hybrid setup where we run certain parts of our ecosystem outside of AWS?...
β Reading time: 9 minutes π Main Learning: Polling or WebSockets: Choosing with Amazon API Gateway πΎ GitHub Repository βοΈ Read the Full Post Online π Hey Reader ππ½ What would you use for quick and regular data updates inside your web app? Or let's phrase it another way: how would you choose between Polling and WebSockets? π Understanding the nuances between these two communication methods is important, as they both come with their own advantages, gotchas, and side effects that are not very...