Join our community of over 8,800 readers delving into AWS. We highlight real-world best practices through easy-to-understand visualizations and one-pagers. Expect a fresh newsletter edition every two weeks.
Hope you're having an awesome week! This week's edition is about implementing Step Function Workflows. One major use-case that Step Functions enable is the Wait for Callback pattern.
Code Included π¨π½βπ»
This edition comes with a special bonus - for the first time, we're including aGitHub repository just for you! Clone the repo and deploy it in your AWS Account. β‘
Step Functions Orchestrate Workflows
What are even Step Functions?
Step Functions orchestrate workflows with different AWS Services. They are all about making different AWS Services work together in harmony.
You can make use of direct integrations like:
πΎ Scanning items in DynamoDB
β‘οΈ Invoking Lambda functions
π€ Sending WebSocket Messages
π Getting data from external APIs
You can have dedicated error & retry mechanisms for each step within a Step Function, making it a powerful orchestrator.
Today, we will build such a workflow together.
The Workflow Designer
AWS offers you a great tool to get started with Step Functions: the workflow designer.
It's a bit like playing with digital LEGO on your screen, where you can drag and drop pieces to build your workflow. The workflow designer is a low-code tool. Meaning that you don't need to code your workflow. Instead, you'll design it.
Here is an example from the workflow designer. It is a demo workflow by AWS:
Example Step Function in the Workflow Designer
However, in the end, you need to have some sort of code to include it in your Infrastructure as Code (IAC) provider. From the workflow designer, you can export a .json file in the Amazon State Language (ASL).
Or with an IAC framework like the CDK (which we use in the GitHub repository), you can build up your whole Step Function with its own syntax.
We prefer the latter one. It can be quite cumbersome to export a huge .json file into your code base every time you update one option in your state.
To get started with Step Functions we recommend using the workflow designer first. It shows you what step functions are capable of. But if you want to have it in production, translate it into IAC.
Callback Patterns Let You Wait for User Input
Today, we're diving into the magic of the callback pattern with Step Functions, a feature that's like putting your workflow on pause. Think of it as a "Hold, please! π" button for your tasks. The callback pattern lets a part of your workflow wait quietly until it's given the green light to go again.
β
Picture this: you grab some data from DynamoDB, send it over to a Lambda function, and thenβhere's where the magic happensβyou tell the next Lambda task to hang tight until it gets a special signal, a callback.
This setup is perfect when you need a human interaction, like a manual check or approval, before moving forward. Want to see how we put this together? Check out the Step Function Definition on GitHub and follow along with us.
Real-world Application: Content Moderation
Our use case for implementing the callback pattern is a Content Moderation System. Imagine hosting a blogging platform such as Hashnode or Medium. You don't want to enable everybody to publish anything. This is why you put some safeguards into place.
Content Moderation System
We've created a streamlined model for this concept. Here's the scenario: when someone submits an article that includes a specific no-go word (yep, "forbidden" is our trigger word), we don't just push it live. Instead, we pause and call in a human β a manual reviewer β to give it the once-over.
If the title includes "forbidden", everything pauses, waiting for a special nod β a task token.
Over in the admin dashboard, there's a new article flagged for review.
Upon approval, the Step Function springs back into action, moving the process forward.
It's a simple but effective way to keep tabs on content.
Diving Into the Architecture
The architecture consists of several parts:
Frontend - a Next.JS/React frontend that handles the submissions and admin dashboard
Lambda API - a REST API based on a Lambda Function URL that handles all business logic
Step Function - a Step Function to handle the moderation workflow
DynamoDB - all workflow executions that need to be reviewed including all URLs
Lambda Approval Receiver - another API to approve or reject the Step Function
This is a birds-eye view. Let's dive into it.
The overall architecture
Triggering Step Functions
β
Invoking a new Step Function
Once the user submits an article we start a new Step Function with the given title.
Content Check: First up, the Step Function scans the title. If the "forbidden" word is present, we'll pause the Step Function.
Requesting a Review: Once paused a Lambda function is executed that sets the workflow into its waiting state. The execution ARN and some other information are saved in DynamoDB.
The Wait for Callback Explained
This waiting game means the Step Function halts until a Task Token is received from the Step Function API. This token is essential; it's the green light our Lambda function needs to proceed.
In the workflow designer, the Lambda Function looks like this:
Lambda with Wait for Callback
We've configured the input of the task token (see taskToken.$: $$.Task.Token), and we set the configuration Wait for callback, which pauses the whole workflow.
In CDK the Lambda Task looks like this:
The Lambda task with all inputs in CDK
Once a Step Function has the word forbidden in its title the Step Function will halt and look like this:
β
Step Function in its waiting state
Our Lambda Function Ask For Manual Approval, standing by for a callback, also creates URLs for approving or rejecting the article in question.
These URLs, along with the details of the waiting Step Function, are stored in DynamoDB. The following item is stored in DynamoDB:
Item stored in DynamoDB
With this system in place, retrieving the list of pending reviews and their respective decision-making URLs becomes a breeze.
Decision Time: Approve or Reject?
β
Approval Workflow
When our Step Function waits, admins jump in. They head to a simple dashboard at the /admin route in our frontend. Here, they see articles waiting for a thumbs up or down, each with its link.
Clicking approve or reject sends their choice straight to a special function. This function tells the Step Function what to do next:
Send Task Success to the Step Function
Once this API call is sent the Step Function continues and stores a boolean value for decision in the output of the Lambda function.
β
A Step Function after approving the decision
β
Recap & Resources
Step Functions are a great service in AWS to orchestrate and handle complex workflows. Using the Callback Pattern is one of the best use cases for Step Functions. While the input and output transformations of the data can be quite complex, it is a very good way to build observable workflows.
Key Takeaways:
Step Functions are your workflow orchestrators, ensuring each step of your process harmonizes with the next.
The Workflow Designer is a good way to get started, but you need IAC in the end.
Callback Patterns offer the flexibility to pause your workflows, awaiting human input or external triggers to proceed.
Through our Content Moderation System example, you've seen how these concepts apply in real-world scenarios, ensuring quality and control.
Go ahead, deploy the GitHub Repository to your AWS account, and change a few things. For example:
store the article in a DynamoDB with a direct integration
check the input for more words than just forbidden
send the user a notification based on WebSockets, Emails, or SMS
Do you like these hands-on editions with included & deployable code?
They cost much more time but we think the value readers get out of that is great. Instead of covering one service in all its details like in the first year of this newsletter, we want to do more deep-dives with code included.
Let us know what you think!
See you soon βπ½ Sandro & Tobi β
β
Tobi & Sandro
our goal is to simplify AWS & Cloud Learning for everybody. You don't need expensive certifications to build on AWS!
Join our community of over 8,800 readers delving into AWS. We highlight real-world best practices through easy-to-understand visualizations and one-pagers. Expect a fresh newsletter edition every two weeks.
β Reading time: 10 minutes π Main Learning: Building a Serverless Platform With SST, Lambda & Next.js βοΈ Read the Full Post Online π Hey Reader ππ½ In this post, we want to guide you through our complete setup for our custom video platform. Our CloudWatch Book's Video Section This starts from the purchase to actually accessing our custom build video-access platform. Overview about our CloudWatch Book Landing Page & Video Platform Architecture We'll explain why we decided against a third-party...
β Reading time: 11 minutes π Main Learning: Step Functions - Express vs. Standard πΎ GitHub Code βοΈ Blog Post Hey Reader while Sandro is learning something new at the AWS Community Day in Munich today, we'll explore Express and Standard Step Functions, the two types of workflows offered by AWS Step Functions. Weβll break down their differences, when to use each, and the benefits of both. Example Application: running both workflow types to see their performance differences If you want to try...
Hey Reader First things first: we apologize for not providing updates on The CloudWatch Book for a while! π’ Sometimes, things don't go as planned and unexpected obstacles arise. But now, we're back in action, creating videos and putting the final touches on the book's content! π₯ Don't just take our word for it! As an early subscriber, here's a free video from one of our favorite chapters: Anomaly Detection π In this deep-dive, you'll learn how to detect unusual patterns in metrics without...