π This is not properly displayed? Here is the online version! π‘
Hi Reader ππ½
new week, new AWS Newsletter π
We just checked our recent newsletters and saw that we've missed two services in the event-driven category, SQS, and SNS. Since both of these services are essential in working with AWS we're adding them now!
This week we start with SQS. The Simple Queue Service.
Can you guess in which year SQS was launched? π€
Hint: AWS launched its first service back in 2004. We'll tell you at the end of this issue π
This newsletter issue gives you an introduction to the following SQS topics:
Let's give you a high-level overview of SQS with our infographic:
β
Amazon Simple Queue Service (SQS) is a fully-managed message queue service by AWS. It allows you to build scalable and high-performant event-driven systems.
SQS decouples your architecture by introducing a message queue. Your message remains in the queue until a consumer picks it up and removes it.
There is almost no cloud application without SQS out there.
One stunning statistic: at Amazon Prime Days 2021, SQS peaked at 47.7 million requests per second.
One common misconception about SQS is that it is a push-based system. The misconception often results from the tight integration with Lambda and SQS.
Let's see one example:
A user sends a message to API Gateway. API Gateway responds after a few milliseconds after it enqueued the message to SQS. The message is in SQS now. After some time Lambda takes the message and works on it.
It all looks like the message is actually pushed into Lambda. But it isn't! In the background there is actually a third component involved, the Event source mapping.
This mapping is constantly polling SQS for new messages. Once a new message is there it will batch the requests and invoke a lambda function.
This is crucial to understand if you want to work with SQS. All features like visibility timeouts, short & long polling, and delays build up on the feature of polling and not pushing messages.
β
There are different types of queues in SQS. Standard, and FIFO queues.
The main difference is in the ordering of the messages. Standard queues donβt order your messages. FIFO queues do order your messages. There are several implications with that.
Standard Queues don't preserve the order. That means your consumer can receive the messages out of order.
FIFO queues on the other hand preserve the ordering within one message group. FIFO stands for First in, First out. That means the first message that comes in is the first that will be processed by a consumer.
β
SQS has many configuration options such as:
One configuration option we want to highlight in this issue is the Visibility Timeout.
The visibility timeout defines how long your message is invisible. Once a consumer picks up a message the state of the message changes from available
to in-flight
. From that point on the visibility timeout starts.
By default, your visibility timeout is set to 30 seconds. You can set it between 0 seconds and 12 hours. Other consumers cannot pick up the same message while it is in the state in-flight
.
Once the visibility timeout is over and the message was not deleted from the queue, it changes its state to available
again.
Setting the correct visibility timeout is not always easy. It is important for your application to work correctly.
There are two main scenarios to consider:
Letβs see them in more detail. You can see all scenarios in the image above.
Visibility Timeout Is Too Short
Imagine your consumer needs 1 minute to work on your message. After that, it removes the message from the queue. If your visibility timeout is only set to 30 seconds your message will be available too early. After 30 seconds your message changes the state from in-flight
to available
. Now it is visible to other consumers.
A second consumer will pick up your message and work with it. By that, your message execution is now duplicated. And even worse: after finishing the consumer is not even able to delete the message anymore, because the MessageHandle has expired. It retries until the death of the message by redrive policy.
Visibility Timeout Too Long
Having a visibility timeout set too long can result in unnecessary waiting times. For example, youβve set your visibility timeout to 10 minutes. Your consumer fails after 1 minute.
The message will still be invisible to other consumers for 9 more minutes. Your queue wonβt retry the message for the next 9 minutes. After 9 minutes the state changes from in-flight
to available
. Now a consumer can pick up the message again.
Set the Visibility Timeout rather too high than too low
Setting the visibility timeout too low is in most cases worse than setting your timeout too high. But it all depends on your business context of how fast messages should be retried.
How should you set it?
That's all about SQS for this issue ππ½ Now let's answer our question from above:
I need to be honest this was kind of a trick question. SQS was the first service launched by AWS. The launch date was back in 2004!
β
SQS is used by almost any application you will use that runs on AWS. It is one of the core services to master.
Next in our event-driven series will be SNS and EventBridge. Stay tuned π
β
This week's star is Philip Riecks. Philip is the guru of testing Java Spring applications. He has an amazing blog, newsletter, and also an AWS Book called Stratospheric.
We've recommended the book Stratospheric in the past and we'll do it again here. If you're interested in building real-world Java applications, check it out.
If you're interested in more, have a look at our
ββAWS Fundamentals blog π
β
Join our community of over 9,300 readers delving into AWS. We highlight real-world best practices through easy-to-understand visualizations and one-pagers. Expect a fresh newsletter edition every two weeks.
Hey Reader ππ½ We've been talking a lot about how great SST's switch to Pulumi was, and many of you have asked us how to use plain Pulumi directly. So today, we're sharing our quick guide to Pulumi - a tool we're really excited about since it lets us build infrastructure with languages we already know and love. No more learning weird syntax - just TypeScript, Python, or whatever we're comfortable with! We spent the last few days playing with it, and here's what we've learned... AWS Community...
Newsletter Header AWS FOR THE REAL WORLD β±οΈ Reading time: 8 minutes π Main Learning: Migrating from Edgio to CloudFront βοΈ Blog Post π» GitHub Repository Hey Reader ππ½ this newsletter is about π₯ AI π€ We haven't talked too much about AI, Bedrock, MCPs, and agents yet - so we want to change that. Please let us know if this it interests you to build AI on AWS, or if you are much more interested on hands-on fundamentals services. Should we focus on AI Services? Yes, I want to learn to build...
Newsletter Header AWS FOR THE REAL WORLD β±οΈ Reading time: 8 minutes π Main Learning: Migrating from Edgio to CloudFront βοΈ Blog Post π» GitHub Repository Hey Reader ππ½ This newsletter edition is all about saving Lambda costs. I (Sandro is writing this one) was recently involved in saving Lambda costs for a client. So, I thought writing down my thought process was a good idea. Have fun with it! If you need help saving AWS costs or improving your infrastructure, just reply to this email! Now...