πŸ”§ Amazon SQS: The Fully-Managed Message Queue Service You Need to Know About


πŸ‘€ This is not properly displayed? Here is the online version! πŸ’‘

Hi Reader πŸ‘‹πŸ½

new week, new AWS Newsletter πŸ’Œ

We just checked our recent newsletters and saw that we've missed two services in the event-driven category, SQS, and SNS. Since both of these services are essential in working with AWS we're adding them now!

This week we start with SQS. The Simple Queue Service.

Can you guess in which year SQS was launched? πŸ€”
Hint: AWS launched its first service back in 2004. We'll tell you at the end of this issue πŸ˜‰

This newsletter issue gives you an introduction to the following SQS topics:

  1. Introduction to SQS πŸ§‘πŸ½β€πŸŽ“
  2. Poll Based Vs. Push Based ↕️
  3. Standard vs. FIFO Queues 🚏
  4. Visibility Timeouts πŸ«₯

Infographic

Let's give you a high-level overview of SQS with our infographic:

​

Introduction to SQS πŸ§‘πŸΎβ€πŸŽ“

Amazon Simple Queue Service (SQS) is a fully-managed message queue service by AWS. It allows you to build scalable and high-performant event-driven systems.

SQS decouples your architecture by introducing a message queue. Your message remains in the queue until a consumer picks it up and removes it.

There is almost no cloud application without SQS out there.

One stunning statistic: at Amazon Prime Days 2021, SQS peaked at 47.7 million requests per second.

Poll Based Vs. Push Based ↕️

One common misconception about SQS is that it is a push-based system. The misconception often results from the tight integration with Lambda and SQS.

Let's see one example:

A user sends a message to API Gateway. API Gateway responds after a few milliseconds after it enqueued the message to SQS. The message is in SQS now. After some time Lambda takes the message and works on it.

It all looks like the message is actually pushed into Lambda. But it isn't! In the background there is actually a third component involved, the Event source mapping.

This mapping is constantly polling SQS for new messages. Once a new message is there it will batch the requests and invoke a lambda function.

This is crucial to understand if you want to work with SQS. All features like visibility timeouts, short & long polling, and delays build up on the feature of polling and not pushing messages.

​

Standard vs. FIFO Queues 🚏

There are different types of queues in SQS. Standard, and FIFO queues.

The main difference is in the ordering of the messages. Standard queues don’t order your messages. FIFO queues do order your messages. There are several implications with that.

Standard Queues don't preserve the order. That means your consumer can receive the messages out of order.

FIFO queues on the other hand preserve the ordering within one message group. FIFO stands for First in, First out. That means the first message that comes in is the first that will be processed by a consumer.

​

Hiding Messages From Other Consumers by Defining the Visibility Timeout

SQS has many configuration options such as:

  • Long polling vs. short polling
  • Retention period
  • Delays
  • ... and many more

One configuration option we want to highlight in this issue is the Visibility Timeout.

The visibility timeout defines how long your message is invisible. Once a consumer picks up a message the state of the message changes from available to in-flight. From that point on the visibility timeout starts.

By default, your visibility timeout is set to 30 seconds. You can set it between 0 seconds and 12 hours. Other consumers cannot pick up the same message while it is in the state in-flight.

Once the visibility timeout is over and the message was not deleted from the queue, it changes its state to available again.

Setting the correct visibility timeout is not always easy. It is important for your application to work correctly.

There are two main scenarios to consider:

  1. Visibility timeout too long - your queue won’t retry the message in a timely manner.
  2. Visibility timeout too short - multiple consumers pick up the message

Let’s see them in more detail. You can see all scenarios in the image above.

Visibility Timeout Is Too Short

Imagine your consumer needs 1 minute to work on your message. After that, it removes the message from the queue. If your visibility timeout is only set to 30 seconds your message will be available too early. After 30 seconds your message changes the state from in-flight to available. Now it is visible to other consumers.

A second consumer will pick up your message and work with it. By that, your message execution is now duplicated. And even worse: after finishing the consumer is not even able to delete the message anymore, because the MessageHandle has expired. It retries until the death of the message by redrive policy.

Visibility Timeout Too Long

Having a visibility timeout set too long can result in unnecessary waiting times. For example, you’ve set your visibility timeout to 10 minutes. Your consumer fails after 1 minute.

The message will still be invisible to other consumers for 9 more minutes. Your queue won’t retry the message for the next 9 minutes. After 9 minutes the state changes from in-flight to available. Now a consumer can pick up the message again.

Set the Visibility Timeout rather too high than too low

Setting the visibility timeout too low is in most cases worse than setting your timeout too high. But it all depends on your business context of how fast messages should be retried.

How should you set it?

  1. Start with some initial settings (like the default).
  2. See how long your Lambda takes to finish the task
  3. Set the timeout to that time + some extra time for longer executions
  4. Understand how crucial it is that your retries are happening fast. Versus how bad is it if you retry only happens after a couple of minutes.

That's all about SQS for this issue πŸ™ŒπŸ½ Now let's answer our question from above:

When Was SQS Launched?

I need to be honest this was kind of a trick question. SQS was the first service launched by AWS. The launch date was back in 2004!

​

That's it ✌🏽

SQS is used by almost any application you will use that runs on AWS. It is one of the core services to master.

Next in our event-driven series will be SNS and EventBridge. Stay tuned 😊

​

...Beyond AWS ⛅️ πŸ‘€

This week's star is Philip Riecks. Philip is the guru of testing Java Spring applications. He has an amazing blog, newsletter, and also an AWS Book called Stratospheric.

We've recommended the book Stratospheric in the past and we'll do it again here. If you're interested in building real-world Java applications, check it out.


If you're interested in more, have a look at our
​
​AWS Fundamentals blog πŸ“š

​

AWS for the Real World

Join our community of over 9,300 readers delving into AWS. We highlight real-world best practices through easy-to-understand visualizations and one-pagers. Expect a fresh newsletter edition every two weeks.

Read more from AWS for the Real World

Hey Reader πŸ‘‹πŸ½ We've been talking a lot about how great SST's switch to Pulumi was, and many of you have asked us how to use plain Pulumi directly. So today, we're sharing our quick guide to Pulumi - a tool we're really excited about since it lets us build infrastructure with languages we already know and love. No more learning weird syntax - just TypeScript, Python, or whatever we're comfortable with! We spent the last few days playing with it, and here's what we've learned... AWS Community...

Newsletter Header AWS FOR THE REAL WORLD ⏱️ Reading time: 8 minutes πŸŽ“ Main Learning: Migrating from Edgio to CloudFront ✍️ Blog Post πŸ’» GitHub Repository Hey Reader πŸ‘‹πŸ½ this newsletter is about πŸ₯ AI πŸ€– We haven't talked too much about AI, Bedrock, MCPs, and agents yet - so we want to change that. Please let us know if this it interests you to build AI on AWS, or if you are much more interested on hands-on fundamentals services. Should we focus on AI Services? Yes, I want to learn to build...

Newsletter Header AWS FOR THE REAL WORLD ⏱️ Reading time: 8 minutes πŸŽ“ Main Learning: Migrating from Edgio to CloudFront ✍️ Blog Post πŸ’» GitHub Repository Hey Reader πŸ‘‹πŸ½ This newsletter edition is all about saving Lambda costs. I (Sandro is writing this one) was recently involved in saving Lambda costs for a client. So, I thought writing down my thought process was a good idea. Have fun with it! If you need help saving AWS costs or improving your infrastructure, just reply to this email! Now...