AWS for the Real World

πŸ”§ Amazon SQS: The Fully-Managed Message Queue Service You Need to Know About

Published 11 months agoΒ β€’Β 4 min read

πŸ‘€ This is not properly displayed? Here is the online version! πŸ’‘

Hi Reader πŸ‘‹πŸ½

new week, new AWS Newsletter πŸ’Œ

We just checked our recent newsletters and saw that we've missed two services in the event-driven category, SQS, and SNS. Since both of these services are essential in working with AWS we're adding them now!

This week we start with SQS. The Simple Queue Service.

Can you guess in which year SQS was launched? πŸ€”
Hint: AWS launched its first service back in 2004. We'll tell you at the end of this issue πŸ˜‰

This newsletter issue gives you an introduction to the following SQS topics:

  1. Introduction to SQS πŸ§‘πŸ½β€πŸŽ“
  2. Poll Based Vs. Push Based ↕️
  3. Standard vs. FIFO Queues 🚏
  4. Visibility Timeouts πŸ«₯


Let's give you a high-level overview of SQS with our infographic:


Introduction to SQS πŸ§‘πŸΎβ€πŸŽ“

Amazon Simple Queue Service (SQS) is a fully-managed message queue service by AWS. It allows you to build scalable and high-performant event-driven systems.

SQS decouples your architecture by introducing a message queue. Your message remains in the queue until a consumer picks it up and removes it.

There is almost no cloud application without SQS out there.

One stunning statistic: at Amazon Prime Days 2021, SQS peaked at 47.7 million requests per second.

Poll Based Vs. Push Based ↕️

One common misconception about SQS is that it is a push-based system. The misconception often results from the tight integration with Lambda and SQS.

Let's see one example:

A user sends a message to API Gateway. API Gateway responds after a few milliseconds after it enqueued the message to SQS. The message is in SQS now. After some time Lambda takes the message and works on it.

It all looks like the message is actually pushed into Lambda. But it isn't! In the background there is actually a third component involved, the Event source mapping.

This mapping is constantly polling SQS for new messages. Once a new message is there it will batch the requests and invoke a lambda function.

This is crucial to understand if you want to work with SQS. All features like visibility timeouts, short & long polling, and delays build up on the feature of polling and not pushing messages.


Standard vs. FIFO Queues 🚏

There are different types of queues in SQS. Standard, and FIFO queues.

The main difference is in the ordering of the messages. Standard queues don’t order your messages. FIFO queues do order your messages. There are several implications with that.

Standard Queues don't preserve the order. That means your consumer can receive the messages out of order.

FIFO queues on the other hand preserve the ordering within one message group. FIFO stands for First in, First out. That means the first message that comes in is the first that will be processed by a consumer.


Hiding Messages From Other Consumers by Defining the Visibility Timeout

SQS has many configuration options such as:

  • Long polling vs. short polling
  • Retention period
  • Delays
  • ... and many more

One configuration option we want to highlight in this issue is the Visibility Timeout.

The visibility timeout defines how long your message is invisible. Once a consumer picks up a message the state of the message changes from available to in-flight. From that point on the visibility timeout starts.

By default, your visibility timeout is set to 30 seconds. You can set it between 0 seconds and 12 hours. Other consumers cannot pick up the same message while it is in the state in-flight.

Once the visibility timeout is over and the message was not deleted from the queue, it changes its state to available again.

Setting the correct visibility timeout is not always easy. It is important for your application to work correctly.

There are two main scenarios to consider:

  1. Visibility timeout too long - your queue won’t retry the message in a timely manner.
  2. Visibility timeout too short - multiple consumers pick up the message

Let’s see them in more detail. You can see all scenarios in the image above.

Visibility Timeout Is Too Short

Imagine your consumer needs 1 minute to work on your message. After that, it removes the message from the queue. If your visibility timeout is only set to 30 seconds your message will be available too early. After 30 seconds your message changes the state from in-flight to available. Now it is visible to other consumers.

A second consumer will pick up your message and work with it. By that, your message execution is now duplicated. And even worse: after finishing the consumer is not even able to delete the message anymore, because the MessageHandle has expired. It retries until the death of the message by redrive policy.

Visibility Timeout Too Long

Having a visibility timeout set too long can result in unnecessary waiting times. For example, you’ve set your visibility timeout to 10 minutes. Your consumer fails after 1 minute.

The message will still be invisible to other consumers for 9 more minutes. Your queue won’t retry the message for the next 9 minutes. After 9 minutes the state changes from in-flight to available. Now a consumer can pick up the message again.

Set the Visibility Timeout rather too high than too low

Setting the visibility timeout too low is in most cases worse than setting your timeout too high. But it all depends on your business context of how fast messages should be retried.

How should you set it?

  1. Start with some initial settings (like the default).
  2. See how long your Lambda takes to finish the task
  3. Set the timeout to that time + some extra time for longer executions
  4. Understand how crucial it is that your retries are happening fast. Versus how bad is it if you retry only happens after a couple of minutes.

That's all about SQS for this issue πŸ™ŒπŸ½ Now let's answer our question from above:

When Was SQS Launched?

I need to be honest this was kind of a trick question. SQS was the first service launched by AWS. The launch date was back in 2004!


That's it ✌🏽

SQS is used by almost any application you will use that runs on AWS. It is one of the core services to master.

Next in our event-driven series will be SNS and EventBridge. Stay tuned 😊


...Beyond AWS ⛅️ πŸ‘€

This week's star is Philip Riecks. Philip is the guru of testing Java Spring applications. He has an amazing blog, newsletter, and also an AWS Book called Stratospheric.

We've recommended the book Stratospheric in the past and we'll do it again here. If you're interested in building real-world Java applications, check it out.

If you're interested in more, have a look at our
​AWS Fundamentals blog πŸ“š


AWS for the Real World

by Tobi & Sandro

Join our community of over 8,800 readers delving into AWS. We highlight real-world best practices through easy-to-understand visualizations and one-pagers. Expect a fresh newsletter edition every two weeks.

Read more from AWS for the Real World

βŒ› Reading time: 11.5 minutes πŸŽ“ Main Learning: Host Web Applications on AWS with the CloudFront Hosting Toolkit πŸ‘¨πŸ½πŸ’» GitHub Code πŸ“ Blog Post Hey Reader πŸ‘‹πŸ½ Happy New Week! I (Sandro) will attend the Serverless Days in Milano next week where Jeremy Daley will hold the keynote. I look forward to meeting many of you and the overall AWS community. This week's newsletter is all about hosting your frontend on AWS. AWS launched a new way of deploying your frontend to it: The CloudFront Hosting Toolkit....

10 days agoΒ β€’Β 6 min read

βŒ› Reading time: 7 minutes πŸŽ“ Main Learning: Visual Regression Testing with AWS CloudWatch Synthetics πŸ‘¨πŸ½πŸ’» GitHub Code πŸ“ Blog Post Hey Reader Keeping your website running smoothly is important.It's also important to keep an eye on its visual appearance and that it doesn't change due to an unwanted change. AWS CloudWatch Synthetic Canaries is a tool that helps you check on your websites (and APIs!) to make sure it’s working well and look as expected. In this newsletter, we’re going to show you...

about 1 month agoΒ β€’Β 4 min read

Hi Reader and happy May the 4th ⭐. We want to start the spring by giving you an amazing discount on our book and other resources. For that, we have partnered with four different AWS Creators around the globe. You can use the code AWSFUND30 to get 30% off. The resources are: πŸ“™ AWS Fundamentals - Like this newsletter, it covers the basics of essential AWS Services for real-world applications. πŸ“˜ The DynamoDB Book - Level up your DynamoDB modeling and finally understand Single-Table Design. πŸ—οΈ...

about 2 months agoΒ β€’Β 1 min read
Share this post