10. Serverless Applications

Explore

Application Integration Services

Amazon SQS

Amazon Simple Queue Service (Amazon SQS) offers a secure, durable, and available hosted queue that lets you integrate and decouple distributed software systems and components. Amazon SQS offers common constructs such as

dead-letter queues⁠

and

cost allocation tags⁠

. It provides a generic web services API that you can access using any programming language that the AWS SDK supports.

⁠

This is known as decoupling / loose coupling and helps to enable elasticity for your application.

Amazon SQS is pull-based, not push-based (like Amazon SNS).

Benefits of using Amazon SQS

Security –

You control⁠

who can send messages to and receive messages from an Amazon SQS queue. You can choose to transmit sensitive data by protecting the contents of messages in queues by using default Amazon SQS managed server-side encryption (SSE), or by using custom

SSE⁠

keys managed in AWS Key Management Service (AWS KMS).

Durability – For the safety of your messages, Amazon SQS stores them on multiple servers. Standard queues support

at-least-once message delivery⁠

, and FIFO queues support

exactly-once message processing⁠

and

high-throughput⁠

mode.

Availability – Amazon SQS uses

redundant infrastructure⁠

to provide highly-concurrent access to messages and high availability for producing and consuming messages.

Scalability – Amazon SQS can process each

buffered request⁠

independently, scaling transparently to handle any load increases or spikes without any provisioning instructions.

Reliability – Amazon SQS locks your messages during processing, so that multiple producers can send and multiple consumers can receive messages at the same time.

Customization – Your queues don't have to be exactly alike—for example, you can

set a default delay on a queue⁠

. You can store the contents of messages larger than 256 KB

using Amazon Simple Storage Service (Amazon S3)⁠

or Amazon DynamoDB, with Amazon SQS holding a pointer to the Amazon S3 object, or you can split a large message into smaller messages.

⁠

Amazon SQS queue types

Amazon SQS supports two types of queues – standard queues and FIFO queues. Use the information from the following table to choose the right queue for your situation. To learn more about Amazon SQS queues, see

Getting started with Amazon SQS standard queues⁠

and

Getting started with FIFO queues in Amazon SQS⁠

queue types

queue types

Standard queues

FIFO queues

Unlimited Throughput – Standard queues support a nearly unlimited number of API calls per second, per API action (SendMessage, ReceiveMessage, or DeleteMessage).

At-Least-Once Delivery – A message is delivered at least once, but occasionally more than one copy of a message is delivered.

Best-Effort Ordering – Occasionally, messages are delivered in an order different from which they were sent.

High Throughput – If you use

batching

⁠

, FIFO queues support up to 3,000 messages per second, per API method (SendMessageBatch, ReceiveMessage, or DeleteMessageBatch). The 3,000 messages per second represent 300 API calls, each with a batch of 10 messages. To request a quota increase,

submit a support request

⁠

. Without batching, FIFO queues support up to 300 API calls per second, per API method (SendMessage, ReceiveMessage, or DeleteMessage).

Exactly-Once Processing – A message is delivered once and remains available until a consumer processes and deletes it. Duplicates aren't introduced into the queue.

First-In-First-Out Delivery – The order in which messages are sent and received is strictly preserved.

⁠

⁠

⁠

Send data between applications when the throughput is important, for example:

Decouple live user requests from intensive background work: let users upload media while resizing or encoding it.

Allocate tasks to multiple worker nodes: process a high number of credit card validation requests.

Batch messages for future processing: schedule multiple entries to be added to a database.

Send data between applications when the order of events is important, for example:

Make sure that user-entered commands are run in the right order.

Display the correct product price by sending price modifications in the right order.

Prevent a student from enrolling in a course before registering for an account.

There are no rows in this table

⁠

Using dead-letter queues in Amazon SQS

Amazon SQS supports dead-letter queues, which source queues can target for messages that are not processed successfully. Dead-letter queues are useful for debugging your application because you can isolate unconsumed messages to determine why processing did not succeed. Once messages are in a dead-letter queue, you can:

Examine logs for exceptions that might have caused messages to be moved to a dead-letter queue.

Analyze the contents of messages moved to the dead-letter queue to diagnose application issues.

Determine whether you have given your consumer sufficient time to process messages.

Move messages out of the dead-letter queue using

dead-letter queue redrive⁠

You must first create a new queue before configuring it as a dead-letter queue. For information about configuring a dead-letter queue using the Amazon SQS console, see

Learn how to configure a dead-letter queue using the Amazon SQS console⁠

. For help with dead-letter queues, such as how to configure an alarm for any messages moved to a dead-letter queue, see

Troubleshooting and best practices for dead-letter queues in Amazon SQS⁠

Using policies for dead-letter queues

Use a redrive policy to specify the maxReceiveCount. The maxReceiveCount is the number of times a consumer can receive a message from a source queue before it is moved to a dead-letter queue. For example, if the maxReceiveCount is set to a low value such as 1, one failure to receive a message would cause the message to move to the dead-letter queue. To ensure that your system is resilient against errors, set the maxReceiveCount high enough to allow for sufficient retries.

The redrive allow policy specifies which source queues can access the dead-letter queue. You can choose whether to allow all source queues, allow specific source queues, or deny all source queues use of the dead-letter queue. The default allows all source queues to use the dead-letter queue. If you choose to allow specific queues using the byQueue option, you can specify up to 10 source queues using the source queue Amazon Resource Name (ARN). If you specify denyAll, the queue cannot be used as a dead-letter queue.

⁠

Amazon SQS short and long polling

Amazon SQS offers short and long polling options for receiving messages from a queue. Consider your application's requirements for responsiveness and cost efficiency when choosing between these two polling options:

Short polling (default) – The

ReceiveMessage⁠

request queries a subset of servers (based on a weighted random distribution) to find available messages and sends an immediate response, even if no messages are found.

Long polling –

ReceiveMessage⁠

queries all servers for messages, sending a response once at least one message is available, up to the specified maximum. An empty response is sent only if the polling wait time expires. This option can reduce the number of empty responses and potentially lower costs. When the wait time for the

ReceiveMessage⁠

API action is greater than 0, long polling is in effect. The maximum long polling wait time is 20 seconds. Long polling helps reduce the cost of using Amazon SQS by eliminating the number of empty responses (when there are no messages available for a ReceiveMessage request) and false empty responses (when messages are available but aren't included in a response).

Differences between long and short polling

Short polling occurs when the

WaitTimeSeconds⁠

parameter of a

ReceiveMessage⁠

request is set to 0 in one of two ways:

The ReceiveMessage call sets WaitTimeSeconds to 0.

The ReceiveMessage call doesn’t set WaitTimeSeconds, but the queue attribute

ReceiveMessageWaitTimeSeconds⁠

is set to 0.

Using Amazon SQS and IAM policies

There are two ways to give your users permissions to your Amazon SQS resources: using the Amazon SQS policy system and using the IAM policy system. You can use one or the other, or both. For the most part, you can achieve the same result with either one.

For example, the following diagram shows an IAM policy and an Amazon SQS policy equivalent to it. The IAM policy grants the rights to the Amazon SQS ReceiveMessage and SendMessage actions for the queue called queue_xyz in your AWS Account, and the policy is attached to users named Bob and Susan (Bob and Susan have the permissions stated in the policy). This Amazon SQS policy also gives Bob and Susan rights to the ReceiveMessage and SendMessage actions for the same queue.

⁠

Visibility timeout

The visibility timeout is the amount of time a message is invisible in the queue after a reader picks up the message.

If a job is processed within the visibility timeout the message will be deleted.

If a job is not processed within the visibility timeout the message will become visible again (could be delivered twice).

The maximum visibility timeout for an Amazon SQS message is 12 hours.

⁠

What is Amazon Simple Queue Service - Amazon Simple Queue Service Queue messages that one component in the application generates to be consumed by another component using Amazon SQS. docs.aws.amazon.com⁠

⁠

Using policies with Amazon SQS - Amazon Simple Queue Service Queue messages that one component in the application generates to be consumed by another component using Amazon SQS. docs.aws.amazon.com⁠

⁠

Benefits of using Amazon SQS

Amazon SQS queue types

queue types

Using dead-letter queues in Amazon SQS

Using policies for dead-letter queues

Amazon SQS short and long polling

Differences between long and short polling

Using Amazon SQS and IAM policies

Visibility timeout

Want to print your doc?
This is not the way.

Try clicking the ⋯ next to your doc name or using a keyboard shortcut (

CtrlP

) instead.