home

Blog


Reduce SQS costs with these three strategies


SQS is a very popular and cheap service from AWS which seems pretty inoffensive to your monthly bill. However, it charges per request, not per message, and that detail is where most of the waste hides. Things like racking up old and unused queues with live consumers, using FIFO indiscriminately and being careless about polling will surely make AWS very happy and your pockets lighter. The good news is: there are a few ways which will help you save some money.

Enable Long Polling

By default, SQS uses short polling: your consumer calls ReceiveMessage, SQS answers immediately (even with an empty response), and your consumer calls again. Every one of those calls are billed, including the empty ones. A worker polling in a tight loop can easily generate millions of requests per month just to hear "nothing new".

With long polling, SQS holds the connection open for up to 20 seconds and only responds when there's a message (or when the wait time expires). One call every 20 seconds instead of dozens per second - same behavior for your application, but a fraction of the requests on your bill. Also, enabling it is a one-liner at the queue level:

aws sqs set-queue-attributes \
  --queue-url $QUEUE_URL \
  --attributes ReceiveMessageWaitTimeSeconds=20

Or per request, by passing it on the ReceiveMessage API request. There might be no downsides for most workloads, but be sure to check if it fits your use case.

Prefer standard queues to FIFO

FIFO queues are convenient, exactly-once processing and strict ordering out of the box, but that convenience costs more per request than standard queues, and it also caps your throughput (you'll also have to pay more to increase it). The thing is, for a lot of systems you can get the same guarantees in the application layer.

Handling duplicates

Standard queues offer at-least-once delivery, so the same message may arrive more than once. Instead of paying FIFO to deduplicate for you, make your consumer idempotent. Give each message a unique ID and record it when processed (a unique constraint in your database or a key in Redis). If the same ID shows up again, skip it (or do something else, I don't know).

Ensuring ordering

Standard queues also don't guarantee ordering, but in practice you might not need the global ordering from the queue. To circumvent that issue, add a timestamp (or version number) to each message at the producer, generated by the application. Then, when consuming, compare it against the last message processed: if the incoming message is older, skip it. An out-of-order message becomes a no-op instead of a bug.

Migrating from FIFO to standard

This is the sad part, where there's no configuration, nothing simple or even magical to do that. To go back to using standard queues you need to create another queue and migrate your application to the newly created one. If you have a lot, like an enormous amount of FIFO queues, and you need to migrate them, I feel for you, I really do.

Delete unused queues

This one sounds obvious, but it's pretty easy to forget a queue created for a feature that was deprecated, an experiment, an old environment and, along with it, a consumer somewhere still polling it 24/7. Since empty receives are billed like any other request, you're literally paying for API calls that will never return anything.

A quick way to find these zombies is to look at the NumberOfEmptyReceives metric in CloudWatch. A queue with a high count of empty receives and zero messages sent is a strong candidate for deletion.

Wrap-up

None of these strategies require redesigning your architecture. Long polling is a configuration change, idempotency and timestamps are small additions to your application code, and cleaning up unused queues is just housekeeping. So, keep up with these tips, and make every SQS request count.

Powered by Simple Blog API