Geek Logbook

Tech sea log book

Sending Athena Query Results to Amazon SQS: Architecture, Costs, and Limitations

Introduction

Amazon Athena is a serverless query service designed for interactive analysis of data stored in Amazon S3. Amazon SQS (Simple Queue Service), on the other hand, is a fully managed message queue used to decouple and scale distributed systems. A common question arises when combining both services: can Athena send query results directly to SQS? The short answer is no. This article explains why, what the correct architecture looks like, and the associated costs.


Why Athena Cannot Send Data Directly to SQS

Amazon Athena is a read-only query engine. Its execution model is limited to:

  • Reading data from S3
  • Writing query results back to S3

Athena has no native integration to push data into downstream services such as SQS, SNS, DynamoDB, or Kinesis. This is a deliberate design decision aligned with its role as an analytical service rather than an event producer.

Reference:
https://docs.aws.amazon.com/athena/latest/ug/what-is.html

The Correct Architecture Pattern

To move data from Athena to SQS, an intermediate step is required.

Standard Flow

  1. Athena executes a SQL query.
  2. Query results are written to Amazon S3.
  3. A compute component reads the results from S3.
  4. Each record is published as a message to Amazon SQS.

Common Implementations

  • AWS Lambda triggered by S3 ObjectCreated events
  • AWS Glue jobs (Python Shell or Spark)
  • External workers using boto3
  • AWS Step Functions orchestrating the workflow

This pattern respects AWS service boundaries:

  • Athena for analytics
  • S3 for durable storage
  • SQS for asynchronous message delivery

Reference:
https://docs.aws.amazon.com/athena/latest/ug/querying.html


Cost Considerations

Amazon Athena

  • Billed per data scanned: USD 5 per TB
  • Output data stored in S3 is billed separately

Reference:
https://aws.amazon.com/athena/pricing/

Amazon SQS

  • 1,000,000 requests per month are free
  • After the free tier:
    • Standard queues: approximately USD 0.40 per million requests
  • Each send, receive, and delete operation counts as a request
  • Messages larger than 64 KB are billed as multiple requests

For moderate batch workloads (for example, tens of thousands of messages per month), SQS usage typically remains within the free tier.

Reference:
https://aws.amazon.com/sqs/pricing/


Message Retention and Delivery Semantics

Messages sent to Amazon SQS:

  • Remain in the queue until explicitly deleted by a consumer
  • Or until the Message Retention Period expires (configurable from 1 minute to 14 days, default 4 days)
  • Are delivered with at-least-once semantics in Standard queues
  • May be delivered more than once unless FIFO queues are used

Reference:
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-lifecycle.html


When This Architecture Is Appropriate

This pattern is suitable when:

  • Analytical results need to trigger asynchronous processing
  • Batch analytics must be decoupled from downstream systems
  • Reliability and retry semantics are required

It is not suitable for:

  • Real-time streaming use cases
  • High-frequency event generation directly from SQL queries

Conclusion

Amazon Athena cannot send data directly to Amazon SQS. The supported and scalable approach is to use Amazon S3 as an intermediate storage layer and a compute service (such as AWS Lambda or AWS Glue) to publish messages to SQS. This design aligns with AWS best practices, keeps costs predictable, and scales reliably.

Tags: