Building Background Jobs with SQS and Lambda

Some work does not belong in a request-response cycle. Generating an export, resizing an uploaded image, or contacting a slow external system can leave a user waiting for something that could happen in the background.

A queue gives that work a place to wait. A worker gives it a place to run. The difficult part is deciding what should happen when the worker runs twice, fails halfway through, or falls behind incoming traffic.

Define what the request promises

Imagine an export endpoint. Its immediate response should mean the job was accepted, not that the file already exists.

Create a job ID and make progress visible through a status endpoint or another notification mechanism. Useful states include queued, running, completed, and failed. Store the result location separately from the message so that a user can retrieve it after the queue has moved on.

Be careful when updating a database and sending a message as separate operations. If one succeeds and the other fails, the job can become stranded. For workflows that require reliable coordination, consider an outbox record written in the same database transaction and published by a separate process.

Keep the message small and meaningful

Prefer identifiers and a message version over copying an entire user record into the queue.

{
  "version": 1,
  "jobId": "export-123",
  "accountId": "account-456",
  "requestedAt": "2026-08-21T09:00:00Z"
}

The worker can load the required data using those identifiers. Decide whether an export represents data at request time or processing time; that product choice affects what must be stored with the job.

Validate the message before processing it. A missing field should become a controlled failure with a useful diagnostic, not an unpredictable exception several function calls later.

Assume a message can arrive again

Lambda's SQS event source mapping polls the queue and invokes the function with batches. Delivery can happen more than once, so the worker must handle duplicates. AWS explains the integration in Using Lambda with Amazon SQS.

For an export, use the job ID as the identity of the operation. Before starting expensive work, claim the job using an atomic state transition or another concurrency-safe mechanism. A plain read followed by an update can allow two workers to start together.

Also plan for a crash after producing the file but before marking completion. A deterministic result location or a recorded completion token can make that recovery easier. Deduplication must cover the side effect, not just the start of the handler.

Treat a batch as several independent outcomes

One failed record should not unnecessarily replay successful records. With partial batch responses enabled on the event source mapping, report the identifiers that failed.

This response shape is an example for a standard queue:

{
  "batchItemFailures": [{ "itemIdentifier": "failed-sqs-message-id" }]
}

Returning that object alone is insufficient without the mapping configuration. For FIFO queues, failure handling must also preserve ordering. Follow AWS's partial batch response guidance for the queue type you use.

Record failures with the job ID and message ID, while leaving credentials and full payloads out of the logs.

Give failures somewhere to go

Configure retry behavior and a dead-letter queue deliberately. A permanently invalid message should not consume processing capacity indefinitely.

Set the visibility timeout in relation to realistic processing duration and the Lambda configuration. If a message becomes visible while its earlier attempt is still working, duplicate execution becomes more likely. Long-running exports might need a different execution service rather than an ever-larger Lambda timeout.

A dead-letter queue also needs an owner. Write down how messages are inspected, how the underlying problem is fixed, and when replay is safe. Replaying a batch without checking idempotency can repeat the original damage.

Watch backlog and recovery

Queue depth alone does not tell you whether users are waiting too long. Monitor message age, failures, dead-letter arrivals, and job completion time together.

Load-test the worker against the database and downstream services before increasing concurrency. More workers can overwhelm the very dependency that caused the slowdown.

A useful background system is one you can interrupt and recover. Test duplicate messages, malformed payloads, and a worker crash between side effects. Those scenarios reveal more about reliability than a successful first run.