Skip to content

Error Handling

Transient errors are handled by a retry.

If Lambda fails after max retries (default is 2), the event is sent to a DLQ (SQS).

Workflow

Lambda processes Parquet files from S3.

If Lambda fails after max retries (default is 2), the event is sent to a DLQ (SQS).

You can inspect or reprocess messages in the DLQ manually or automatically.

Setup

  1. Create an SQS Queue for DLQ You can do this via the AWS Console or CLI: aws sqs create-queue --queue-name parquet-processing-dlq

  2. Attach DLQ to Lambda If using the AWS CLI:

aws lambda update-function-configuration \ --function-name parquetProcessor \ --dead-letter-config TargetArn=arn:aws:sqs:us-east-1:123456789012:parquet-processing-dlq If using the AWS Console:

Go to your Lambda function

Configuration → Asynchronous Invocation → DLQ

Select the SQS queue

Make sure the Lambda function has sqs:SendMessage permission for this DLQ.

  1. IAM Policy for Lambda to Send to DLQ Attach this IAM permission to the Lambda role:
{
  "Effect": "Allow",
  "Action": "sqs:SendMessage",
  "Resource": "arn:aws:sqs:us-east-1:123456789012:parquet-processing-dlq"
}

Sample Python Lambda Code with Error

import boto3
import pyarrow.parquet as pq
import io

def lambda_handler(event, context):
    s3 = boto3.client('s3')

    for record in event['Records']:
        bucket = record['s3']['bucket']['name']
        key = record['s3']['object']['key']

        try:
            obj = s3.get_object(Bucket=bucket, Key=key)
            table = pq.read_table(source=io.BytesIO(obj['Body'].read()))
            # Process table...
            print(f"Processed file: {key}")
        except Exception as e:
            print(f"Error processing file {key}: {str(e)}")
            raise e  # Rethrow to trigger retry and potentially DLQ
  1. View Messages in DLQ (Optional Reprocessing)
aws sqs receive-message --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/parquet-processing-dlq

🔍 Notes DLQ only captures asynchronous invocations (e.g., from S3, SQS, EventBridge). DLQ stores the original event payload that failed.