Overview

Parquet files are written every 10 minutes to S3. We will review the options for a scalable, reliable and event driven architecture on AWS. Workflow can optionally be driven by step functions.(Step functions are charged based on number of state transitions)

AWS Lambda

S3 → EventBridge/SQS → Lambda → DynamoDB (state) → Target System

AWS Glue

S3 → Glue Crawler → Glue Catalog → Glue Job → Processed Data

AWS Athena

S3 → Athena → Target System