Back to selected work

Data Engineering

Serverless Data Lake with Quality Checks

AWS CDK PoC for bounded public-data ingestion, Spark validation, partitioned S3 Parquet and Athena quality reporting.

Illustration for Serverless Data Lake with Quality Checks
DATENovember 2026
BUILT WITH
Amazon S3AWS GlueAmazon AthenaAmazon EventBridge SchedulerAmazon EventBridgeAmazon CloudWatch

Problem

Demonstrate a practical, repeatable batch lake pattern that separates clean and rejected rows, exposes data-quality metrics and makes analysis queryable without claiming unverified production results.

Architecture

CDK provisions four encrypted, private S3 zones (raw, curated, rejected, Athena results), explicit Glue catalog tables, a Glue 4.0 Spark ETL job with schema/null/range/deduplication checks, partitioned Parquet, JSON audit summaries, and an Athena workgroup with a per-query scan cap. An optional disabled EventBridge Scheduler trigger and a CloudWatch job-failure alarm are included. A bounded Chicago public taxi sample ingestion script, SQL examples, tests and manual OIDC deployment workflow live in the repository. The image field names the portfolio asset; the demoUrl is a proposed repository URL, not a hosted demo.

Results

Pending deployment. Local CDK assertion tests and Spark-independent validation unit tests are included, but no deployed quality figures, job duration, Athena queries, alarm behavior or observed costs are claimed. Capture these after a sandbox deployment.