Serverless Data Lake

Human-development indicators are useful for longitudinal and cross-country analysis, but a raw CSV is a poor long-term analytical interface. The goal was to build a small but production-inspired data platform that could ingest the source data, impose a usable schema, preserve the raw source, produce an analytics-ready representation and allow questions to be answered with SQL without operating a database cluster.

The lake is organised around S3 storage zones: raw data is preserved separately from processed data, while Athena query results have their own location. AWS Glue provides schema discovery and cataloguing, with the transformation path cleaning nulls, renaming and casting fields, and writing Parquet output partitioned by year. Athena then queries the processed data directly through the Glue Catalog, while IAM and Lake Formation apply least-privilege, resource-scoped access across the raw, processed and query-result zones, and CloudWatch gives the pipeline's managed components basic operational visibility.

S3 over a traditional database

The project is primarily an analytical data platform rather than a transactional application. S3 provides durable, low-operations storage while keeping the raw source available for reproducibility and allowing processed representations to evolve independently.

Athena over an always-on warehouse

The workload is analytical and relatively bursty, so a serverless query engine fits the usage pattern better than paying to keep a database or warehouse cluster running. Athena also keeps the query layer close to the S3 data rather than introducing another storage system.

Parquet over processed CSV

The processed layer is an analytical representation, not merely a cleaned copy of the source. Parquet gives the query layer a columnar format and, combined with year partitioning, creates a more appropriate foundation for selective analytical queries.

Lake Formation layered over IAM

Bucket- and role-level IAM policies are already enough to keep the lake private. Lake Formation was added anyway, to practise resource-level governance — the kind of scoping that matters once more than one consumer needs different visibility into the same lake, rather than everyone sharing one bucket policy.

Free Tier as an architectural constraint

The entire design had to remain within AWS Free Tier limits during the 10-day build. That constraint pushed the architecture toward managed, event-oriented and serverless services while discouraging always-on infrastructure that would add cost without improving the learning objective.

Raw-to-curated data path

The source file, hdi-indicators.csv, contains country/year indicators including HDI, life expectancy, education index and GNI per capita. The processing layer handles missing values, renames and casts fields, then writes the cleaned records to Parquet partitioned by year.

Catalog-driven analytics

Glue's Data Catalog provides the schema boundary between object storage and SQL. Athena can therefore address the processed files as a table rather than requiring application code to parse every object on each analysis run.

Operational visibility

CloudWatch captures Glue job and query activity so failures — a bad crawler config, an IAM boundary, a malformed record — surface as logs rather than a silently empty table.

The interesting engineering work was in the boundaries: keeping the source immutable, making the processed schema predictable, choosing a storage format suited to analytical access, and ensuring the query layer could discover the resulting partitions. The project is deliberately small, but the architecture follows the same raw-to-processed separation used in larger data platforms.

The result is a serverless AWS data-lake prototype capable of taking the HDI source dataset from raw CSV through catalogue, governance, transformation and partitioned Parquet storage into Athena queries. Example analytical use cases include South Africa's HDI trend, country improvements, and regional or yearly average HDI views. The AWS account used for the build was later lost, so the environment isn't currently live — what's documented here is the completed architecture and implementation, not a running deployment.

A data lake is an architecture, not an S3 bucket

The value comes from the separation of raw and processed data, explicit schemas, repeatable transformations and a query layer. Storage alone does not make the system analytically useful.

Partitioning is part of query design

Choosing year as a partition key connects the physical layout of the data to the questions the system is expected to answer. Data layout should be designed with query patterns in mind.

Cost constraints can improve architecture

Working within Free Tier limits forced the design toward services that scale with use and reduced unnecessary infrastructure. That made the project cheaper to run while also making the serverless architecture more deliberate.

Governance is a design decision, not a retrofit

Layering Lake Formation over IAM while the lake still had one consumer made access scoping something chosen up front, rather than something bolted on once a second consumer actually needed different visibility.