IMDb Analytics Pipeline

IMDb publishes its full catalog as raw, messy public data — ~200 million records across seven datasets, riddled with placeholder values, string-encoded numbers, and multi-valued fields. In that state it's useless for analytics. The goal was to turn it into a clean, trustworthy warehouse that powers self-service BI.

An end-to-end pipeline from raw TSV to dashboards:

• Ingestion of seven compressed IMDb TSV datasets (~200M records total, up to 91M rows in a single table)
• Data profiling and cleaning in Alteryx + Python — normalizing IMDb's \N placeholders to NULLs, casting string-encoded numerics, and validating logical constraints
• Azure Data Factory pipelines for automated, dependency-aware loads with error handling and re-runs
• A two-layer Snowflake warehouse: 7 staging tables (cleaned raw) feeding a curated dimensional layer optimized for BI
• Power BI and Tableau dashboards over the curated layer — genre trends, rating distributions, and cast/crew participation

The trade-offs — and why:

• Separated staging from a curated analytics layer so heavy cleaning never slows down BI queries
• Profiled every dataset before loading to catch quality issues upfront instead of debugging dashboards later
• Leaned on ADF's dependency-aware orchestration because the seven datasets must load in order to preserve referential integrity
• Gave the 51M-row alternate-titles dataset (title.akas) special handling to enable region and language analysis

The hard parts:

• Scale: single tables up to 91M rows meant profiling and loads had to be batched and pushed down into Snowflake rather than done in memory
• Messy source data: IMDb's \N placeholders, string-encoded numerics, and multi-valued fields all needed consistent global cleaning rules
• Trust: added row-count reconciliation and primary-key uniqueness checks across every pipeline layer so the dashboards could be relied on

A clean, queryable IMDb warehouse validated end-to-end with row-count reconciliation and PK-uniqueness checks, serving Power BI and Tableau dashboards for genre trends, rating distributions, content-type comparisons, and cast/crew analysis.

Incremental refreshes as IMDb updates its datasets, adding dbt tests to the transformation layer, and a semantic metrics layer for consistent KPIs across BI tools.