Course Overview
Interns build the automated data flow behind the prototype: collect data from sensors, the app and forms, clean it, store it, and serve it to a dashboard with the product's key metrics. Tools: Python (pandas) in VS Code or Colab, SQL with Supabase/PostgreSQL, n8n, GitHub Actions or cron, Looker Studio or Metabase. The course runs for 4 weeks (12 sessions), with the last session used to showcase and evaluate your deliverable.
Course Curriculum & Modules
Course Deliverable
Working Data Pipeline and KPI Dashboard: a data flow diagram; an automated pipeline (extract, transform, load) running on a schedule; a database schema with raw and clean tables; data quality checks with logging and alerts; a KPI dashboard tracking the prototype's usage and performance; and pipeline documentation.
Data Engineering Department Pathway
Data Engineering focuses on relational SQL/NoSQL databases, automated ETL data pipelines, big data streaming (Spark/Kafka), cloud data warehouses, and enterprise data governance frameworks.
Computer & Hardware Requirements
Key Learning Outcomes & Benefits
- ✓Draw a data flow diagram and extract data from APIs, files, MQTT streams and forms.
- ✓Clean and validate data with pandas and add a rejects table for bad records.
- ✓Design raw and clean tables and load data into Supabase on a schedule.
- ✓Add logging, retries and alerts and track 5 KPIs on a live dashboard.
