Technical Blog

Tag: #bigquery

Deep dives into data engineering, cloud architecture, AI/ML, and the technologies that power modern data platforms.

August 17, 2024•data-engineering

GCS to BigQuery ETL Pipeline: Complete Airflow DAG with Validation & Slack Alerts

Production-grade ETL pipeline from GCS Avro files to BigQuery using Cloud Composer. Features partition gap detection, schema validation, data count verification, and Slack alerts for monitoring daily/hourly data loads.

Read article

April 23, 2024•terraform

BigQuery - Creating Primary Key/Foreign Key in Terraform

terraformgoogle-cloudbigquery

Example

Read article

April 22, 2024•python

High-Performance Async Python ETL: Processing Millions of API Records to BigQuery

pythonasyncapi

Production-ready async Python pattern for extracting millions of API records with automatic pagination, retry logic with exponential backoff, and seamless BigQuery integration using aiohttp and uvloop for maximum throughput.

Read article