Technical Blog

Tag: #google-cloud

Deep dives into data engineering, cloud architecture, AI/ML, and the technologies that power modern data platforms.

November 21, 2024•data-engineering

GCS to GCS Bucket Sync: 5 Airflow Operators Compared with Code Examples

data-engineeringapache-airflowgoogle-cloud

Complete comparison of GCS bucket synchronization methods in Airflow. Covers BashOperator with gcloud rsync, GCSSynchronizeBucketsOperator, GCSToGCSOperator, and Storage Transfer Service with pros, cons, and production code samples.

Read article

August 17, 2024•data-engineering

GCS to BigQuery ETL Pipeline: Complete Airflow DAG with Validation & Slack Alerts

data-engineeringapache-airflowbigquery

Production-grade ETL pipeline from GCS Avro files to BigQuery using Cloud Composer. Features partition gap detection, schema validation, data count verification, and Slack alerts for monitoring daily/hourly data loads.

Read article

April 23, 2024•terraform

BigQuery - Creating Primary Key/Foreign Key in Terraform

terraformgoogle-cloudbigquery

Example

Read article