Technical Blog
Deep dives into data engineering, cloud architecture, AI/ML, and the technologies that power modern data platforms.
Complete comparison of GCS bucket synchronization methods in Airflow. Covers BashOperator with gcloud rsync, GCSSynchronizeBucketsOperator, GCSToGCSOperator, and Storage Transfer Service with pros, cons, and production code samples.
Production-grade ETL pipeline from GCS Avro files to BigQuery using Cloud Composer. Features partition gap detection, schema validation, data count verification, and Slack alerts for monitoring daily/hourly data loads.
Production-ready Apache Airflow setup with PostgreSQL, Celery Executor, and RabbitMQ on macOS. Covers database migration, worker scaling, Flower monitoring, and troubleshooting common distributed task queue issues.
Step-by-step Apache Airflow installation on macOS with SQLite. Covers PyPI install, AIRFLOW_HOME configuration, database migration, admin user creation, scheduler/webserver setup, and daemon process management for production pipelines.
