My Airflow Journey
Starting from scratch is impractical, Google Cloud Platform's Composer is expensive, and debugging Airflow on Windows is challenging
Hi there! My name is Paulo and I am an analytics/data engineer and endurance runner based in Berlin. Welcome to my blog.
When I am not building data pipelines, you will find me training for my next endurance race. I write about both.
Currently preparing for Spartathlon 2026 and documenting the journey in my 246 series.
Linking to BigQuery in Azure Data Factory
In this article, we go over the steps, requirements and recommendations when creating a BigQuery linked service in Azure Data Factory
Nudging The End User
In PBI, you can nudge the user towards selecting the data they need first, before it is actually displayed in the report
Is My dbt Ready?
Set up activities in Azure Data Factory to track when your containerized dbt project has finished running
Keeping Table Highlights when Exporting to PDF in Power BI
Highlights in Power BI are a great presentation tool which can unfortunately act oddly when exporting to PDF
Alerting for dbt Errors in Azure Data Factory Using Databricks
Stay on top of errors that may come up when running your dbt build
Pulling Azure Container Instance Logs into Azure Data Factory
A description of how to include Azure Container Instance logs into your Data Factory pipeline.
Top 10 Books Read in 2022
Selected quotes from what I considered were the best books I read/listened to in 2022.
Syncing a Cloud Source Repository to your Composer Airflow Environment
And thus starting your CI/CD pipeline