Data Engineering

Building a data platform involves more than writing SQL. These notes cover the infrastructure, transformations and governance that make a platform maintainable in production.

Managing Snowflake with Terraform covers the infrastructure side. dbt as infrastructure connects versioned transformations with testing and deployment. The dbt collection goes deeper into model documentation and quality checks.

dbt-guard: My First Python Package (and Why I Needed It)

5 April 2026·1486 words·7 mins

Data Engineering Open Source dbt Python Data Engineering Open-source projects

Publishing a package on PyPI. It’s one of those things that looks intimidating from the outside, but turns out to be a matter of good timing and a precise enough problem to solve.

dbt Data Tests in YAML: Stop Praying Your Data Is Correct

27 February 2026·Updated: 28 October 2026·1601 words·8 mins

Data Engineering Data Quality dbt Data quality Data Engineering Testing

A report spits out weird numbers. An analyst tells you “the totals don’t match,” and you spend your day tracing back up the chain. The habit I keep coming back to is boring: dbt data tests, declared in YAML beside the columns, and actually run in CI. Tell the pipeline what you expect before a dashboard tells you what went wrong.

Documenting a Source Database with Multi-Agent LLMs

6 February 2026·Updated: 27 October 2026·1488 words·7 mins

Data Engineering AI dbt LLM Data Engineering AI

Documenting columns in a source database is the kind of task nobody wants to do. You have an operational system with hundreds of tables, thousands of columns, and documentation ranging from “nonexistent” to “a 2017 comment that says TODO: document this.”

dbt Documentation: YAML, persist_docs and Governance

16 January 2026·Updated: 28 October 2026·1392 words·7 mins

Data Engineering Data governance dbt Snowflake Data Governance Data Engineering

In my dbt projects, YAML is where descriptions, tests and ownership live together. I use persist_docs to carry model descriptions into Snowflake, then CI checks to stop the documentation quietly falling behind. The governance part comes from what we build around that metadata. It doesn’t happen because a file happens to be YAML.

dbt and Terraform: My Snowflake Infrastructure as Code

26 December 2025·Updated: 28 October 2026·1953 words·10 mins

Data Engineering dbt Snowflake Data Engineering SQL

I use Terraform to provision my Snowflake environment and dbt to describe the transformations inside it. Databases, roles and warehouses on one side; models, tests and documentation on the other. Both are code, reviewed in git and deployed through a pipeline.

Snowflake + Terraform: Stop Managing Your Data Infrastructure in SQL

5 December 2025·1692 words·8 mins

Data Engineering Infrastructure Snowflake Terraform Data Engineering Infrastructure as Code

There’s a moment in every data engineer’s life when you find yourself staring at a 300-line SQL file that creates roles, grants, warehouses, and you wonder how you got here. This is my story.

↑