Publishing a package on PyPI. It’s one of those things that looks intimidating from the outside, but turns out to be a matter of good timing and a precise enough problem to solve.

You know the feeling: a report spitting out weird numbers, an analyst telling you “the totals don’t match,” and you spend your day tracing back up the chain to find where the data went wrong. Often, the problem could have been detected automatically if someone had put a test somewhere.

Documenting columns in a source database is the kind of task nobody wants to do. You have an operational system with hundreds of tables, thousands of columns, and documentation ranging from “nonexistent” to “a 2017 comment that says TODO: document this.”

Documentation is the thing nobody wants to do. Especially in data. You have hundreds of columns across dozens of tables, and someone asks “what’s the status field in the orders table?” And the honest answer is often “uh… an enum I think that probably means X.”

Snowflake is fundamentally SQL-first. That’s its strength: everything is driven by SQL, from grants to object creation to transformations. Infrastructure, we’ve seen how to tame it with Terraform in the previous article. But data transformations fall into a blind spot. SQL scripts scattered everywhere, no tests, no serious versioning, one colleague who knows what order to run things in.

There’s a moment in every data engineer’s life when you find yourself staring at a 300-line SQL file that creates roles, grants, warehouses, and you wonder how you got here. This is my story.