Streaming Postgres data to Databricks Delta Lake in Unity Catalog

Over the many years I’ve been pounding my keyboard … Perl, PHP, Python, C#, Rust … whatever … I, like most programmers, built up a certain disdain for what is called Low Code / No Code solutions. In my rush to worship at the feet of the code we create, I failed, in the beginning, to recognize some important axioms …

• more code is not always better
• complicated is not always better
• provide business value first

So, in that spirit, I took it upon myself to try one of these new Low Code ETL tools … and to do something rather hard as well. Streaming data from an RDS Postgres database to Delta Lake on Databricks with Unity Catalog. Not that ain’t no joke.

I did this with Estuary.

It went better than I expected, in fact, it was downright impressive. Watch below for the rest of the story.