, ,

PyArrow vs Polars (vs DuckDB) for Data Pipelines.

I’ve had something rattling around in the old noggin for a while; it’s just another strange idea that I can’t quite shake out. We all keep hearing about Arrow this and Arrow that … seems every new tool built today for Data Engineering seems to be at least partly based on Arrow’s in-memory format. So, […]

Read more
, , ,

The Abstractions Are Making You Dumb (rise of the Shallow Expert)

When I was young and full of myself, writing Perl and PHP, while your ma was still reading you a bedtime story and giving you a stuffy to fall asleep with, I had to program uphill, both ways, in the rain and snow. Not like you milk toast Data Engineers clickty clicking around Databricks and […]

Read more
,

What I’ve Learned After A Decade Of Data Engineering

After 10 years of Data Engineering work, I think it’s time to hang up the proverbial hat and ride off into the sunset, never to be seen again. I wish. Everything has changed in 10 years, yet nothing has changed in 10 years, how is that even possible? Sometimes I wonder if I’ve learned anything […]

Read more
,

Building Open-Source Python Packages – SparklePop

One of the things I love about Python is its flexibility and huge community, a community that puts out a never-ending stream of useful packages for the average Software Engineer. In a show of solidarity to the open-source community, I thought I would publish a PYPI package that will probably be used by 5 people […]

Read more
,

Databricks Buys Tabular – 1 Billion Dollar Deal. Iceberg vs Delta Lake?

The battle for the Data Warehouse, Data Lake, Lake House, or whatever you want to call it, in the age of AI just got more interesting. In an unsurprising move, Databricks has announced plans to buy Tabular for 1 billion dollars, beating out Snowflake who was reportedly trying to do the same thing.

Read more
, ,

Is Python OOP the Devil? Or Savior?

Nothing will raise the hackles on the backs of hairy and pale programmers who’ve been stuck in their mom’s basement for a decade like bringing up OOP (Object Oriented Programming), especially in the context of Python. It’s like two fattened calves prepared for slaughter, sharpen your knives, and take your place, it’s time to feast […]

Read more
, , ,

Building Data Platforms (from scratch)

Of all the duties that Data Engineers take on during the regular humdrum of business and work, it’s usually filled with the same old, same old. Build new pipeline, update pipeline, new data model, fix bug, etc, etc. It’s never-ending. It’s a constant stream of data, new and old, spilling into our Data Warehouses and […]

Read more

Why Data Engineering Pays So Well …. For Some, and Poor For Others

If you’ve ever been in the market for a Data Engineering job, or you’re alive and on Linkedin, you’ve probably been constantly inundated with job postings and requests pounding on your emails like a constant mountain stream even bubbling down a hill. If that’s not the case, then head over to the quarterly salary discussion […]

Read more

Why You Should Replace Pandas with Polars

I’m still amazed to this day how many folks hold onto stuff they love, they just can’t let it go. I get it, sorta, I’m the same way. There are reasons why people do the things they do, even if they are hard for us to understand. It blows my mind when I see something […]

Read more