,

The Difficulties of Senior Engineer …. are not Engineering

Well, I hate to break the news to you. I was the same when I first started, writing code that is. I was a zealot. I was zealous for every new thing I learned, every new language, every new approach, I would find the preacher who was preaching the message I wanted to hear … OOP, functional, Kimball, this, that, the other thing.

You’re young and full of life. You think that your Software career revolves around … software. The pinnacle of your mountain seems to be becoming that “perfect programmer” who can write anything without any bugs.

Yet, when you get to the top of the mountain you find you’ve been deceived. Not all that glitters is gold.

Read more

,

Polars vs Spark

, ,

SQL Bad, Reddit Mad

, ,

SparkSQL is Destroying your Pipelines

It’s true, even if you don’t want it to be. SparkSQL is destroying your data pipelines and possibly wreaking havoc on your entire data team, infrastructure, and life. In your heart of hearts, you’ve probably known it for years. With great power comes great responsibility. We all know that even us Data Engineers are human and fallible.

Once those tentacles of SparkSQL get their hold on you, the probability of survival is low. Sure, there are a few wizened old engineers with enough battle scars to make it through unscathed. The rest of us will be maimed.

Read more

, ,

Datafusion SQL CLI – Look Ma, I made a new ETL tool.

Sometimes I just need something new and interesting to work on, to keep me engaged. A few days ago I was lying by the river next to a fire, with the cold air blowing on my face and the eagles soaring above. Thinking about and contemplating life and data engineering … something flitted across my mind, just a little fragment of an idea someone had written about.

The little fragment had to do with Datafusion, a Rust-based query engine, and something about it having a SQL CLI interface.

What an interesting thing. I’ve used Datafusion a few times, here and there, I love Rust because it’s fast. I’m a Data Engineer so I’m eternally enslaved to SQL whether I like it or not. This whole thing just seemed like an interesting little tidbit to poke at.

It basically made me wonder if I could combine the Datafusion SQL CLI with bash into a new ETL tool. Simple, small, fast, and maybe fun? Just because I can?

Read more

, ,

How to be Better Than Everyone Else

Ok. Get off your high horse. You are human just like the rest of us. Just like your ancient ancestors who were throwing rocks and sticks at each other a thousand years ago … you are looking for a leg up on the competition. Isn’t that the world we live in? At the end of the day, no one is looking out for you, besides you.

Do you know what all those famous “influencers” have over you? Those you look at with boiling bitterness buried somewhere inside you. Besides a smattering of luck here and there, there is one other trait they have that most people don’t.

There are general branches and streams of actions that come off that main branch. But without that central drive, all else is useless.

What is it that apparently 1% of the data population has that the 99 don’t? Simply put, in the words of your grandparents, hard work.

Read more

,

Fleetclusters for Databricks + AWS to reduce Costs.

Show me the money. That’s what it’s all about. I have a question for you, to tickle your ears and mind. Get you out of that humdrum funk you are in. Here is my question, riddle me this all you hobbits.

“Of what use is, and what good does the best and most advanced architecture provide if your bosses boss starts to look at the cost and starts to ask questions?”

The thing is, as engineers and developers we love to do fun stuff. We follow the crowds, don’t we? We are like those lemmings, we just do what is cool.

Read more

, , ,

The Ultimate Data Engineering Chadstack. Running Rust inside Apache Airflow.

Is there anything more Chad than Apache Airflow … and Rust? I think not you whimp. What two things do I love most? At the moment Rust and Airflow are at least somewhere at the top of that list. I wring my hands sometimes, wishing that things and technologies somehow come together into some bubbling soup and witches concoction from the depths. Then I had a strange thought while laying in bed one night.

What would happen if I ran my Rust inside my Apache Airflow? What would happen? Would the sun go dark? Would SQL Servers everywhere puke up their log files and go to Davey Jones’s locker? Birds fall from the sky? Why hasn’t anyone done this before, why isn’t anyone making this happen in real life?

Read more

, , , ,

DuckDB + Delta Lake (the new lake house?)

I always leave it to my dear readers and followers to give me pokes in the right direction. Nothing like the teaming masses to set you straight. Recently I was working on my Substack Newsletter, on the topic of Polars + Delta Lake, reading remove files from s3 … I left a question open on my LinkedIn account.

I had someone jog my leaky memory in favor of DuckDB. I haven’t touched DuckDB in some time, and I’m sure it’s under heavy development what with that Mother Duck and all.

So, it’s time to talk about DuckDB + Delta Lake.

Read more

, ,

The Role of DevOps and CI/CD in Data Engineering

In the vast world of data, it’s not just about gathering and analyzing information anymore; it’s also about ensuring that data pipelines, processes, and platforms run seamlessly and efficiently. Nothing screams “why are flying by night,” than coming into a Data Team only to find no tests, no docs, no deployments, no Docker, no nothing. Just a mess and tangle of code and outdated processes, with no real way to understand how to get code from dev to production … without taking down the system.

This is where the principles of DevOps and Continuous Integration/Continuous Deployment (CI/CD) come into play, especially in the realm of data engineering. Let’s dive into the importance of these practices and how they’ve become indispensable in modern data engineering workflows.

Read more