, , ,

Airflow vs Dagster

Dagster, the first few times I read the name, I just couldn’t take the tech stack seriously …. it’s still kinda hard. Today I want to compare Airflow vs Dagster, mostly explore what Dagster is and does. But I want to compare it to the popular Apache Airflow project so people have some context for it. It’s kinda hard keeping up on all the new stuff these days, I usually just wait till I see enough articles and tweet floating around about it, then I know it’s maybe worth a peak. Let’s crack open Dagster, and see if it’s better then the name chosen for it.

Read more
, ,

What the Marooned Ben Gunn Teaches us about Solo Data Engineers

I always envied Ben Gunn in Treasure Island a little bit. Alone all those years, digging up gold and treasure, hunting wild goat, and living in a nice little cave. Living off the land, king of his island, gone half mad, but somewhat still there. Happy to see other people, but always a little bit of a recluse … too many years alone. I think there is a lesson for us here, and it has to do with the solo Data Engineer. That lone ranger, out there in the data wilderness, surviving.

Read more
, , ,

Introduction to Apache Flink for Data Engineers

Not going to lie. I’ve been trying to figure out for awhile where Apache Flink fits in the Data Engineering world for awhile now. A year or two ago I didn’t seem much content posted about it, but it seems to be picking up stream. I’ve mostly managed to avoid understanding what Flink is or does, but I figured it’s time to give my brain a much needed workout. When I was ignoring Flink, I just chalked it up as another messaging/streaming system like Kafka or Pulsar. Apparently I was wrong … no surprise there.

Read more

Contract to Hire (Take a Chance With Us …. We Won’t With You).

LIfe’s no fun if you don’t keep things interesting. It’s time to ruffle a few feathers.

Read more
, , , ,

The 3 Types of Data Engineers, Which One Are You?

Every good story starts with a few different characters right? It’s like the spice of life, little bit of this, little bit of that. It’s the way of the world. In all my data wandering I’ve come across lot’s of different types of data engineers. I can usually put them into three different categories, somewhat similar but in many ways quite different.

Read more
, ,

A Piece of DevOps that most Data Engineer’s Ignore.

I am always amused by the apparent contradictory nature of working in the world of data. There is always bits and pieces that come and go, the popular, the out of style … new technology driving new approaches and practices. One of the hot topics the last decade has produced is DevOps, a now staple of most every tech department. Like pretty much every other newish Software Engineering methodology, data world has struggled to adopt and keep pace with DevOps best practices. Once these is always a thorn in my side, making my life more difficult. The simplicity with which it can be adopted is amazing, and the unwillingness and lack of adoption is strange.

Read more