Why Data Engineers Should Care about DataBricks IPO.

Some poor Data Engineer is sweating and typing away in a dark closet … moving data, solving bugs, just trying to get through the day. Why should the ‘ole Data Engineer care about the huff-a-luff around the billion dollar series recently done by DataBricks? I mean what possible reverence could it have on the day to day life of a Data Engineer and why should they care at all? You ever heard of that proverbial light at the end of the tunnel is actually a train steaming your way ready to pulverize you? That’s why.

Why Should Data Engineers Care about the DataBricks billion dollar series?

This is the question, should Data Engineers who just write code all day really care about this?

YES.

Why? Because it tells the story about the future, where the world is going, and you might want to follow along eventually.

We’ve seen this story play about before.

This is a story that plays out slowly over time, and it can be kind of hard to catch on if you’re not watching. I would liken this to the rise of Kubernetes for example. There was a time when it was just another up and coming technology trying to fight its way to the top. But eventually you start getting signs from above when certain technologies start to really make an impact.

For example, if you are a developer in the data world, or any world for that matter, and you never learned Kubernetes …. you are probably behind the 8ball. It’s popular, it’s out there, and a lot of infrastructure is running on it now, and will probably only increase in the future.

Can you get by without knowing new technology … of course, do you want to?

Don’t miss the writing on the wall with DataBricks and Snowflake.

The data god’s are trying to tell you something about the shift in the Matrix.

Distributed systems are getting popular, Spark is getting even more popular, distributed SQL like cluster systems .. aka Redshift and Snowflake are getting popular.

The shift is clearly towards organizations trying to take advantage of these big data distributed processing systems, all the while having the “complex” management of these systems abstracted away!

There is a lesson to learn written between the lines.

Data Engineers should sit up and take note. Don’t be that dude still writing Cobol 15 years from now. Like it or not there is a clear trend towards managed services for distributed big data systems …. like DataBricks. Why else would they be making nearly 500 million in ARR? Because orgs can’t get enough of it.

Baaaa … it’s just Spark I know Spark. You could always say that. But an effective Data Engineer is always aware of the systems and options available to solve problems. Maybe you haven’t heard of Delta Lake? That’s my point.

Learn the new technologies and vendors, understand what they offer and why people use it. Your 5 years older self will thank they now self.