Why did Golang lose to Rust for Data Engineering?

A few years ago I wasn’t sure, who was going to win, Golang seemed to be popular, and still is for that matter. When I first wrote a little Golang (~2+ years ago) I was just trying to see what the hype was all about. The funny thing is, at the time, and today, it seems like the Golang syntax is much simpler than Rust, easier to learn and pick up by far.

I’ve written enough Rust by now to learn most of its concepts and be able to pretty much hack out anything I want, but it still takes time and a lot of paying attention to get it done right. Golang on the other hand is a little more forgiving. So what’s the deal?

Why has Rust beat out Golang for Data Engineering dominance (for building tools and the like)?

Logically, it doesn’t seem to make sense on the surface at first glance. But as someone who’d been doing Data Engineering awhile, after using both tools and seeing the landscape the answer is sorta obvious.

What Rust offers Data Engineering that Golang doesn’t.

Of course, I know I’m taking my life in my hands when talking about this, the Golang zealots will come for my head, but even they can no longer argue that Golang won in the Data Engineering race for backend language (no Python). With tools like Polars, Datafusion, etc it’s clear that Rust has a bright future at the forefront of backend Data Engineering tooling while Golang will be relegated to the corners.

I know what I’m about to list isn’t just specific to Rust, but it’s about all these things together, in their totality, that have caused Rust to step on Golangs head for first place in Data Engineering.

• Ownership and Borrowing concepts in Rust cause bullet-proof software to be built (at least more than Golang)
• Rust is very, very fast.
• Rust integrates well with Python (think Pyo3).
• Rust has reliable Dataframe tools (Datafusion and Polars).
• Rust has an imperative programming style.
• Rust handles errors very well (Golang doesn’t).
• Rust handles a wide variety of problems very well.
• The package and crate system is easy and popular to use.

It isn’t just one of these things … it’s all of them.

When I first started writing Golang in a Data Engineering context I was amazed at the lack of what I would call “first-class citizen” tooling to support common DE-specific tasks. It was clear from the start that Golang would never take over Data Engineering as the backend language of choice.

At the time the way it handled errors was horrible and tedious, and no good DataFrame packages, the community although large seemed to be skewed toward infrastructure. It also lacked many common packages you could find in Rust and Python.

Also, Rust did something very smart. It made itself easy to be wrapped up into Python packages, a genius idea. Python will always rule the world, inserting yourself as an option underneath as a core language to build other tooling on top of was a great foresight and move.

In the end, I think Golang was and is a great tool, and it seems like it will probably continue to grow and hold market share, but probably not in Data Engineering, in fact, with the rise of Rust, its use will drop off in Data Engineering. Sometimes I wonder if Golang will be the next Scala, dying on the vine.

Do you want to know the most important thing for me?

Golang isn’t as fun to write as Rust. You write something in Rust that compiles and it feels like you accomplished something … you built something that would run for the next 20 years without problems. Heck, wrap it up in a Python package and you have something the masses will drool over for a decade.