Ballista (Rust) vs Apache Spark. A Tale of Woe.

Sometimes it seems like the Data Engineering landscape is starting to shoot off into infinity. With the rise of Rust, new tools like DuckDB, Polars, and whatever else, things do seem to shifting at a fundamental level. It seems like there is someone at the base of a titering rock with a crowbar, picking and prying away, determined to spill tools like Java, Scala, Python, Spark, and Airflow, the things we’ve known and loved for years, from their lofty thrones.

Maybe they all have had their time in the Data Engineering sun, maybe it’s time to shake things up. It seems to be happening. It’s always hard to have those we hold dear be poked and prodded at. I’ve been using Spark since before it was cool, so when I started to hear the word Ballista start to show up here and there, I took note.

Besides, I’ve been dabbling my grubby little fingers in Rust for some months now, and have seen The Light. Is it possible I could be living at the dawn of a new era? A new and exciting frontier of Data Engineering, finally, after all this time? Could Rust really take over? Will something like Ballista pull that old Spark from its distributed processing tower and claim its rightful place?

Ballista – Rust-based Distributed Processing.

When I start to eyeball something new, some new Data Engineering tool, most of the time I would consider myself an eternal skeptic. I’ve been around the old block one too many times. The problems have changed very little in the last decade. The tools haven’t changed much, that is until lately.

Also, many of the tools that I see are released only incrementally better than their supposed competitors. Maybe a fancy UI update, a few Python decorators, and yet they claim to be some groundbreaking new tech that will revolutionize Data Teams. Yeah … not.

So why give my precious time to Ballista?

• Distributed Processing has become standard, and practically boring these days.
• Spark has been reigning supreme for a long, long time.
• Curiosity
• Knowing how game-changing Rust is.
• My eternal desire to push against the norm.

What is Ballista

So let’s start to get into Ballista. What is it? What can we expect from it?

“Ballista is a distributed compute platform primarily implemented in Rust, and powered by Apache Arrow.”

So, a Rust distributed system that uses Apache Arrow (columnar in-memory) format. Sounds almost too good to be true. Kinda makes a person wonder why we haven’t heard more rumblings about it, but I guess Spark does have a stranglehold. What else can we know about Ballista?

• Scheduler and an executor process that are standard Rust executables
• SQL and DataFrame queries can be submitted from Python and Rust
• SQL queries can be submitted via the Arrow Flight SQL JDBC driver
• The scheduler has a web user interface for monitoring query status as well as a REST API.

I mean that all seems straightforward. Dataframes, SQL, master, and executors, are par for the course. This does bring up a thought. I guess the only thing we gain over Spark is that Ballista is Rust based, with a better memory model, no Gargabe Collection overhead, and better serialization and deserialization, so it’s just faster?

I have noticed over the years that vendors and new tech likes to push the whole “I’m faster than everyone else” argument. But it only goes so far in the real world. We expect things to be fast. It needs to be fast AND offers something its competitors don’t … maybe better developer experience, ease of deployment, better APIs and interfaces, anything. People don’t like change generally, they need to have something that nudges them over the edge.

What am I looking for in Ballista?

I’m going to be looking for a few high-level things when trying out Ballista.

• How easy is it to install and set up a cluster?
• How easy is it to submit jobs to a cluster?
• How easy is it to use Datafame API?
• How easy is it to use the SQL API?
• How does performance compare to PySpark?
• How good or bad is the documentation?

I mainly just want to see if I have any trouble doing the basics and see if I feed any weak spots. One good indication of the maturity of a product is simple use it from scratch, see what kinda problems appear, how easy or not easy things are, and see if you can use the documentation. My hopes are high.

Setting up and installing a Ballista cluster.

Well, I hate to say it, but this is where I ran into my first trouble. It was hard and almost impossible to find the documentation for setting up a standalone Ballista cluster. Hard enough to find stuff about the Kubernetes deployment. Luckily after I about gave up, I did find some.

Call me old school, whatever, I like trying to install my own clusters from scratch. Why? Because it teaches you something about the systems you are working on. For example, many years ago when I was starting to get into Spark, I would set up and configure my own mini clusters using Linode. It helped me learn how the master would talk to nodes, how much memory to configure for a worker, and so on.

I want that same sort of very raw experience with Ballista. Sure, you can jump to just running something on your laptop, but I believe you’re leaving some important learnings on the table when you do that.

I’m going to use Linode and configure a small 3-node cluster manually.

So I have three 4 GB memory nodes, one scheduler, and two nodes running Ubuntu. The first thing I will do is harden all the servers, block all traffic incoming, then whitelist the ips for the different servers.  I highly recommend fail2ban and ufw on Linux machines, you use for projects.

Also, of course, I needed to install Rust  on the boxes, via

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

snap install rustup --classic
rustup install stable
rustup default stable

cargo install --locked ballista-executor

sudo apt install build-essential

sudo apt install protobuf-compiler