, , ,

Writing Apache Spark with Rust! Spark Connect Introduced.

I never thought I would live to see the day, it’s crazy. I’m not sure who’s idea it was to make it possible to write Apache Spark with Rust, Golang, or Python … but they are all genius.

As of Apache Spark 3.4 it is now possible to use Spark Connect … a thin API client on a Spark Cluster ontop of the DataFrame API.

You can now connect backend systems and code, using Rust or Golang etc, to a Spark Server and run commands and get results remotely. Simply amazing.  A new era of tools and products is going to be unleashed on us.  We are no longer chained to the JVM. The walls have been broken down. The future is bright.

, , ,

Building Databricks Data Pipelines 101

Have you ever wondered at a high level what it’s like to build production-level data pipelines on Databricks? What does it look like, what tools do you use?

, ,

How To Build and Open Source PYPI Python Package

Ever wondered how to build and end-to-end project for an Open Source Python Package that gets published to PYPI? I built out lakescuman open-source package to help with Databricks Unity Catalog Delta Lake tables querying with Polars, DuckDB, or PyArrow. https://github.com/danielbeach/lakescum

, ,

Is Devin Going To Take My Software Engineering Job?

Unless you’ve been hiding a rock you’ve probably heard the hubbub over Devin the new AI Software Engineer that is going to take your job.

While this is a genius piece of marketing … it’s a bunch of crud.

Never fear, you are in no more danger of losing your job in Software than when ChatGPT and CoPilot hit the market. In fact, the opposite is true, there will be more Software jobs than ever, not less.

AI tools might look pretty in a video, but in reality, by those who use them, the are very bad at programming still and don’t do a very good job. Most of what a Senior Engineer does that makes them Senior is not that spitting out of gobs of code. Any dingdong can do that.

, , ,

DuckDB has MAJOR Problems! OOM Errors.

I recently did a challenge. The results were clear. DuckDB CANNOT handle larger-than-memory datasets. OOM Errors.  See link below for more details.

… DuckDB vs Polars – Thunderdome. 16GB on 4GB machine Challenge. 

 

,

Never Put Databricks Notebooks in Production

Recently an Architecture at Databricks recommended people use Notebooks for Production workloads. Very bad and horrible idea. Very expensive compute for most people (All Purpose Clusters) and it leads to horrible development practices. It set off a firestorm on Linkedin when I commented people SHOULD NOT follow this advice.

Read here and here

 

 

, ,

The Best Piece of Software Engineering Advice

You probably think this is another internet clickbait title uh? Just trying to get you to clickty clickty and sell you some Google Ads. Two problems. I don’t have Google Ads, and I know a small percentage of people will actually listen to this advice. Whatever. There is a reason some developers struggle to move past the Senior role.

Instead of making you scroll to the bottom to get what you can for, I’m going to give it to you off the top. How nice of me.

Read more

, ,

Why I Love Rust, but Deploy Python

I’m not sure if others have this same problem, maybe they are lucky, they get to build in their favorite language 24/7, it’s their tool of choice. I feel like I have a great burden to bear, a heavy one. I love to write Rust … but I deploy Python. Even when I know I could write Rust … Python gets deployed.

Read more

,

New SQL Practice Problems

New SQL Practice Problems

I’m trying something new. I get a lot of questions from folks about getting into the Data Engineering space, how to get better, grow, learn, etc.

So I came up with a solution. SQL Practice Problems.

Read more

, ,

The Abstraction Problem – A Great Evil

There is a great evil Spirit that is haunting the streets of code in the land of programmers. It’s a Spirit of obfuscation and twisting things into what they are not. The Spirit wanders around on the loose looking for someone, and it finds ready victims among the ranks of new programmers and the innocent young minds in University. It also finds a few old wizened souls that have been lost wandering for decades around the halls of some musty Fortune 500 company.

It’s the Abstraction Problem.

Read more