, ,

Data Analytics Suck! Worst Job Ever!

Being Data Analytics is a meat grinder, it’s the worst job ever. Horrible it is. It will crush you.

Read more
,

DuckDB Out Of Memory – Has it been fixed?

Back in March, I did a writeup and experiment called DuckDB vs Polars, Thunderdom, 16GB on 4GB machine challenge. The idea was to see if the two tools could process “larger than memory” datasets with lazy execution. Polars worked fine, DuckDB failed in spectacular fashion. I also noted how many people had opened issues in […]

Read more
,

Transitioning to Senior Engineer

It’s probably what every single person wants to accomplish first after they’ve been writing code for a year professionally. How do I get to Senior Engineer? What skills do I need? “I am a good coder, give me the Senior Engineer title.” Sadly, most Junior and Mid-Level Engineers think that being a Senior Engineer is […]

Read more
, ,

Databricks Doubles Cost. Reddit Explodes. I’m in Trouble!

I recently did a post on Linkedin and Reddit about Databricks removing Standard Tier and forcing folks into Unity Catalog. The post got big traction and blew up, more than I thought. Enough for the Databricks folk to hunt me down at work and tell me I’m naughty. I will be writing a more in-depth […]

Read more
, , ,

Writing Apache Spark with Rust! Spark Connect Introduced.

I never thought I would live to see the day, it’s crazy. I’m not sure who’s idea it was to make it possible to write Apache Spark with Rust, Golang, or Python … but they are all genius. As of Apache Spark 3.4 it is now possible to use Spark Connect … a thin API […]

Read more
,

How to JOIN datasets in Polars … compared to Pandas.

It’s been a while since I wrote about Polars on this blog, I’ve been remiss. Some time ago I wrote a very simple comparison of switching from Pandas to Polars, I didn’t put much real effort into it, yet it was popular, so this is my attempt at trying to expand on that topic a […]

Read more
, , ,

Building Databricks Data Pipelines 101

Have you ever wondered at a high level what it’s like to build production-level data pipelines on Databricks? What does it look like, what tools do you use?

Read more
, ,

How To Build and Open Source PYPI Python Package

Ever wondered how to build and end-to-end project for an Open Source Python Package that gets published to PYPI? I built out lakescuman open-source package to help with Databricks Unity Catalog Delta Lake tables querying with Polars, DuckDB, or PyArrow. https://github.com/danielbeach/lakescum

Read more
,

The Path To Senior Engineer

Want to know how to grow to the Senior Engineering position? Take a look.

Read more
,

Don’t Be So Smart

Most Software Engineers think of themselves as too smart. They think they are the best and brightest coder alive or that has ever lived. Doing so, they stunt themselves from becoming Senior Engineers and become hard to work with, the nightmare of the PR process. You don’t need to be the smartest person in the […]

Read more