,

Google Fires Python. What Next?

What is going on? Is the world coming to an end? I thought Python was going to live forever. Well, apparently not at Google. Recently Google announced it was laying off its entire North American-based Python team that was supporting Google’s special needs with Python, in favor of cheaper offshore workers.

Apparently, some of these Engineers were GOAT-level employees.

Is that really that much of a surprise? Probably not. Ever since Bard, Google has been failing hard. They still make a ton of money but old Google has gone by-by. It’s a new Era of Corporate Google, and it’s here to stay.

Heck, you can still make bank working in Software at Google … but you better save some of those greens for when your name gets called.

,

Reaction to Data Engineering Survey for 2024

, , , , ,

Why did Golang lose to Rust for Data Engineering?

A few years ago I wasn’t sure, who was going to win, Golang seemed to be popular, and still is for that matter. When I first wrote a little Golang (~2+ years ago) I was just trying to see what the hype was all about. The funny thing is, at the time, and today, it seems like the Golang syntax is much simpler than Rust, easier to learn and pick up by far.

Read more

, ,

If:Else Logic and Complexity – Hiding the Pea.

I was recently confronted with an interesting conundrum when writing a complex data pipeline. It was an interesting problem that arose from my quest to reduce complexity in part of the design, which found itself creeping into another part, re-enforcing the classic idea of whether you can really make the complexity pea go away, or if you simply shuffle the pea somewhere else to hide it.

Read more

, ,

Data Analytics Suck! Worst Job Ever!

Being Data Analytics is a meat grinder, it’s the worst job ever. Horrible it is. It will crush you.

,

DuckDB Out Of Memory – Has it been fixed?

Back in March, I did a writeup and experiment called DuckDB vs Polars, Thunderdom, 16GB on 4GB machine challenge. The idea was to see if the two tools could process “larger than memory” datasets with lazy execution. Polars worked fine, DuckDB failed in spectacular fashion.

I also noted how many people had opened issues in GitHub about this very thing, but the issues were either ignored or closed. Someone on YouTube said some of these OOM issues were fixed in recent releases.

Read more

,

Transitioning to Senior Engineer

It’s probably what every single person wants to accomplish first after they’ve been writing code for a year professionally. How do I get to Senior Engineer? What skills do I need? “I am a good coder, give me the Senior Engineer title.”

Sadly, most Junior and Mid-Level Engineers think that being a Senior Engineer is all about coding, when in fact, it’s not. Being good at writing code is only the first step, the are many other way harder skills to learn to grow from Junior to Senior Engineer.

, ,

Databricks Doubles Cost. Reddit Explodes. I’m in Trouble!

I recently did a post on Linkedin and Reddit about Databricks removing Standard Tier and forcing folks into Unity Catalog. The post got big traction and blew up, more than I thought. Enough for the Databricks folk to hunt me down at work and tell me I’m naughty.

I will be writing a more in-depth post soon on Substack about the downsides of Vendor Lockin and how Data Teams should think about such things.

, , ,

Writing Apache Spark with Rust! Spark Connect Introduced.

I never thought I would live to see the day, it’s crazy. I’m not sure who’s idea it was to make it possible to write Apache Spark with Rust, Golang, or Python … but they are all genius.

As of Apache Spark 3.4 it is now possible to use Spark Connect … a thin API client on a Spark Cluster ontop of the DataFrame API.

You can now connect backend systems and code, using Rust or Golang etc, to a Spark Server and run commands and get results remotely. Simply amazing.  A new era of tools and products is going to be unleashed on us.  We are no longer chained to the JVM. The walls have been broken down. The future is bright.

,

How to JOIN datasets in Polars … compared to Pandas.

It’s been a while since I wrote about Polars on this blog, I’ve been remiss. Some time ago I wrote a very simple comparison of switching from Pandas to Polars, I didn’t put much real effort into it, yet it was popular, so this is my attempt at trying to expand on that topic a little.

Recently, while laying flat on back on my sunporch soaking up the vitamin D beating down on me, dreaming about code, which I always do, it struck me.

Read more