Have you ever wondered at a high level what it’s like to build production-level data pipelines on Databricks? What does it look like, what tools do you use?
Ever wondered how to build and end-to-end project for an Open Source Python Package that gets published to PYPI? I built out lakescuman open-source package to help with Databricks Unity Catalog Delta Lake tables querying with Polars, DuckDB, or PyArrow. https://github.com/danielbeach/lakescum
Want to know how to grow to the Senior Engineering position? Take a look.
Most Software Engineers think of themselves as too smart. They think they are the best and brightest coder alive or that has ever lived. Doing so, they stunt themselves from becoming Senior Engineers and become hard to work with, the nightmare of the PR process.
You don’t need to be the smartest person in the room.
When you’ve been data modeling as long as I have, it gets to be the same old … same old.
People make data modeling harder than it has to be. There is a lot of jargon that gets thrown around … third-normal-form, OLAP, OLTP … I give you the 3-4 basics that are at the core of data modeling.
Unless you’ve been hiding a rock you’ve probably heard the hubbub over Devin the new AI Software Engineer that is going to take your job.
While this is a genius piece of marketing … it’s a bunch of crud.
Never fear, you are in no more danger of losing your job in Software than when ChatGPT and CoPilot hit the market. In fact, the opposite is true, there will be more Software jobs than ever, not less.
AI tools might look pretty in a video, but in reality, by those who use them, the are very bad at programming still and don’t do a very good job. Most of what a Senior Engineer does that makes them Senior is not that spitting out of gobs of code. Any dingdong can do that.
I recently did a challenge. The results were clear. DuckDB CANNOT handle larger-than-memory datasets. OOM Errors. See link below for more details.
… DuckDB vs Polars – Thunderdome. 16GB on 4GB machine Challenge.
Recently an Architecture at Databricks recommended people use Notebooks for Production workloads. Very bad and horrible idea. Very expensive compute for most people (All Purpose Clusters) and it leads to horrible development practices. It set off a firestorm on Linkedin when I commented people SHOULD NOT follow this advice.
You probably think this is another internet clickbait title uh? Just trying to get you to clickty clickty and sell you some Google Ads. Two problems. I don’t have Google Ads, and I know a small percentage of people will actually listen to this advice. Whatever. There is a reason some developers struggle to move past the Senior role.
Instead of making you scroll to the bottom to get what you can for, I’m going to give it to you off the top. How nice of me.