Nothing will raise the hackles on the backs of hairy and pale programmers who’ve been stuck in their mom’s basement for a decade like bringing up OOP (Object Oriented Programming), especially in the context of Python. It’s like two fattened calves prepared for slaughter, sharpen your knives, and take your place, it’s time to feast upon the boiling cauldron of emotions simmering away in the interwebs.
Of all the duties that Data Engineers take on during the regular humdrum of business and work, it’s usually filled with the same old, same old. Build new pipeline, update pipeline, new data model, fix bug, etc, etc. It’s never-ending. It’s a constant stream of data, new and old, spilling into our Data Warehouses and Lake Houses with a constant bubbling of some spring-time swollen stream.
What doesn’t happen that often is the infamous “Greenfield Project.” Otherwise known as new stuff … like brand new stuff. Like we don’t have this stuff and now we want this stuff.
I think it’s safe to say there are Greenfield Projects, and then there are Greenfield Projects. It can be hard to know what some people mean, or what their definition of new actually is. Probably just depends on the Data Team, but for the purposes of this discussion let’s assume it means something slightly different … aka you’re building an entirely new, end-to-end, Data Platform … and all that it entails.
This is indeed probably the most fun a Data Engineer can have, and the most stressful thing a Data Engineer can do. It’s both. You will learn the most, and cry the most. It will turn a milk toast Engineer into a real one. No hiding. It will either work or it won’t, and since your customers will be Engineers, you will find out right away what you got right, and what you got wrong.
When you’re building something brand new from scratch (yes I’ve done this a few times), it’s easy to get caught up in the details, but this is a bad idea. There is a mantra you should repeat to yourself in the beginning, to keep from getting stuck in the weeds.
“I must understand the big picture and put the peices together, do my due dillegence, there will be bumps in the road, don’t stress the details.”
The classic gotcha I see a lot of younger Engineers make is to think that “it’s all about the code.” They simply cannot get past “but how will we make this one thing work like this.” They get utterly consumed by the code.
Experienced Engineers understand that the code will always work itself out, Engineers WILL find solutions to problems, that’s what they get paid to do. So trying to pre-solve coding problems before they exist will ensure you focus on the wrong things and forget the important things.
At the same time, I fully believe that these projects are best done by Senior+ Engineers and not out-of-touch Architects.
Here is where to start when building Data Platforms from scratch. Of course, there is a lot of hidden detail here, but the point is to break up the building of brand new Data Platforms into manageable parts.
Let’s talk about this a little.
I know I’m beating the proverbial horse to death, but the first thing on the list is important, the order of operations in the above chart. Typically what the Engineers want to do is show up with whatever their bright and shiny solution is, before they even really have documented the needs and requirements. So annoying.
The funny part about these few steps is that they are probably the most neglected, heck I’m the same way, I just want to jump with both feet and do something fun. Making sure requirements are gathered, and understood, and that you’ve carefully examined the ins and outs of all the tech can be very tedious.
Who wants to read through mountains of documentation and make lists of this and that, double-checking and cross-checking features, etc? Well, I’m sure there are a few weirdos out there who do like it. Maybe I do a little. Don’t tell anyone.
So, if you’re just a normal human who likes to take shortcuts and refuses to do what you’re told, I would suggest at the very least if you’re going to skip this pre-work, that you should AT LEAST do the “Sketch out the rough design and POC if necessary.”
Drawing your new Data Platform out on a piece of paper, even the details will make you think about what you’ve missed, showing you visually what the system is supposed to look like, and bringing out any problem areas.
So what if you’ve never really built a Data Platform from scratch and you don’t even know what you don’t know, where to start, what to do, what to care about … I mean what are the baseline things a Data Platform should consist of and provide?
Never fear, I’ve stumbled and stubbed my toe enough to give you a little insight.
I would say at a minimum, a Data Platform should at least cover these five areas to be considered complete and be able to support a production-ready Data Team, a platform ready for all the problems headed its’ way.
I’m not going to dive into depth on each of these topics, I think merely naming them out loud should be enough for most Data Engineers to get on board with what a Data Platform should consist of. Each one is worthy of a book in it’s own right.
If you stop and think about each one of these, they are almost a project in themselves, as they should be. Building Data Platforms from scratch is not easy. Each piece connects to and integrates with the other ones, each decision you make on what to use in each area will affect the other areas, the complexity, and how well the Platform runs at scale when built.
AND, we haven’t really talked about the code itself and the specific business use cases that might drive certain decisions based on how your Platform needs to deal with “special” situations specific do your data. The reality is that each business usually has these sorts of caveats.
A question that comes up often … “How do I develop Production Level Databricks Pipelines?” Or maybe someone just has a feeling that using Notebooks all day long is expensive and ends up being an unreliable way to produce Databricks Spark + Delta Lake pipelines that run well … without error.
It isn’t really that hard and revolves around a few core ideas.
Watch below video for the full details.
Have you ever wondered about being explicit in your code vs being vague? I think about this a lot as I’m writing code on a daily basis. I’ve found I like being explicit and verbose when writing code, rather than being vague in what I’m doing most of the time.
A few years ago I wasn’t sure, who was going to win, Golang seemed to be popular, and still is for that matter. When I first wrote a little Golang (~2+ years ago) I was just trying to see what the hype was all about. The funny thing is, at the time, and today, it seems like the Golang syntax is much simpler than Rust, easier to learn and pick up by far.
I was recently confronted with an interesting conundrum when writing a complex data pipeline. It was an interesting problem that arose from my quest to reduce complexity in part of the design, which found itself creeping into another part, re-enforcing the classic idea of whether you can really make the complexity pea go away, or if you simply shuffle the pea somewhere else to hide it.
Being Data Analytics is a meat grinder, it’s the worst job ever. Horrible it is. It will crush you.
Back in March, I did a writeup and experiment called DuckDB vs Polars, Thunderdom, 16GB on 4GB machine challenge. The idea was to see if the two tools could process “larger than memory” datasets with lazy execution. Polars worked fine, DuckDB failed in spectacular fashion.
I also noted how many people had opened issues in GitHub about this very thing, but the issues were either ignored or closed. Someone on YouTube said some of these OOM issues were fixed in recent releases.
It’s probably what every single person wants to accomplish first after they’ve been writing code for a year professionally. How do I get to Senior Engineer? What skills do I need? “I am a good coder, give me the Senior Engineer title.”
Sadly, most Junior and Mid-Level Engineers think that being a Senior Engineer is all about coding, when in fact, it’s not. Being good at writing code is only the first step, the are many other way harder skills to learn to grow from Junior to Senior Engineer.
