What I’ve Learned After A Decade Of Data Engineering
After 10 years of Data Engineering work, I think it’s time to hang up the proverbial hat and ride off into the sunset, never to be seen again. I wish. Everything has changed in 10 years, yet nothing has changed in 10 years, how is that even possible?
Sometimes I wonder if I’ve learned anything at all, maybe I’m just like the moras of Data Warehouses moldering out there in forgotten and beaten SQL Servers. The technology has shifted drastically under my feet, yet I’ve managed to keep my fingernails firmly sunk into the edge of the cliff of technical and personal obsolesce that seems intent on dragging me away to the purgatory of useless programmers and tools.
Putting into words the impossible.
Being no stranger to the interwebs and the legions of geniuses and savants that lie waiting in every corner for some unsuspecting word to escape my fingers and allowing their righteous anger to pore forth in a burning stream of hatred.
Some would say at this point I’ve come to enjoy it. Like some crazed addict shooting into my veins the vitriol of the world, using it as jet fuel to make my fingers pour forth more maddened technical takes sure to raise the ire of the mediocre Data Engineer.
The problem with trying to summarize 10 years of learning in any field, Data Engineering included, is that (is this the first thing I’ve learned?) no two Data Teams are alike. Therefore we can probably conclude that no two experiences would be the same, we all travel our own unique paths and are the sum of the foibles we commit along the way.
Lest I wax too strong and too long I’m going to plunge straight into the deep end without further ado. If I were sitting in front of someone who asked me what I’ve learned in doing Data Engineering work for 10 years, a bloody decade, this is what I would say, in no particular order.
If you’ve been working in data as long as I have, your mind would be as muddled as mine.
- The best Engineers and Programmers think long and hard before they start writing code.
- Being a mediocre programmer with better writing and communication skills will get you further, faster.
- Learn a language on top of SQL and Python to move from Mid to Senior+.
- Focus on DevOps and CI/CD skills … Bash, Docker, etc, these skills will come in handy.
- Think about the Big Picture more, and work on System Design and Architecture.
- Work on your craft outside of work hours, do personal coding projects, etc (this is where most learnings come from).
- Find people smarter than yourself to work with, in open source or your company.
- Learn time management early on.
- Learn how to mentor persons more junior than yourself.
- Learn to set aside your pride, and don’t take everything so personally.
- Try to divine what the new popular technology is, then learn it (think Databricks or Rust).
- Keep your PRs small and to the point.
- Write lots of tests.
- Learn to work with non-engineers, like Marketing, Product, etc.
- Move jobs more often than you think or want to, you will learn more and get paid more over the long haul.
- Must be a master of Data Modeling, both Big Data and RDBMS
- Understand Distributed Systems at a reasonable level, aka you know what NOT to do in Spark etc.
- Not learning DSA at some point will probably stunt your ability to move beyond a certain point. Be ok with that.
- Be known as the person who will “get it done” at work.
- Watch other people and your coworker’s careers and actions closely, and learn from their mistakes and victories.
- Find the “classic” books and read them. (Data Warehouse Toolkit, Pragmatic Programmer, etc.)
- Learn how to make tradeoffs when building and developing.
- Learn how to deliver and plan large and medium-sized projects on time.
Well, that’s all my feeble mind can muster at the moment, the memories of years gone past quickly fade, along with their lessons. I’m very curious about what others have learned after their years of mistakes and hard work, as I know each person is different.
At this point, if I had to look back and boil it down to a single thing, one of the strangest realizations that probably took the longest for me to learn was “code is not the end all be all, you can be an awesome programmer and a terrible engineer.”
A great engineer is not just a great coder, but a great human, capable of working around the edges of software.
Thank you so much, I’m an undergraduate majoring in Data Science in a third-word country. My understanding of this matter is almost 0 after all the years and on my way to crawling back to this hard to break in job. My coding skills are mediocre compared to people my age. Love to learn this lesson early on in life.
I’m only about 3 years in and this looks like a really good list. Although my first gig taught me something I want to share with other juniors, particularly those just starting out. Especially if you are remote! You are worth your seniors time. You deserve answers to questions that don’t have obvious answers, like architecture explanations. You are not expected to “figure it out”, at least not anywhere with any sophistication.
I made the error of muddling through and trying to explore my way to understanding our technical estate. Then I watched a 6 year engineer join our team and just make 30 minute meetings wherein people literally drew him a map. Just put time on their calendar and go ask them to show you around.
Hello Daniel, enjoyed reading your article.
I am a CS graduate with 20 years of association with software. But never I felt that I am part of cloud development or cybersecurity enhancement ( neither a hacker ) or data engineering or into AI.
I too feel the same way as you.