Do this, not that. Well, I’ve got my own list. With everyone jumping on the PySpark / Databricks / EMR / Glue / Whatever bandwagon I thought it was long overdue for a post on what to do, and not to do when working with Spark / PySpark. I take the pragmatic approach to working with Spark, it’s honestly very forgiving well and far into the 10s of TBs of data. Once you wander past that point things tend to get a little spicy if you don’t have it all dialed in. As with most things in life if you get a few things right, and of course don’t do some things, that will get you a long way, the same applies to Spark.
https://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.png00Danielhttps://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.pngDaniel2022-10-13 00:51:502022-10-13 20:55:05New to PySpark? Do this, not that.
Sometimes I feel like I’ve been doing this too long, life gets busy, and I don’t have much to say … but here I am 5 years later. I’m still making people mad and making a fool of myself, some things never change. This will probably be short and sweet. I will cover the top 10 most popular blog posts from those 5 years, what the traffic has looked like over time, and what I’ve learned from writing blogs for so long, the good, the bad, and the ugly.
https://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.png00Danielhttps://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.pngDaniel2022-10-07 14:33:222022-10-07 14:33:435 Years of Blogging – Most Popular Articles, Traffic Stats, and other Thoughts.
So, you’ve heard about dbt have you. I honestly can’t decide if it’s here to stay or not, probably is, enough folks are using it, and preaching about it. I personally have always been a little skeptical of dbt, not because it can’t do what it says it can do, it can, but because I’m old and bitter from my many years of Data Engineering, and I always see the problems in things.
But, I will let you judge that for yourself. Today I want to give a brief overview of dbt, kick the tires, muse about its features, and most importantly, look at dbt from a Data Engineering perspective, ferret out the good, the bad, and the ugly. I will try my best to be nice but don’t count on it. Code is on GitHub.
https://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.png00Danielhttps://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.pngDaniel2022-10-04 09:10:242022-10-04 14:16:07Introduction to dbt … for Data Engineers.
When I think back many moons ago, to when I started in Data Engineering world … even though it went by many different names back in the olden days … I didn’t know what I didn’t know. All those years ago Kimball’s Data Warehouse Toolkit was probably the only resource really available at the time that touched on the general concepts that most “Data Engineers” at the time were working on. The field has come a long way since those days and changed for the better, it’s less often you see classic Data Warehouses running on legacy SQL Servers, with stored procedures with hundreds and thousands of lines of SQL code.
That had me thinking about designing data load patterns in the Modern Data Stack. I want to talk about general data loading patterns, how to design your data pipelines, at a high level, and the basic principles and practices that apply to 99% of all the transformations and data loads done by most Data Engineers.
https://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.png00Danielhttps://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.pngDaniel2022-09-27 23:10:302022-09-29 16:39:57Introduction to Designing Data Load Patterns.
I’ve been thinking more about the topic of ML and MLOps lately. To me, it seems like the buzz has quieted down over the last few years about ML and MLOps, at least somewhat, in favor of other topics like Data Quality, Data Lakes, Data Contracts, and the like. I’ve been wondering why this is the case and comparing my experience over the last few years of working in, on, and around ML pipelines and systems. I’ve seen ML done at companies with a few thousand employees, and with a handful of employees. The problems and hurdles at the same across the board, and mostly everyone is not very good at it.
https://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.png00Danielhttps://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.pngDaniel2022-09-26 21:35:562022-09-27 14:33:16Machine Learning from the viewpoint of an average Data Engineer.
I’m not sure if DataFrames in Golang were created by Gandalf or by Saruman, it is still unclear to me. I mean, if I want a DataFrame that bad … why not just use something normal like Python, Spark, or pretty much anything else but Golang. But, I mean if Rust gets DataFusion, then Golang can’t be left out to dry, can it!? I mean I guess if you’re hardcore Golang and nothing else will do, and you’re playing around with CSV files, then maybe? Seems like kind of a stretch. But, I have a hard time saying no to Golang, it’s just so much fun. Kinda like when Gandalf told them little hobbits and dwarfs to not stray from the path going through Fangorn Forest, those little buggers did it anyways. Code available on GitHub.
Best practices are always a touchy subject, I’m going to forget someone’s pet best practice, I can already feel it. I’ve always been a firm believer in the basics, keeping things simple. I also ascribe to the 80/20 rules, and I don’t think Data Engineering is any different in that respect. Learning to do a few things well, in the long run will probably solve most of your major problems encountered in data teams and architectures. Today I want to give you 8 Data Engineering best practices to hopefully give you some food for thought at least.
https://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.png00Danielhttps://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.pngDaniel2022-09-21 18:33:292022-09-21 18:33:588 Data Engineering Best Practices
Ever since playing with Great Expectations with Spark some time ago, I’ve been on the lookout for more Data Quality at-scale tools. The market still has a long way to go with these tools, not enough options, hard to use, and the typical Data Engineering travails. I came across soda-core recently, a self-proclaimed…
“Data reliability testing for SQL- and Spark- accesssible data.“
soda-core docs
Doing anything at scale, well … that’s usually the problem. Data Quality and Observability are topics were hear a lot about these days. The reality often doesn’t meet the expectations most of the time. Even Great Expectations, being awesome, can get complicated real quick-like. Let’s hope that soda-core pair with Spark can show us some real promise. Code available on GitHub.
https://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.png00Danielhttps://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.pngDaniel2022-09-14 21:33:012022-09-21 18:34:55Soda-Core. Data Quality at Scale.
I think it’s funny that DataFrames are so popular these days, I mean for good reason. They are a wonderful and intuitive way to work with and on datasets. Pandas … the nemesis of all Data Engineers and the lover of Data Scientists. Apache Spark is really the beast that brought DataFrames to the masses. Even those little buggers over at Apache Beam give you DataFrames.
Of course, when anything gets popular, you start getting little things that start to pick and peck at the heels. I would probably say that is what DataFusion with Rust seems to be. Seems more like a contender against Pandas rather than Spark to me. I guess if you’re just using Spark locally or on a single node, sure you could consider using DataFusion. Code available on GitHub.
https://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.png00Danielhttps://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.pngDaniel2022-09-11 15:50:562022-09-12 14:16:37DataFusion courtesy of Rust, vs Spark. Performance and other thoughts.
I’ve always been a firm believer in using the right tool for the job. Sometimes I look at a piece of code … and ask … why? I mean just because you can do something doesn’t mean that you should. I see a lot of my job as someone who writes code … as not just my ability to write code, but the ability to reason about problems and design simple and elegant solutions that solve the problem at hand.
I try not to let my love of a tool, language, or package color my view of the world as it is. In fact, there is wisdom to be found in being critical of those languages and tools you love the most. Be aware of their shortcomings and failures. This leads to better software and architecture designs, and less complexity. Too often I’ve seen folks picking their tool of choice and then sticking with it till the bitter end, and it usually is bitter. There is more to life than writing obtuse Scala code that is illegible for some mundane task.
This sort of thing is a blight on everyone and every system. Now I must descend from my high horse and join the peasants on the dusty road of life. Today I want to look at some very common Data Engineering tasks, namely cloud storage, and what it is like to do such a thing with Golang, Rust, and Python. I will let you draw your own conclusions. Maybe. Code available on GitHub.
https://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.png00Danielhttps://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.pngDaniel2022-09-10 13:32:232022-09-11 15:20:44Working with Cloud Storage (s3). Golang vs Rust vs Python. Who shall emerge victorious?
