Do this, not that. Well, I’ve got my own list. With everyone jumping on the PySpark / Databricks / EMR / Glue / Whatever bandwagon I thought it was long overdue for a post on what to do, and not to do when working with Spark / PySpark. I take the pragmatic approach to working with Spark, it’s honestly very forgiving well and far into the 10s of TBs of data. Once you wander past that point things tend to get a little spicy if you don’t have it all dialed in. As with most things in life if you get a few things right, and of course don’t do some things, that will get you a long way, the same applies to Spark.
https://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.png00Danielhttps://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.pngDaniel2022-10-13 00:51:502022-10-13 20:55:05New to PySpark? Do this, not that.
Ever since playing with Great Expectations with Spark some time ago, I’ve been on the lookout for more Data Quality at-scale tools. The market still has a long way to go with these tools, not enough options, hard to use, and the typical Data Engineering travails. I came across soda-core recently, a self-proclaimed…
“Data reliability testing for SQL- and Spark- accesssible data.“
soda-core docs
Doing anything at scale, well … that’s usually the problem. Data Quality and Observability are topics were hear a lot about these days. The reality often doesn’t meet the expectations most of the time. Even Great Expectations, being awesome, can get complicated real quick-like. Let’s hope that soda-core pair with Spark can show us some real promise. Code available on GitHub.
https://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.png00Danielhttps://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.pngDaniel2022-09-14 21:33:012022-09-21 18:34:55Soda-Core. Data Quality at Scale.
I’ve always been a firm believer in using the right tool for the job. Sometimes I look at a piece of code … and ask … why? I mean just because you can do something doesn’t mean that you should. I see a lot of my job as someone who writes code … as not just my ability to write code, but the ability to reason about problems and design simple and elegant solutions that solve the problem at hand.
I try not to let my love of a tool, language, or package color my view of the world as it is. In fact, there is wisdom to be found in being critical of those languages and tools you love the most. Be aware of their shortcomings and failures. This leads to better software and architecture designs, and less complexity. Too often I’ve seen folks picking their tool of choice and then sticking with it till the bitter end, and it usually is bitter. There is more to life than writing obtuse Scala code that is illegible for some mundane task.
This sort of thing is a blight on everyone and every system. Now I must descend from my high horse and join the peasants on the dusty road of life. Today I want to look at some very common Data Engineering tasks, namely cloud storage, and what it is like to do such a thing with Golang, Rust, and Python. I will let you draw your own conclusions. Maybe. Code available on GitHub.
https://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.png00Danielhttps://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.pngDaniel2022-09-10 13:32:232022-09-11 15:20:44Working with Cloud Storage (s3). Golang vs Rust vs Python. Who shall emerge victorious?
Just when I think it cannot get more popular, it does. I have to admit, PySpark is probably the best thing that ever happened to Big Data. It made what was once a myth, approachable to the average person. No need for esoteric Java skills, no more MapReduce, just plain old Python. Another amazing thing about Spark in general, and by extension PySpark, is the sheer amount of out-of-the-box capabilities. I wanted to dedicate this post to a few amazing and wonderful features of PySpark that make Data Engineering fun and powerful.
It still seems like the wild west of Data Quality these days. Tools like Apache Deque are just too much for most folks, and Data Quality is still new enough to the scene as a serious thought topic that most tools haven’t matured that much, and companies dropping money on some tool is still a little suspect. I’ve probably heard more about Great Expectations as a DQ tool than most.
With the popularity of PySpark as a Big Data tool, and Great Expectations coming into its own, I’ve been meaning to dive into what it would actually look like to to use Great Expectations at scale and answer some simple questions. How easy is it to get up and running with Spark, what’s the path of least resistance to getting some basic Data Quality checks in place in a data pipeline.
https://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.png00Danielhttps://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.pngDaniel2022-06-02 22:46:402022-09-21 18:37:45Great Expectations with Databricks and Apache Spark. A Tale of Data Quality.
Probably one of the hardest hurdles to jump over when starting out in anything new, including Data Engineering and Data Pipelines, is knowing where to start. It always can be a little daunting. One aspect that can make or break any project, giving you the confidence to move forward like Sparticus to conquer, is having a good project template for your repository of code and logic that will encapsulate and present your code to others.
I’ve created a free and hopefully helpful Python blank GitHub project template that you can clone, change, and steal to your heart’s desire. I hope it will be helpful and set you going in the right direction for your next project.
Not going to lie, I do enjoy the vendor wars that this marketing craze called “The Modern Data Stack” has created. I like to keep just about everything in life at arm’s length. Kinda like the way you look at your crazy third cousin out of the corner of your eye at the family reunion. I mean it’s nice to have all these options to choose from these days when building data pipelines.
One tool I haven’t been able to poke the tires on yet is Prefect. It appears to be another data orchestration tool for Python, but we shall find out. I want this to be an introduction to Prefect, we shall just try it out and let the chips fall where they may.
https://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.png00Danielhttps://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.pngDaniel2022-04-30 21:09:032022-05-11 14:46:12Review of Prefect for Data Engineers
As the years drag by in Data Engineering, there are a few things that I have come to appreciate more and more. One of those topics that is close to number one on the list is complexity reduction. Today’s modern data stacks are filled to the brim with technologies and tools, full to the brim, and overflowing. So many tools with such wonderful features, sometimes all the magic comes with a downside. Complexity. Complexity can turn something wonderful into a nightmare.
Reducing (not avoiding) complexity seems to be one of the main tenets I work on these days when designing resilient, reliable, and repeatable data pipelines that can process terabytes of data. One of those tools is COPY INTO feature of Databricks + Delta Lake.
https://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.png00Danielhttps://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.pngDaniel2022-04-14 00:37:432022-05-11 14:45:48Reducing Complexity with Databricks + Delta Lake COPY INTO
Mmmm … Data Quality … it is a thing these days. I look forlornly back to the ancient days of SQL Server when nobody cared about such things. Alas, we live in a different world, where hundreds of terabytes of data are the norm, and Data Quality becomes a thing. I’ve been meaning to give Great Expectations a poke for like a year, but just haven’t had the time or inclination to do so, but times are changing, and so should I.
I’m not really planning on giving an in-depth guide to Data Quality with Great Expectations, what I’m more interested in are topics like, how easy is it to set up and use, what’s the overhead, what are the main features and concepts and are they easy to understand. I find this sort of review of Data Engineering tools to be more helpful than simply a regurgitation of the documentation.
https://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.png00Danielhttps://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.pngDaniel2022-03-17 15:54:252022-09-21 18:38:22Data Quality – Great Expectations for Data Engineers
It’s not often I yearn for the good old days of SQL Server, but I’ve had a few of those moments lately. Some things I miss, some I don’t, and it’s probably because I’m getting old and crusty, stuck in my ways, by permissioning is one of those topics where I think about the good old days. Data access control and permissions are topics that we all kinda ignore as not that important … until we actually are trying to do something with them. Then all of sudden we start complaining about complexity and why this isn’t easier. That’s a good way to describe my reaction to having to work on Databricks Access control.
But, I learned a few things and I think they will be helpful for someone. Read on for the basics of handling permissions and access control in Databricks and Delta Lake.
https://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.png00Danielhttps://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.pngDaniel2022-03-03 15:03:562022-03-05 20:26:50Databricks Access Control – The 3 Most Important Steps
