What I’ve greatly feared has come to pass. I’ve come to love on of the most confusing parts of Python. AysncIO. It has this incredible ability for data engineers building pipelines in Python to take out so much wasted IO time. It saves money. It’s faster. People think you’re smarter than you are. Tutorials are one thing but implementing it in your complex code is typically mind bending and a test of your patience and self-worth.
Read more
Raise your hand if you’ve every used Dask? ……. Me either. With tools like Spark, I’ve only recently started seeing some articles and podcasts pop up around Dask. Written in Python and claiming to be a distributed data processing framework I figured it was about time to check it out. Suprisingly when reading up on the Dask website, it appears they don’t necessarily claim to be out to replace things like Spark. Time to kick the tires.
Read more
It doesn’t take long reading articles on Medium or Towards Data Science to become enamored with Machine Learning. Especially the people and companies who do it in “production.” I always read about the big picture, the fancy algorithms, the cloud computing, but you have that feeling there is something missing. It’s all the details that are missing. Where is the force behind it all, bringing everything together? I like to think it’s called Data Engineering, with some Dev Ops for good measure.
Read more