, , ,

Apache Airflow for Data Engineers

On again, off again. I feel like that is the best way to describe Apache Airflow. It started out around 2014 at Airbnb and has been steadily gaining traction and usage ever since, albeit slowly. I still believe that Airflow is very underutilized in the data engineering community as a whole, most everyone has heard of it, but it’s usage seems to be sporadic at best. I’m going to talk about what makes Apache Airflow the perfect tool for any Data Engineer, and show you how you can use it to great effect while not committing to it completely.

Read more
, ,

Introduction to Postgres with Python

Python and Postgres, a match made in heaven.

If there was ever a match made in heaven, it’s using Python and Postgres together. They were made for each other. Both are fun and easy to use, addicting, both have so many surprises and hidden gems. Like Gandalf and Frodo, the two just go together. Today I want to go through the basics of interacting with Postgres using Python. In the beginning of my data career this was often a point of pain, even though it seems like it should be easy. Let’s hit on the basics and then a few of the not-so-obvious things I wish I would have known in the beginning.

Read more
, ,

Exploring ElasticSearch with Python

Exploring Elasticsearch with Python.

What’s Elasticsearch precious? I feel like Gollum when confronted by taters. Elasticsearch has been around for awhile now, based on Lucene, it’s become a well known name in the field of text and semi structured data storage, analysis and retrieve category. Even though it’s popular enough to get name recognition I’ve rarely run across it in the wild. We are going to dip our toes into Elasticsearch by working on a small project to store and search a book(s). It just give us enough simple problems to solve that by the end we should have at least a basic understanding of how to connect, store, and retrieve simple documents with Elasticsearch.

Read more
, ,

3 (Or More) Ways to Open a CSV in Python

Ah. What a classic. The one piece of code that I end up writing over and over again, you would think I would have stashed it away by now. Not going to lie I usually have to Google it, while thinking, is this the right way? Should I just open the csv file and iterate it? Should I import the csv module? Should I just use Pandas? Does it matter? Probably not.

Read more
, , ,

Thunderdome for Geospatial Tools in Python. It’s to the Death.

A fight to the death. A comparison of geo-spatial tools in Python. What’s easy and fast to use.

It’s a fight to the death people… that’s why it’s called Thunderdome. This will be no different. Last time we talked about the very basics of the strange world of geo-spatial tools for data engineering. The next most obvious thing do of course is to see what tool is the best. By best I mean what tools can be used to load and do simple manipulation of data in a fast and relatively simple manner.

Read more
, ,

Python Async File Operations – Juice Worth the Squeeze?

Async file operations in Python, juice worth the squeeze?

What I’ve greatly feared has come to pass. I’ve come to love on of the most confusing parts of Python. AysncIO. It has this incredible ability for data engineers building pipelines in Python to take out so much wasted IO time. It saves money. It’s faster. People think you’re smarter than you are. Tutorials are one thing but implementing it in your complex code is typically mind bending and a test of your patience and self-worth.

Read more
,

Data Engineering a Machine Learning Pipeline In Real Life.

The pipelines behind the machine learning.

It doesn’t take long reading articles on Medium or Towards Data Science to become enamored with Machine Learning. Especially the people and companies who do it in “production.” I always read about the big picture, the fancy algorithms, the cloud computing, but you have that feeling there is something missing. It’s all the details that are missing. Where is the force behind it all, bringing everything together? I like to think it’s called Data Engineering, with some Dev Ops for good measure.

Read more
, ,

Concurrently Download Large Files from GCS.

Waiting for large files to download is boring.

There is nothing more annoying than sitting around waiting for files to download. That was true while I was in high school staring at LimeWire, it’s still true today. Especially when you’re a data engineer who’s supposed to make data pipelines fast. You’re in luck! Yes, it is possible to download a large file from Google Cloud Storage (GCS) concurrently in Python. It took a little digging in Google’s terrible documentation for their Python cloud storage wrapper (hear my snarky-ness), but I found a diamond in the rough.

Read more
,

Data Engineering vs Data Science – Where’s The Love??

The Tech Fight of the Century, Data Engineering vs Data Science.

It seems like a never ending battle for supremacy. Articles about Data Science being the bee’s knees, then more articles about how Data Engineering holds up the world of Data Science like Atlas. Whenever I read something in one of these two categories on Medium or wherever, it just seems more like ego clash to me. It’s human nature to want to be the best, to be better, to feel like you are the person who really makes it all happen.

Read more
,

Thoughts on Distributed Data Pipelines – Spark vs Kubernetes

Data Pipelines – Spark vs Kubernetes, or both?

Data gets bigger and teams want to process data faster, what else can you do? There is only so much code tweaking you can do, threads, processes, asyncio, it’s only going to get you so far. At some point you have terabytes of data to process, and it requires a decision about some sort of distributed processing system.

In my experience I’ve mostly used two different distributed data processing systems in production, Spark and Kubernetes. To be honest the choice has always been obvious when to choose one over the other. The data usually dictates which system you choose. I’m sure there are super fans of each system who would argue there’s always a way to do any transform or process on each, but sometimes the point is, which system is setup to easily and quickly move the data from one point to another, and transform it as needed.

Read more