, ,

Thoughts on HTTP and JSON with Golang. And other Headaches.

Photo by Nik Shuliahin 💛💙 on Unsplash

I’ve been playing with Golang off and on for a few weeks, when I find the time, which is every few weeks between kids and fishing. I have become a little bit of a fan, wishing for more projects to take on with Go. It seems like a fairly straightforward language to pick up, the learning curve isn’t that bad, and it’s fast and powerful. I’ve found it a little more intuitive than Scala for example. I mean don’t get me wrong, nothing will take the place of Python in my life, but there’s always room for one more.

That being said, “But I have this against you…” when it comes to Go, and it has to do with JSON. All code is on GitHub.

More thoughts on Golang.

I’ve written recently with my thoughts on Go as a language for Data Engineering here. I really enjoyed using it, it’s a fast language with what appears to be not that bad of a learning curve. A few things I liked about Golang …

  • easy and intuitive to define functions.
  • easy to incorporate error handling.
  • simple and familiar data types
  • fast

I’ve been wanting to extend my poking at Goland beyond files and CSV processing so decided on the next most obvious and easy Data Engineering tasks.

  • HTTP (aka API response ingestion)
  • JSON

HTTP / API ingestion with Golang

Photo by Barn Images on Unsplash

Doing HTTP calls to API’s to grab data is probably one of the OG classic Data Engineering tasks. Especially if you are learning Python it’s usually one of the first skills you build. I’ve found these same HTTP workings to be utter agony in many other languages, so I’ve been curious if the same thing is true for Go or not.

I was pleasantly surprised by how easy it was to ingest an API with Golang. For my example, I decided to use the free Nasdaq API to pull some GDP and Oil pricing data. I highly suggest setting up a free account and using it for some data in your personal projects, lots of good fun to be had.

Let’s walk through this example project while talking about HTTP with Golang, and then later JSON parsing.

After doing the simple imports, I wrote an easy main function that set out the uris for the API’s I want to hit, and the function to grab each API response.

Easy peasy so far, let’s take a look at the central function grabbing the API responses. If I was sleepy and it was early in the morning I would have thought I was writing Python. Not going to lie, what a breath of fresh air. To get the speed and agility of a language like Golang, and be able to make a http call so easily, is hard to beat.

I mean really … http.Get(url), that’s some easy stuff. I love being able to call out any Errors with the resp, error := notation. Honestly, working with errors like this in Golang seems to be extremely fluid and brings a new kind of thought process to the forefront when writing code, something that doesn’t happen as easily with Python for example. It seems to force good behavior on the developer’s part.

Also, note the use of defer … defer resp.Body.Close()

“A defer statement defers the execution of a function until the surrounding function returns.”

Golang docs

In the end the function simply takes a String that is the uri and returns some bytes that is the response of the call to the API(s) endpoint.

This is where the fun stopped. The next part, messing with the JSON returned by the API turned out to be painful for me.

JSON with Golang. Yikes.

As I mentioned, this is where things got a little strange for me, as someone new to Golang. Of course the API response(s) that my HTTP methods were returning are all JSON formatted, as one would expect. What I didn’t expect was for the pain that was to follow for me to unwind that JSON with my Go.

You can see the two methods I wrote below to get the Oil and GDP responses. Sure they look simple enough on the surface, right? A simple call to json.Unmarshal(body, ...). But hold up, you can see that I set some variable var data Oil for example, and the data gets passed into the Unmarshal.

What is this var data Oil or var GrossDProduct you ask? Glad you asked, it’s a pain is what it is.

What is the pain? It’s the entire struct of those JSON responses, in their entirety. This was enough to make me pull my hair out.

I’m probably spoiled because of working with JSON with Python for example, is so easy. I was not expecting to have to make a complete struct that encapsulated the entire response including names and data types etc. In all fairness, again, this makes you actually understand what is happening and the data your working with “earlier” in the development process. But, If all I care about is a few data points from some very large API response … I guess I get to do a lot of work for nothing.

Musing on Golang with HTTP and JSON

I’m still a big fan of Golang after trying out a few more tasks like HTTP and JSON. The language is intuitive to write, and seems to enforce good programming best practices, especially around error handling. HTTP was so simple and it was just a breath of fresh air, and to get that simplicity with a super-fast language like Go was a real treat.

I was a little miffed about the JSON parsing with Golang, I mean I don’t know what I’m doing, but I could find no other way around that problem than to make a struct for the entire JSON response from the API. This had both good and bad side effects. It would defiantly be easier and faster to do the same task in Python, but of course, forcing yourself to understand the response is better in the long term.

Looking forward to continuing my journey through Golang land and using it more for Data Engineering tasks.