, ,

My Journey from Python to Scala – Part Deux

In Part 1 of my laborious journey from Python to Scala, I did some work with file operations, CSV files, and messing with the data. It took me a little longer then I expected to wrap my head around the Scala functional/object/immutable approach to software design. But, in the end if felt satisfying and I’m starting to be a convert. Scala makes you think a little harden then Python, is less forgiving, and requires more of you as the developer. In part deux, I figured the next topic to grapple with some simple retrieval of remote files and writing those files to disk. Also, I wanted to take a crack at Classes in Scala.

Read more
, , ,

Solving the Memory Hungry Pandas Concat Problem.

One of the greatest tools in Python is Pandas. It can read about any file format, gives you a nice data frame to play with, and provides many wonderful SQL like features for playing with data. The only problem is that Pandas is a terrible memory hog. Especially when it comes to concatenating groups of data/data frames together (stacking/combing data). Just google “pandas concat memory issues” and you will see what I mean. Basically what it comes down to is that Pandas becomes quickly useless for anyone wanting to work on “big data” unless you want to spend $$$ on some machines in the cloud with tons-o-ram.

Read more