AWS Lambdas – Python vs Rust. Performance and Cost Savings.

Save money, save money!! Hear Hear! Someone on Linkedin recently brought up the point that companies could save gobs of money by swapping out AWS Python lambdas for Rust ones. While it raised the ire of many a Python Data Engineer, I thought it sounded like a great idea. At least it’s an excuse to play with Rust, and I will take all those I can get. It does seem like an easy and obvious step to take in this age of cost-cutting that has come down on us all like that thick blanket of fog on a cool spring morning.

I can personally attest to the fact that I’ve written a number of Python AWS lambdas that are doing a non-trivial amount of data processing, currently running in Production and being triggered many times a day. Today, I’m going to reproduce both a Python and Rust lambda running on my personal AWS account doing pretty much the same exact work. Let’s see what the difference actually is in performance and see if it’s possible to find some cost savings.

What’s a real-life Data Engineering use of AWS lambda?

Rather consume this content as a video? Check out the corresponding video on YouTube.

So what is the use case for our AWS lambda we would be building in both Python and Rust? It’s going to be processing gzip flat-files from fixed-width to tab-delimited, with s3 as the source and s3 as the destination. Say we daily receive the bain of many Data Engineers fixed-width compressed flat files and we want to covert them to a more standard tab-delimited versions so we can have a common file type prior to ingestion into some Data Lake.

An AWS lambda is the perfect tool for this right? Simply trigger when a file hits s3, kick off a lambda, covert the file to what we want, and deposit the result back into another s3 location.

So to do this project, the first thing we are going to need is some flat files for testing, one fixed-width delimited and the other tab-delimited. We will use the Backblaze hard drive free dataset, taking one of those files and creating the two we need.

All this code is available, including the test file, on GitHub. If you are not familiar with fix-width files, just Google it. The data points are all found between an index range, so there is no “delimiter” for these files. It makes them a big pain to deal with, especially if downstream systems like Spark are supposed to ingest them. Very annoying.

Here is a fixed-width delimited file I made from the above dataset.

Let’s start with a 0index and define the fixed location of each column … as we will need this for both our Python and Rust lambda’s to parse the file.

start-end locations … (technically the location stops before the last number).

• date -> 0:15
• serial_number -> 15:36
• model -> 36:79
• capacity_bytes -> 79:98
• failure -> 98:109

Python lambda.

So, lambdas. They are pretty easy to work with on AWS, if you use the provided documentation you don’t really have much work to do besides swap out your code into the provided template(s).  At a high level pretty much every lambda you design, unless you change it, by default uses a function called `lambda_handler()` as the single entry point into the lambda. Aka that is what your lambda is going to call when it triggers.

Our AWS lambdas are going to be triggered when our fixed-width file hits s3. Our lambdas are going to receive a message into the lambda_handler function that is going to contain the unique s3 remote uri of the file that just hit the bucket. That means both our Python and Rust lambdas are going probably to follow the same general workflow.

• get the s3 uri of the fixed-width file.
• download the file.
• convert fixed-width to tab-delimited.
• write the new file back out to s3.

Let’s get cracking. All code is available on GitHub. You can use the aws cli to do all this, but for ease I just created the empty lambda’s via the UI.

Python lambda to convert a fixed-width to a tab-delimited file, trigger on the s3 bucket to launch the lambda, and run the conversion. Don’t forget, the code is available on GitHub.

Works like a charm to convert the fixed-width file. One thing to note is how simple and straightforward the code is, easy to reason about and debug. Sure, people on the internet say that Rust in the long term has no more cognitive and developer burden than any other language, but we all know that is crap.

Readability affects your codebase. Python’s readability is better than most, and that’s why people use it. So, when we drop the fixed-with gzipped file into our s3 bucket, does the lambda work? Of course.

Here are some logs for AWS. Performance, aka runtime, and the amount of memory used.

*******UPDATE********

Due to numerous complaints here is the updated runtime for Python using LESS memory, 200MB max, instead of 350MB. This is closer to what was allocated to the Rust lambda. This of course made Python even slower.

REPORT RequestId: 63aed740-42dc-49a1-92e8-e41349bcdfdd Duration: 15698.93 ms Billed Duration: 15699 ms Memory Size: 200 MB Max Memory Used: 140 MB Init Duration: 260.43 ms

******UPDATE***********

REPORT RequestId: d8a7cae5-fb99-4e81-bc1a-fbb8a4fd646e Duration: 8361.76 ms Billed Duration: 8362 ms Memory Size: 350 MB Max Memory Used: 139 MB

Yikes, that’s about 8.362  15.69 seconds! Slow Slow! Sure Python might not be perfect, but that’s life. There are 230,000records in this file, which are not very many. Most production files are much bigger. So you can imagine the runtimes on a few million records.

Let’s see what Rust can do.

Rust lambda.

So, I’m going to be looking for a few things here. I think it’s obvious that Rust is going to be faster … that’s a given. But how much faster, also will the memory usage be reduced if we try to write the Rust in the same way? I also want to know how much code in Rust this is going to take. How complex it will be.

The reality is that humans are human, if the Rust code is overly burdensome to write, Data Engineerings just won’t do it, regardless of the cost savings. They will just use Python and move on with life.

It’s one thing to re-write Python over to Rust, but in the real world, you have to have developers troubleshoot, debug, and write that code. If it’s twice the amount of code, sure it’s cheaper to run because it’s faster … but it might not be cheaper to maintain over time! Here are some Rust docs for lambda.

What I did was just copy the example lambda given in most of the AWS docs and ripped out the code, replacing it with mine.

Here is my ever-so-sad Rust code.  Here are some things of note, use cargo-lambda to create and build the binary bootstrap.zip that you will need to actually create our AWS Lambda. Something like cargo lambda build –release –output-format zip

I promise someday my Rust will get better.

So of course my Rust is verbose and has more lines of code than Python, but that’s pretty much to be expected. Especially when I am writing it, only a few months into Rust. Honestly, though, the Rust isn’t really that much more complicated, it’s just the static typing that makes it look more complicated.

What about speed?

REPORT RequestId: ccf34808-00ae-47cb-81ef-3dc33d5a483e Duration: 6452.89 ms Billed Duration: 6491 ms Memory Size: 128 MB Max Memory Used: 85 MB Init Duration: 38.03 ms

The Rust lambda is faster, clocking in around 6.452 seconds, vs Python’s 8.362 15.69, and uses less memory with 85MB vs Python’s 139MB. All of which adds up to cost savings. I do have to say, I thought the Rust lambda would run much faster. I mean, there is a high probability that my Rust is probably written not so well.

Rust is ONLY %60 percent faster on the runtime but uses %40 less memory as well. Let’s be honest, that is me a terrible Rust programmer simply migrating Python code to Rust, not even really trying. If you are running a large number of lambdas doing similar work in production, of course, you are going to save a noticeable amount of money.