, , , ,

DuckDB … reading from s3 … with AWS Credentials and more.

In my never-ending quest to plumb the most boring depths of every single data tool on the market, I found myself annoyed when recently using DuckDB for a benchmark that was reading parquet files from s3. What was not clear, or easy, was trying to figure out how DuckDB would LIKE to read default AWS […]

Read more
, ,

Testing DuckDB’s Large Than Memory Processing Capabilities.

I am a glutton for punishment, a harbinger of tidings, a storm crow, a prophet of the data land, my sole purpose is to plumb the depths of the tools we use every day in Data Engineering. I find the good, the bad, the ugly, and splay them out before you, string ’em up and […]

Read more
, ,

Skip Lines of CSV files with DuckDB and Polars

There are some things you don’t need until you need them. I ran into that situation recently with needing to process some CSV / Flatfiles on short notice. At first, it appeared to be easy, but then I realized, as usual, there was a little monkey wrench thrown into the middle of it. It is […]

Read more

What is the WordPress drama about?

I figured a few of us might need the WordPress drama explained like we are 5. So, here you go. WordPress is the GOAT of internet website builders WordPress was founded by Matt Mullenweg With much of the internet running on WordPress … hosting WordPress is of course … lucrative and a big business. The […]

Read more
, ,

How to make the PEFECT Pull Request (PR)

Is there anything worse than the PR process (Pull Request) at most companies? Probably not. It’s the dreaded 600-pound gorilla in the room that no one wants to talk about. Everyone hates it, everyone has to do it. But, it doesn’t have to be like that. There are a few tried and true ways to […]

Read more
, , ,

The Death of the Data Warehouse, replaced by the Lake House. Or Has It?

This is an interesting one indeed, it’s one that teases and puzzles the brain to no end. Has the Data Warehouse finally died, has that unruly upstart the Lake House finally taken its place atop the seething mass of data we call home? Can we say that after all these decades the Data Warehouse Toolkit […]

Read more
, ,

Hosted (SaaS) vs DIY Data Tools

I’ve been hacking around with tools and programming since Perl was a thing. I’ve worked the gambit of Data Platforms from large organizations to tiny startups, and all those in between. I’ve worked on Data Platforms that dropped ungodly amounts of money on SAP products, and places where we would build our own massive data […]

Read more
, , , ,

AI (LLMs) and Software Engineering (Writing Code)

I recently wrote on my Substack (Data Engineering Central) about how I used the new OpenAI o1 model to do some basic Data Engineering tasks surrounding PostgreSQL. It did ok. I’ve also been using CoPilot and ChatGPT for over a year now to assist me with my daily code that I have to write for […]

Read more
, ,

Data Modeling in the Brave New Lakehouse World

It is a Brave New World out there these days. The new tools and features come out faster than your mom on Sunday morning getting you ready for church. The same goes for the context and advice being produced on a myriad of platforms, the ole’ Like and Subscribe, and all that bit. It does […]

Read more
, ,

The 3 Types of Data Engineers.

Did you know there are only 3 types of Data Engineers? It’s true. I hope you are the right one.

Read more