What the Marooned Ben Gunn Teaches us about Solo Data Engineers

I always envied Ben Gunn in Treasure Island a little bit. Alone all those years, digging up gold and treasure, hunting wild goat, and living in a nice little cave. Living off the land, king of his island, gone half mad, but somewhat still there. Happy to see other people, but always a little bit of a recluse … too many years alone. I think there is a lesson for us here, and it has to do with the solo Data Engineer. That lone ranger, out there in the data wilderness, surviving.

The Tale of Marooned Ben Gunn and the Solo Data Engineer

First, if you’ve never read Treasure Island then call your parents and tell them they have failed you, because they did. Second, the story is of pirates, treasure, shipwreck, trials, tribulations, and poor ole’ Ben Gunn. Marooned by his fellows on an island all by himself for years. Dug up buried treasure he could do nothing with, went half mad, but was very useful in the end.

I’m sure there are some life lessons in there for us. Especially for the marooned and Solo Data Engineer.

Ben Gunn and Solo Data Engineer have a lot in common.

• when you have only yourself to talk too, you get somewhat eccentric.
• when you’ve been alone too long, you become a recluse and it’s hard to recover.
• you yell things from the sidelines and people don’t know if they should believe you.
• when you get back to the real world, you squander your chance.

The Echo Chamber.

The reason why Ben Gunn went half mad was being all alone, with only your own thoughts, tends to be an echo chamber. The same is true for the Solo Data Engineer.

Seeing data engineering from the same perspective, all the time, doesn’t make us better at what we do, it makes us worse. Writing code the same way, the way we like it all the time, or using the same technologies over and over again won’t help us grow as data engineers. Having other engineers to question your solution, your code, point out certain things can be annoying, but very helpful in the long run.

• being asked why you wrote a piece of code a certain way makes you stop and think.
• seeing other ways to solve the same problem give us a broader view for the future.
• exploring different tech stacks may not solve the problem at hand, but might in the future.
• knowing other people will look at and question your code, makes you think harder about what you’re doing in the first place.
• growth comes through learning new things, different ideas and viewpoints of other engineers will supercharge that process.

Communication.

The other piece that Ben Gunn and the Solo Data Engineer are missing is the communication. The best engineers, the smartest, who are devoid of the ability to communicate effectively are throwing all their talent out the window.

There are multiple ways for data engineers to communicate, and they are key to long term success of that engineer and their code….

• communicating via code comments, documentation, and README’s ( if you work solo the chances of you doing these well are slim to none).
• communicating architecture and high level design and decisions via PR’s, meetings, and code reviews.
• learning communicate through disagreement ( every engineer think’s they are the smartest one).

Ben Gunn on his island all alone, with nothing but his own thoughts and no one to talk too, didn’t have good communication skills when other people showed up. This is the danger with the Solo Data Engineer, no learning to talk about ideas and code out-loud, not being to articulate architecture and technology decisions, and failing to learn out to communicate through disagreements without taking everything as a personal attack.

Musings

I generally think being alone in anything is probably not the best for us humans. Being challenged by other engineers, or even having to teach other engineers is going to make us grow and be better.

Thinking harder and communicating our high level architecture decisions, as well as how and why we wrote code a certain way is helpful, regardless of if we are “right” or “wrong.” The best engineers know it’s a journey and can learn from all people.

Try not to be a Solo Data Engineer, seek out community, the reddit data engineering community is a good place to start. See how other people solve problems, write code, work through ideas. Seek out others smarter than yourself, learn from others.