Embracing mistakes: lessons for developers

Why Developers shouldn’t be ashamed of their errors.

As developers, we often hold ourselves to impossibly high standards.
We strive for perfection in our code, and when we inevitably make mistakes, it can be easy to fall into a spiral of self-doubt and self-criticism.
But truth is, everybody makes mistakes.

It’s a natural part of the development process, and it’s how we learn and grow as professionals.
In fact, even the biggest and most well-known companies in the world have made massive mistakes that have resulted in costly bugs and downtime.

“ A human mistake was a collective failure, not an individual one. ”

The aim of this article is to examine several typical scenarios and amusing glitches that I have recently come across in our industry.

To begin with, I will disclose some typical blunders I have made in my career. Afterward, we will investigate together certain problems encountered by skilled developers at prominent companies.

Through this exploration, I hope to help you see that making mistakes is not a reflection of your abilities as a developer, but rather a normal part of the process.

So, let’s take a deep breath, relax, and remember that we’re all in this together.

🎶 https://open.spotify.com/track/08fiQ3uoQjtcLyOd8KLRiF

Database Removal Specialist

A comic strip made by CommitStrip.com of a developer erasing a production database.

As a Senior Developer at at Antistatique, a Swiss Agency, I am currently involved in numerous projects, sometimes simultaneously. Accessing databases is a common task that I perform on a daily basis. As a result, you might not be surprised to learn that I have accidentally deleted a production database.

“ If having a coffee in the morning doesn't wake you up, try deleting a table in a production database instead ”
Juozas Kaziukenas

I think almost all developers can relate. Accidentally using the wrong environment is the slightest error with the biggest consequences.

It’s ironic that I can’t recall what task I’ve made when I accidentally dropped a crucial production database. Still, what stands out in my memory is the intense feeling the instant I hit the “Return”.
It’s that unmistakable sensation of emptiness in your stomach when you realize you’ve just made a grave mistake in a live environment.

I must admit that it has happened to me more than once, and although it is not a proud moment, overreacting in such situations can lead to more errors. If you find yourself in a similar predicament, there are a few steps you can take to minimize the damage:

Don’t hesitate to seek assistance from your colleagues; they are there to support you, and everyone makes mistakes, so there is no need to be embarrassed. They will not judge you.
If available, immediately activate the maintenance or offline mode of your project.
Assign someone to notify the account holder of the current situation.
Initiate the recovery procedure.

“ Remember, backups are not enough without regular testing of recovery process. ”

Fake News Delivery

When working on project, it can be necessary to synchronize local or staging databases with production databases. Ideally, data-anonymizer scripts should always be executed, but occasionally I may become too lazy to run them. Consequently, customer email addresses may end up in my local or staging database.

Then during testing, some actions may trigger notifications that are routed to actual customers instead of mailcatcher or dummy customer email addresses.

“ I love accidentally send testing emails to real customers. It's like a walk in the park, so relaxing and effortless to deal with the aftermath ”

Cheer-up, there are only a few situations where email testing errors can result in significant issues. However certain situations — such as sending test emails to the wrong recipients — may lead to a literal disaster (who says data breach ?).

Don’t panic. Once you’ve sent out a message containing sensitive information (like email addresses and names), there’s no point to freaking-out since the damage has already been done.

Should you ever encounter such a scenario, there are a couple of measures you can implement to mitigate the impact:

As soon as you realize that your test emails are being sent to actual inboxes, check if it’s possible to halt the sending process (provided it’s still running).
Notify your team. The primary objective is to keep all relevant team members informed of the situation to address it effectively.
Determine what leaks, who was impacted, and what kind of damage was caused.
Send apologies to your customers.

Once the incident has been resolved, it’s crucial to learn from the experience. Analyze what went wrong and what measures should be taken to prevent a recurrence of the same situation for anyone else.

With that said, let’s explore some missteps made by more experienced developers.

The Razer & Docker for Windows friendship

In 2018, there was a strange compatibility issue between Docker for Windows and Razer Synapse driver management tool.

Running both programs simultaneously was not possible. If one program was running, the other one would refuse to launch.

It turned out that both programs created a global mutex using the GUID of their .NET assembly to ensure that only one copy of themselves would run.

What’s funnier is that the developers of both Docker for Windows and Razer Synapse found and integrated the wrong solution from the same Stack Overflow post (https://stackoverflow.com/questions/502303/how-do-i-programmatically-get-the-guid-of-an-application-in-c-sharp-with-net/502323#502323).

The Stackoverflow post was made on 2009 by a user who asked how to get the GUID of the running assembly.
Twelve minutes later, someone provided an answer, which turned out to be incorrect.

Instead of basing the GUID on their own code, they both used the GUID of a part of .NET itself, resulting in both programs using the same GUID.

The moral of this story is that even experienced developers use Stack Overflow and can fall victim to simple mistakes. This serves as a reminder to verify the accuracy of any solutions found on the internet.

Gabe Newell-inator

Back in 2015, Linux users who moved Steam’s per-user folder to another location and then launched the client were faced with a dangerous command that would delete all files recursively from the root directory.

The issue was caused by a variable that could end up empty if Steam’s files were moved or missing, and was passed as an argument to rm -rf.

The code in question lacked a check for an empty $STEAMROOT variable, and even included a comment that read # Scary!.

# figure out the absolute path to the script being run a bit
# non-obvious, the ${0%/*} pulls the path out of $0, cd's into the
# specified directory, then uses $PWD to figure out where that
# directory lives - and all this in a subshell, so we don't affect
# $PWD
STEAMROOT="$(cd "${0%/*}" && echo $PWD)"

# Scary!
rm -rf "$STEAMROOT/"*

Notifications everywhere

Do you recall the infamous case of accidentally sending a testing email to real customers?

Recently, Airbnb experienced a similar mishap where they sent numerous Android users a test notification in 2022.

The AirBNB Test dev Notification sent in 2022 — Credits: 9To5Google

Similarly, Burger King also made a mistake by sending thousands of customers a blank order receipt one week before, citing it as an internal processing issue.

“ Remember that we're all in this together. ”

Sources

Foone (Feb, 2020) Docker for Windows won’t run if you have the Razer Synapse driver management tool running.
https://twitter.com/Foone/status/1229641258370355200

Shaun Nichols (Jan, 205) Scary code of the week: Valve Steam CLEANS Linux PCs
https://www.theregister.com/2015/01/17/scary_code_of_the_week_steam_cleans_linux_pcs

Emma Roth (Aug, 2022) Airbnb accidentally sent tons of Android users a ‘test’ notification
https://www.theverge.com/2022/8/17/23309796/airbnb-android-users-accidental-test-notification

David Cassel (Jun, 2017) The Junior Dev Who Deleted the Production Database
https://thenewstack.io/junior-dev-deleted-production-database/

Hugo Rocha (Sep, 2020) What we learned after I deleted the main production database by mistake
https://itnext.io/what-we-learned-after-i-deleted-the-main-production-database-by-mistake-13c0a5815de5

Diana Lepilkina (Mar, 2020) What to Do When You Have Sent Test Emails to Your Customers
https://mailtrap.io/blog/sent-wrong-emails/

Stone Tao (Feb, 2019) Generate random Big Bang Theory esque episode names
https://stonet2000.github.io/BBT-Name-Generator/

ChatGPT (2022)
http://chat.openapi.com

Midjourney (2022)
https://midjourney.com