Making Mistakes (and owning up to them)
Fessing Up
No one wants to admin that they messed up. But the reality is, YOU ARE GOING TO MESS UP. It’s one of those when not if situations. The worst thing you could possibly do is lie about it, or blame it on someone else. Which brings us to the next paragraph.
Hiding It Makes It Worse
Don’t try and hide it. Most companies have really good system administrators. They know exactly when you were in the production environment, what you did, how long you stayed, and exactly what you changed. Just admit you made a mistake, document it, and then work with IT to fix it as fast as possible. It’s a million times better when you go to your manager and tell them you made a mistake than it is having your manager come to you and ask if and why you made a mistake.
Honest and Prompt Admission Helps A Lot
The faster you say something after you realize that you made a mistake the faster and easier it can be fixed, (most of the time). If you realize you made a mistake and then spend 2 hours deciding on what you should do and who you should tell, it will only make it worse.
Fix It If You Can But Don’t Make It Worse
If you accidentally changed one field in a production database and you are relatively certain that you can write a quick query and fix it then do it, but make sure you still tell someone and document it. You don’t want to make the problem worse so unless you are certain that you can fix it without any more problems then just leave it alone and get help.
Don’t Share Credentials
The only thing worse than introducing a breaking change into production is getting blamed for it when you had nothing to do with it. Most of the companies I have worked for have employees sharing credentials with other employees at some point. Maybe you have access to the production server but you are going on vacation so you give your login to another employee. Don’t do that, force management to grant credentials to the person taking over in your absence.
Most Issues Are Fixable
Most importantly, remember that software systems are designed with redundancies. If you accidentally delete a git repo, chances are that some other developer has a recent clone or pull on their system. There is rarely a mistake that can’t be rectified.
Wouldn’t the obvious solution be…
Some people listening to/reading this are going to be like, “Yeah, but shouldn’t you be giving the advice that you just shouldn’t make changes to production directly?”
Yea, that is how it is supposed to work. You make a change in DEV then it gets QA’d, then you push to STAGING, then it gets “smoke tested”, then you push to PROD.
But if you’ve been doing this for more than a few weeks you know that the reality is that you are occasionally asked to fix stuff in production.
My advice is to never do it, but this podcast is supposed to help when you DO DO it and mess something up.