October 2010

Here at PalominoDB, we recently had the opportunity to work through a few mistakes and their aftermath.  Why is this pertinent to a blog by a database management team?  Regardless of your use of MySQL, Oracle, Cassandra or whatever technology you've deemed appropriate, it is the management of operations that becomes the true differentiator in your service providing.  Dealing with these recent mistakes got me thinking on what a crucial component of growth these moments are.  In the world of operations, mistakes can be quite visible, and are not uncommon to any of us.  One might argue that one of the biggest parts of our jobs is managing risk and mitigating impact of mistakes that will inevitably occur.

In the moment, it is easy to get very upset with someone who makes a mistake, or two or even three in a week.  I've learned from the best clients on how to deal with mistakes, by how they deal with ours. Mistakes are typically insights into a broken process, a bad behavior pattern or a lack of information and tools.  If you do manage risk and have planned appropriately, you should be able to survive and handle the immediate effects of the mistake.  Then, you have an opportunity to identify the issues, and improve them.  Anyone who sees a mistake (and I mean a true mistake of ignorance, versus a mistake of gross negligence) as a chance only to yell, blame and generally act discourteously is not interested in growth.  

I've found in life that growth is inevitably fostered by mistakes, occasional hardship, and even pain.  With an attitude of growth, these become quite manageable, if not welcome.  Recognizing this should help you reduce your stress around your own mistakes, and focus on the growth opportunities present.

So how do you deal with that mistake?  I learned a long time ago that the ability to truly, genuinely apologize is another one of those crucial life skills, and here is the perfect time to use that skill.  An immediate acknowledgment of your mistake, recognition of the impact your mistake has made, and a sincere request to offer amends are the parts of a good apology.  Don't focus on avoiding responsibility, trying to pass the buck, share blame, etc...  You are responsible for your part in this, and that is what you are apologizing for.

Next, you dig into what happened, and why it happened.  Perform a post-mortem, talk in ways that do not make anyone feel attacked, and find out where the root issue is.  Rather than focusing on blame and punishment, focus on the core issue, and put together a concise plan of attack to address the issue once it is exposed.  With this approach, continued improvement, and an environment that fosters openness and transparency becomes the norm.

I try to eliminate warnings and errors in the error log as much as possible - not just fixing actual issues, but also to eliminate the noise in the error log.  As I have been doing 5.1 upgrades lately, I have been seeing a lot of the following errors in the error log when upgrading:

[Warning] '--log_slow_queries' is deprecated and will be removed in a future release. Please use ''--slow_query_log'/'--slow_query_log_file'' instead.

And every time that happens I have to remember the proper syntax for slow query logging.  Finally I just wrote it in my ongoing cheat sheet, it's just a short text document that I keep on my desktop, and items rotate in and out of it frequently.

Without further ado, here is my cheat sheet for upgrading your slow query log from MySQL 5.0 to  MySQL 5.1:

change 

log-slow-queries=/var/log/mysqld/log-slow-queries.log 

to 

log_output=FILE # also can be FILE,TABLE or TABLE or NONE 

slow-query-log=1 

slow_query_log_file=/var/log/mysqld/log-slow-queries.log