Version Consistency
Everyone puts lip service to the concept of keeping versions consistent between servers but it is consistently one of the most broken best practices I see amongst my clients. The problems with such inconsistency are legion, and I’ll point out a few here.
Mismatch between production and development: Development environments are often neglected, particularly when it comes to OS and DBMS patches. When this happens I consistently find myself in a catch-22 when it is time to do patching in production. With nowhere to test a patch, you have to either go in blind and pray (never the most optimal choice) or neglect the patch, potentially retaining security vulnerabilities or bugs or limiting the access to potential performance or feature improvements. This is especially concerning in complex environments like an Oracle RAC cluster, where patches can be tricky at best. Of course, when working with any features outside of pure DML, you can expect occasional glitches as you introduce change, particularly when working with triggers, procedures, replication, partitioning, or any other complex feature.
Mismatch between MySQL masters and slaves: In MySQL, a lot of pain is taken to allow a slave’s version to be higher than the master’s. This can be a godsend in an upgrade scenario but I wouldn’t rely on this without significant testing. There are numerous features that may function incorrectly in such a scenario, particularly if you are using technologies that are relatively new and thus might be changing significantly between release versions, such as triggers in a 5.0.xx release. My recommendations are simple. If you’re running in day to day operations, you have to maintain consistency in your database versions. If you want to use replication as a way to do a rolling upgrade across a replicated cluster, test it thoroughly in development and make sure you test any procedures, views, triggers and functions against every possible storage engine and partitioned vs. non-partitioned tables. If your application does DDL, then test that as well. I can’t stress this one enough. I’ve had clients who were plagued with replication problems that “magically” went away once we brought their RDBMS and OS versions into sync.
Consistency between functional clusters: When I say functional cluster, I’m talking about a group of databases, usually replicated or physically clustered that support the same application component. In high activity environments, this kind of functional partitioning can be crucial in a scaling strategy. However, lack of discipline can cause these clusters to be built at or migrated to different versions over time. While this may not lead to isaster if your team is disciplined, at it’s best you will find your resources taxed. You’ll need to maintain multiple server builds, you’ll need to do a lot more qa at the data access tier and will require more steps in case you need to trade hardware between functional clusters. If you don’t have rock solid documentation and processes, new servers will be built at different versions, potentially even introduced into production with the wrong version. There will be redo work and troubleshooting that plagues you to varying levels.
To recap, when it comes to the same database in various environments (dev, test, stage etc…) there is no reason to allow for version drift. You must incorporate proper upgrade and deployment processes that, combined with discipline, will maintain that consistency. A proper configuration database can help with this as well. When it comes to a functional cluster, there absolutely must be consistency as well. There is a place for version mismatch in a rolling upgrade strategy, but only here, and only with extensive testing where possible. And finally, if there are valid reasons to have different versions across different functional clusters, be prepared for rigorous discipline and overhead in your administrative processes.
