» Process http://palominodb.com/blog Sun, 08 Nov 2009 16:35:29 +0000 http://wordpress.org/?v=2.9.2 en hourly 1 Version Consistency http://palominodb.com/blog/2007/11/11/version-consistency/ http://palominodb.com/blog/2007/11/11/version-consistency/#comments Sun, 11 Nov 2007 22:03:45 +0000 Administrator http://palominodb.com/blog/2007/11/11/version-consistency/ Everyone puts lip service to the concept of keeping versions consistent between servers but it is consistently one of the most broken best practices I see amongst my clients. The problems with such inconsistency are legion, and I’ll point out a few here.

Mismatch between production and development: Development environments are often neglected, particularly when it comes to OS and DBMS patches. When this happens I consistently find myself in a catch-22 when it is time to do patching in production. With nowhere to test a patch, you have to either go in blind and pray (never the most optimal choice) or neglect the patch, potentially retaining security vulnerabilities or bugs or limiting the access to potential performance or feature improvements. This is especially concerning in complex environments like an Oracle RAC cluster, where patches can be tricky at best. Of course, when working with any features outside of pure DML, you can expect occasional glitches as you introduce change, particularly when working with triggers, procedures, replication, partitioning, or any other complex feature.

Mismatch between MySQL masters and slaves: In MySQL, a lot of pain is taken to allow a slave’s version to be higher than the master’s. This can be a godsend in an upgrade scenario but I wouldn’t rely on this without significant testing. There are numerous features that may function incorrectly in such a scenario, particularly if you are using technologies that are relatively new and thus might be changing significantly between release versions, such as triggers in a 5.0.xx release. My recommendations are simple. If you’re running in day to day operations, you have to maintain consistency in your database versions. If you want to use replication as a way to do a rolling upgrade across a replicated cluster, test it thoroughly in development and make sure you test any procedures, views, triggers and functions against every possible storage engine and partitioned vs. non-partitioned tables. If your application does DDL, then test that as well. I can’t stress this one enough. I’ve had clients who were plagued with replication problems that “magically” went away once we brought their RDBMS and OS versions into sync.

Consistency between functional clusters: When I say functional cluster, I’m talking about a group of databases, usually replicated or physically clustered that support the same application component. In high activity environments, this kind of functional partitioning can be crucial in a scaling strategy. However, lack of discipline can cause these clusters to be built at or migrated to different versions over time. While this may not lead to isaster if your team is disciplined, at it’s best you will find your resources taxed. You’ll need to maintain multiple server builds, you’ll need to do a lot more qa at the data access tier and will require more steps in case you need to trade hardware between functional clusters. If you don’t have rock solid documentation and processes, new servers will be built at different versions, potentially even introduced into production with the wrong version. There will be redo work and troubleshooting that plagues you to varying levels.

To recap, when it comes to the same database in various environments (dev, test, stage etc…) there is no reason to allow for version drift. You must incorporate proper upgrade and deployment processes that, combined with discipline, will maintain that consistency. A proper configuration database can help with this as well. When it comes to a functional cluster, there absolutely must be consistency as well. There is a place for version mismatch in a rolling upgrade strategy, but only here, and only with extensive testing where possible. And finally, if there are valid reasons to have different versions across different functional clusters, be prepared for rigorous discipline and overhead in your administrative processes.

]]> http://palominodb.com/blog/2007/11/11/version-consistency/feed/ 0 The Prototype http://palominodb.com/blog/2007/10/15/the-prototype/ http://palominodb.com/blog/2007/10/15/the-prototype/#comments Mon, 15 Oct 2007 02:19:59 +0000 Administrator http://palominodb.com/blog/2007/10/15/the-prototype/ The first start-up stage I’ve worked within is the prototype phase. Within this phase traffic is not an issue for performance or scale, it’s about functionality. Low traffic and small datasets can hide atrocious code quite easily. The nice thing about this stage is that you should not have to invest a lot of time or money into your database and instead can focus on functionality and business development. Over-engineering at this point can be a devastating waste of very precious resources.

Here are the three criteria I would recommend focusing on:

1. Standards – I’m not talking about long lists of rules or apocryphal abbreviations here; rather focus on simplicity, consistency and usability. Don’t make decisions without asking yourself if the choice you’ve made is simple to practice, can be done so consistently, and would be easy for a new employee to use. Without standards, you will find that your database quickly becomes difficult to navigate and utilize – and a headache to maintain. Important standards include database object names, file system layouts, documentation design and locations.
2. Backups – There is nothing more frustrating than losing days or weeks of work. One of the key jobs of a DBA is making sure that your data can be recovered in case of a crisis. From the beginning, you need a solid backup and recovery process. Again, at this stage, nothing elaborate is required. It may be as simple as a nightly dump, or you may need more frequent dumps and perhaps even point in time recovery depending on the amount of change and number of people making the changes. Another key factor here is actually documenting how to perform the restores and practicing them regularly. Backups do fail and sometimes that failure is subtle. Regular practice will help to insure a successful recovery.
3. Documentation – As you make decisions about standards, set up processes, start building scripts, and implement tools and management utilities you need to document them for repeatability. Wikis are great for this, but again you must maintain a culture of discipline in regards to documentation. Tasks should not be completed without documentation and those responsible for doing so must be held accountable for it. This is an excellent way to start building discipline into the culture. As the prototype becomes operational more rigorous processes can build on this, such as change and problem management.
]]> http://palominodb.com/blog/2007/10/15/the-prototype/feed/ 0