http://dev.mysql.com/tech-resources/articles/testing-partitions-large-db.html

http://datacharmer.blogspot.com/2006/03/mysql-51-improving-archive-performance.html

http://blog.plasticfish.info/categories/tech/mysql/mysql-partitioning/

http://mysqlguy.net/2008/02/20/using-events-manage-table-partitioning-date-wrap

As far as partition management goes, I like the idea of DB events, primarily because you don’t have to worry about crontabs continually slamming a database that’s down, and the events are portable with the database making failovers, migrations and copies that much easier. So, what events do we need to manage partitions? There are three cases that I have been using. Remember, these are for date based partitions (logs, events etc…)
1) Create the next day (or week or month etc…) ’s partition.

2) Purge any partions older than n days.

3) Check that the necessary partition for the current time period exists, and if not, create it on the fly. After all, what happens if the DB is down when the event is supposed to fire? We need to remember the edge cases.

So, in the interest of sharing, here are some events I created. I consider these rudimentary at best. Remember that it isn’t just about functionality. These need to be robust. I’m still planning on adding more error handling and email notifications on failure but I wanted to share. (I’m a giver)
DELIMITER $

CREATE EVENT log_add_partition
ON SCHEDULE
EVERY 1 DAY STARTS ‘2008-02-19 23:59:00′
DO
BEGIN
SET @stmt := CONCAT(
‘ALTER TABLE log ADD PARTITION (’
, ‘PARTITION p’
, DATE_FORMAT( DATE_ADD( CURDATE(), INTERVAL 1 DAY ), ‘%y%m%d’ )
, ‘ VALUES LESS THAN (’
, TO_DAYS( CURDATE() ) + 1
, ‘))’
);
PREPARE stmt FROM @stmt;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
END
$

DELIMITER ;

DELIMITER $

CREATE EVENT log_purge_partition
ON SCHEDULE
EVERY 1 DAY STARTS ‘2008-02-20 00:00:01′
DO
BEGIN
DECLARE done INT DEFAULT 0;
DECLARE part_name VARCHAR(25);
DECLARE cur1 CURSOR FOR
SELECT partition_name
FROM information_schema.partitions
WHERE table_name = ‘log’
AND str_to_date(substr(partition_name from 2), ‘%y%m%d’) < date_sub(now(), INTERVAL 31 day);
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = 1;

OPEN cur1;

REPEAT
FETCH cur1 INTO part_name;
IF NOT done THEN
SET @stmt := CONCAT(
‘ALTER TABLE log DROP PARTITION ‘
, part_name
);
PREPARE stmt FROM @stmt;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
END IF;
UNTIL done END REPEAT;

CLOSE cur1;
END
$

DELIMITER ;
DELIMITER $

CREATE EVENT log_check_partition
ON SCHEDULE
EVERY 15 MINUTE STARTS ‘2008-02-20 00:00:01′
DO
BEGIN
DECLARE no_rows INT DEFAULT 0;
DECLARE part_name VARCHAR(25);
DECLARE cur1 CURSOR FOR
SELECT partition_name
FROM information_schema.partitions
WHERE table_name = ‘log’
AND str_to_date(substr(partition_name from 2), ‘%y%m%d’) = date_format(now(), ‘%y%m%d’) ;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET no_rows = 1;

OPEN cur1;

FETCH cur1 INTO part_name;

IF no_rows=1 THEN

SET @stmt := CONCAT(
‘ALTER TABLE log ADD PARTITION (’
, ‘PARTITION p’
, DATE_FORMAT ( CURDATE(), ‘%y%m%d’ )
, ‘ VALUES LESS THAN (’
, TO_DAYS( CURDATE() ) + 1
, ‘))’
);
PREPARE stmt FROM @stmt;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;

END IF;

CLOSE cur1;
END
$

DELIMITER ;

1. Prototype
2. Initial Production
3. Pain Point 1: Availability
4. Pain Point 2: Performance
5. Pain Point 3: Scalability
6. Maintenance Mode

Each of these phases must be taken with a viewpoint of the entire business. While all of us OCD architects and administrators would absoutely love to design the perfect solution from scratch, it is not realistic for a company on a shoestring budget or that has no clue about the traffic patterns they’ll be driving. Bottlenecks and pain points need to be looked at as indicators of needs for a) growth or b) improved processes. Taken in this manner, one can create a roadmap for growth based on the individual site’s needs rather than busting the budget immediately or optimizing for scenarios that will never occur.

Of course, there is an exception, and that is pain points that compromise a site’s revenue or reputation. Luckily, in 80 – 90% of the scenarios (potentially more!) these are problems that have already been solved to some degree. In the immortal words of Tyler Durden – You are not a beautiful or unique snowflake. That’s where my job comes in, to help utilize those existing lessons learned before investing in new solutions from scratch. Each phase has them, and that’s where I will be focusing my initial posts.