September 2012

If you regularly manage enormous data sets, you've probably heard about Tumblr's exciting new toolkit called Jetpants, which automates common processes for MySQL data management, most notably in the area of rebalancing shards for more efficient scaling. In evaluating and using Jetpants in various environments, we identified some interesting installation and usage gotchas, and we've documented those here - let us know if you find other issues worth exploring.  

Jetpants Installation/Usage Gotchas for Ubuntu 12.04:

Jetpants has to-date only been tested on RHEL/CentOS distributions, though working with the author (Evan Elias of Tumblr, thank you much!) I was able to get it running on Ubuntu 12.04 and Mint Maya/13. There will be more work to do, since RHEL/CentOS report back a service status ("service mysql status" output, see later in this post) differently than Ubuntu/Mint, and there may be other differences as well.

For your Jetpants Console workstation, install Ruby 1.9.3 (also 1.9.1 will be installed), and check /etc/alternatives/ruby points at ruby1.9.3.  Also be sure you installed the MySQL client libraries.

# apt-get install ruby1.9.3 rubygems libmysqlclient-dev
# ln -sf /usr/bin/ruby1.9.3 /etc/alternatives/ruby
# ln -sf /usr/bin/gem1.9.3 /etc/alternatives/gem

After that, do gem install Jetpants. You may get scary warnings about inability to convert ASCII to UTF8 characters during documentation installation as below:

# gem install jetpants

Be sure you've created /var/www writeable by the user you'll execute Jetpants as, and be sure you have an /etc/jetpants.yaml that is similar to the template provided on the Jetpants website.

And now you're ready to run Jetpants!

$ jetpants 

Tasks:

  jetpants activate_slave            # turn a standby slave into an active slave

  jetpants clone_slave               # clone a standby slave

  jetpants console                   # Jetpants interactive console

  jetpants destroy_slave             # remove a standby slave from its pool

  jetpants help [TASK]               # Describe available tasks or one specific task

  jetpants pools                     # display a full summary of every pool in the topology

  jetpants pools_compact             # display a compact summary (master, name, and size) of every pool in the topology

  jetpants promotion                 # perform a master promotion, changing which node is the master of a pool

  jetpants pull_slave                # turn an active slave into a standby slave

  jetpants rebuild_slave             # export and re-import data set on a standby slave

  jetpants regen_config              # regenerate the application configuration

  jetpants shard_cutover             # truncate the current last shard range, and add a new shard after it

  jetpants shard_offline             # mark a shard as offline (not readable or writable)

  jetpants shard_online              # mark a shard as fully online (readable and writable)

  jetpants shard_read_only           # mark a shard as read-only

  jetpants shard_split               # shard split step 1 of 4: spin up child pools with different portions of data set

  jetpants shard_split_child_reads   # shard split step 2 of 4: move reads to child shards

  jetpants shard_split_child_writes  # shard split step 3 of 4: move writes to child shards

  jetpants shard_split_cleanup       # shard split step 4 of 4: clean up data that replicated to wrong shard

  jetpants summary                   # display information about a node in the context of its pool

  jetpants weigh_slave               # change the weight of an active slave

As mentioned before, Jetpants is currently only well-tested for RHEL/CentOS distributions. RHEL/CentOS report back a service status ("service mysql status" output) differently than Ubuntu/Mint. Following is the requirements of Jetpants and actual output:

/sbin/service mysql status # check if mysql daemon running; output must include the string 'not running' if mysql is not running

This is how it works on both Ubuntu 12.04 and Mint Maya/13:

# service mysql stop
mysql stop/waiting
# service mysql status
mysql stop/waiting
# service mysql start
mysql start/running, process 31346
# service mysql status
mysql start/running, process 31346

I decided to check the return code, but it's always 0 regardless of the service state, so we must parse the output in Jetpants. See the following output:

# service mysql status ; echo $?
mysql stop/waiting
0
# service mysql start
mysql start/running, process 31629
# service mysql status ; echo $?
mysql start/running, process 31629
0

To support Debian-alikes, Jetpants must check for the other string, since the strings being output by "service" makes it impossible to check for a single substring across distros.

Over the coming days, I will be identifying other CentOS/RHEL-specific code and writing patches to make it work with at least also Ubuntu 12.04. In the future, if you're trying to port Jetpants to another distribution, you'll be able to use my pull request as a template for the areas that are distribution-specific.

When running Jetpants in EC2, there are some more gotchas (or perhaps they're just "be sure to follow best-practices"):

1. You must allow root login to your cluster. Generate a privkey/pubkey pair and allow root logins. 

      a. Edit ~root/.ssh/authorized_keys and get rid of the part where it has the "command=..." that prints an error and logs you out.

     b. Edit /etc/cloud/cloud.cfg and set disable_root: 0

     c.  Edit /etc/ssh/sshd_config and set PermitRootLogin without-password

2. You must be sure your master/slaves replication topology uses only IP addresses. If your slaves have, eg "Master_Host: ec2-5-2-3-8.compute.amazonaws.com" then Jetpants will fail. IP addressses is how you want your cluster configured in production, so do it, but it may surprise you if you're trying to build a small test cluster.

3. You must allow root login from localhost on all your MySQL instances. In other words, this command must return valid output on your slaves: mysql -e 'show slave status\G' - note that you can put root login credentials in ~root/.my.cnf rather than allow passwordless MySQL root logins.

In general, it's good to remember that Jetpants is a set of utilities for controlling an actual large MySQL cluster, so it will not work so well for tiny little test deploys where you've cut corners, and this is how it should be. Here are more Jetpants Environment Requirements/Limitations, from the documentation:

1. Need pigz installed on your cluster. If you use the Palomino Cluster Tool to build your shards, pigz will be installed for you.

2. No auto_increment on sharded tables. Must have ID generator.

3. PK of all sharded tables should start with the shard key. This will vastly improve performance of long-running processes like shard splits.

4. Port :3306 everywhere. One instance per node.

5. (Probably) doesn't work with MyISAM.