karlstoney.com

Redis-Sentinel TwemProxy Agent

Karl Stoney -

I was recently introduced to Redis by a colleague. For those of you who don't know what it is, it is essentially an incredibly fast key-value store.

We had decided to adopt this technology at work, so we began to think about Performance, Redundancy etc, and after a day or two of playing around discovered that there isn't really an established solution for what we wanted to do.

As a result I ended up combining a few different technologies to get the setup we required, to give us redundancy, automatic failover, and horizontal scaling.

Prerequisites

The whole solution utilises the following technologies:

  • Redis
  • Redis Sentinel
  • nodejs
  • Redis-TwemProxy Agent
  • HAProxy
  • TwemProxy

Redis Configuration

The purpose of this article isn't to tell you how to set each of these technologies up individually, it is to show you how I link them together, and more specifically, the Redis-TwemProxy Agent that does the linking of Redis-Sentinel and TwemProxy

This diagram best explains it:

Redis slaves in each shard are read-only copies of their master.

Redis-Sentinel constantly monitors your master nodes in each shard, should a master node go down, a slave node is promoted to the master, and the other slaves are reconfigured to be slaves of this new master.

The Redis-TwemProxy agent script is subscribed to Redis-Sentinel, and detects the master-change event, when this happens, the TwemProxy config is updated and the TwemProxy service is restarted.

Redis-TwemProxy Agent

This was the missing link for me. Redis Sentinel worked perfectly, and promoted my slaves to masters when the master went down, however TwemProxy was still directing traffic to the old, now dead, slave.

The script I have written over on GitHub, is this glue. It is a simple nodejs script that is run permanently on boot on the TwemProxy server, and keeps the TwemProxy config up to date.

The Result

If you tail your TwemProxy Sentinel log, you should see something like this:

[11:31:04] Loading TwemProxy config
[11:31:04] Redis Sentinel TwemProxy Agent Started on: Fri Sep 20 2013 11:31:04 GMT+0000 (UTC)
[11:31:04] Subscribing to sentinel.
[11:31:04] Connection to Redis Sentinel established.
[11:31:04] Getting latest list of masters...
[11:31:04] Master received: redis1 172.19.111.21:6379
[11:31:04] Updating Master redis1 to 172.19.111.21:6379
[11:31:04] Master received: redis2 172.19.111.22:6379
[11:31:04] Updating Master redis2 to 172.19.111.22:6379
[11:31:04] Saving TwemProxy config
[11:31:04] TwemProxy restarted with output:
[11:31:04] Stopping ...
TwemProxy stopped
Starting TwemProxy server...
[11:09:40] Received switch-master: { details:
   { 'master-name': 'redis2',
     'old-ip': '172.19.111.22',
     'old-port': '6379',
     'new-ip': '172.19.111.21',
     'new-port': '6380' } }
[11:09:40] Nutcracker config updated with new content:
[11:09:40] twem1:
  listen: 172.19.111.15:22121
  hash: fnv1a_64
  distribution: modula
  auto_eject_hosts: false
  redis: true
  server_retry_timeout: 100
  server_failure_limit: 500
  servers:
   - 172.19.111.21:6379:1 redis1
   - 172.19.111.21:6380:1 redis2

[11:09:40] Nutcracker restarted with output:
[11:09:40] Stopping ...
TwemProxy stopped
Starting TwemProxy server...

Script Configuration

The script configuration is simple really, edit the /lib/cli.js script and set the rather self explanatory configuration settings.

I hope this helps those of you who are tasked with setting up and configuring Redis!