Recently, Merge ran into the need to move to Redis Cluster for a few workloads. A quick primer on the differences:

Redis Cluster takes some getting used to, especially if you need commands such as LPOPRPUSH or others that operate on multiple keys. The reason for this is that cluster-mode moves your data onto multiple “shards” (individual Redis instances in the cluster) but as a result, it cannot guarantee that any two keys are on the same shard, and two different shards cannot share the command if each has one of the keys.

While things like this can be worked around in application logic, something that you probably won’t be able to do without is a way to, how do you say, TEST this setup. Sure, you could run a test suite against your development environment, but that will slow down your developers’ iteration speed if they can’t run this on their laptops.

Your Redis-classic docker-compose section may look like:

redis:
 image: redis:6.x.y-alpine
 networks:
   - your-local-network
 ports:
   - "6379:6379"

But don’t worry, the cluster-version isn’t THAT much worse especially if you copy a few of our scripts further down!

Configuring Redis Cluster

Before we begin, I’d just like to shout out this medium article for the basics. Any Redis Cluster needs:

• Individual Redis instances
• And a configuration script that tells them about each other
• And what each nodes’ job is (primary vs replica, how many shards there are)

The reason these scripts are better is that this script does not require you to set IP addresses for your Redis nodes. To do that, we leverage the <code class="blog_inline-code">getent</code> command to figure out what the Docker network IP address is from within the Docker containers. Let’s start with those individual Redis instances in docker-compose:

redis-cluster-node-#:
 image: redis:6.x.y-alpine
 command: redis-server /usr/local/etc/redis/redis.conf
 networks:
   - your-local-network
 ports:
   - "9079:6379"
 volumes:
   - ${PWD}/path/redis_conf_folder:/usr/local/etc/redis
redis-cluster-node-#+1:
 image: redis:6.x.y-alpine
 command: redis-server /usr/local/etc/redis/redis.conf
 networks:
   - your-local-network
 ports:
   - "9080:6379"
 volumes:
   - ${PWD}/path/redis_conf_folder:/usr/local/etc/redis

Importantly:

• Allocate as many nodes as you want, be sure to bump the number in the name
• Keep in mind that ports...need unique ports or your local machine will complain about port conflict
• The left half of the volume refers to the folder where you are keeping a redis.conf file

Speaking of redis.conf, it will look like:

port 6379
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes

Which is pretty standard, you won’t need to modify anything there.

Now for the (slightly) complicated part, configuring the cluster and telling these nodes to work together. First off, make a new docker-compose section as follows:

redis-cluster-configure:
 image: redis:6.x.y-alpine
 command: /usr/local/etc/redis/redis-cluster-create.sh
 networks:
   - your-local-network
 depends_on:
   - redis-cluster-node-0
   - redis-cluster-node-1
...
 volumes:
   - ${PWD}/path/redis_conf_folder:/usr/local/etc/redis

You may notice it follows the same pattern, except that we are running a custom script called <code class="blog_inline-code">redis-cluster-create.sh</code>. That file needs to go in your local path/redis-conf-folder (whatever you want to name that) and should be:

# wait for the docker-compose depends_on to spin up the redis nodes usually takes this long
sleep 10


node_0_ip=$(getent hosts redis-cluster-node-0 | awk '{ print $1 }')


node_1_ip=$(getent hosts redis-cluster-node-1 | awk '{ print $1 }')


...


redis-cli --cluster create $node_0_ip:6379 $node_1_ip:6379 ... --cluster-replicas 1 --cluster-yes

So let’s explain what it does. The sleep is just there to allow you to run “docker-compose up redis-cluster-configure”. When you do that, Docker sees it depends on your individual Redis nodes and spins those up. The 10 second sleep is to allow that to complete since those nodes need to be ready. There’ll be more elegant ways to do this waiting, of course.

What follows is a number of lines (depending on your desired cluster size) that set variables of the IP addresses of your individual Redis nodes. This is the magic part, it allows you to refer to the name you set in the docker-compose of the individual Redis nodes as opposed to hard coding an IP address for them on the local Docker network.

Finally, we run “redis-cli --cluster create …” which takes in those IP addresses (and it must be IP addresses or else we would have passed in the docker-compose node names directly here) which stitches them together into a single cluster.

This all reminds me of the takeaways from one of our prior articles where the interactions between Celery and Kubernetes resulted in a very specific configuration requirement to get the behavior we desired. Similarly here, we are fulfilling the IP-address requirement using a linux command running inside a dockerized virtual network to get our desired outcome. It just goes to show that in the current era, expertise across a variety of systems is required to get to optimal solutions.

Happy hacking!

{{blog-cta-100+}}