100 Days of Learning: Day 19 – Installing Redis on faasd and using Python RQ as a task queue

Photo by Eden Constantino on Unsplash

Here is my Log book

Git repository for the functions.

The next part in my Lego database learning project will be to schedule tasks that will go and download the Images. In order to do this I will need a task queue system.

In the past I have used Celery but I came across RQ (Redis Queue) while reading the Flask Mega Tutorial and decided since I like using Redis this will be a natural fit.

Redis is an in-memory data structure store, used as a database, cache, and message broker.

Installing Redis on faasd instance

As luck would have it, the book Serverless for Everyone Else covers how to install redis directly on your faasd instance.

Learning action point: Ideally all the extra services will be installed and configured from a single config file and not needing me to manually do it. I am guessing cloud-init can take care of that.

SSH into the faasd instance.

# SSH into the faasd instance
$ ssh -i ~/.ssh/id_rsa -o PreferredAuthentications=publickey ubuntu@192.168.64.4
...
ubuntu@faasd:~$

Create the directory in which the Redis data will be stored.

ubuntu@faasd:~$ sudo mkdir -p /var/lib/faasd/redis/data
ubuntu@faasd:~$ sudo chown -R 1000:1000 /var/lib/faasd/redis/data

# Confirm
ubuntu@faasd:~$ sudo ls -la /var/lib/faasd/redis
total 12
drwxr-xr-x 3 root   root   4096 Mar 27 20:05 .
drw-r--r-- 4 root   root   4096 Mar 27 20:05 ..
drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 20:05 data

Who is user and group with ID 1000?

ubuntu@faasd:~$ whoami
ubuntu
ubuntu@faasd:~$ id
uid=1000(ubuntu) gid=1000(ubuntu) groups=1000(ubuntu),4(adm),20(dialout),24(cdrom),25(floppy),27(sudo),29(audio),30(dip),44(video),46(plugdev),117(netdev),118(lxd)
ubuntu@faasd:~$ id root
uid=0(root) gid=0(root) groups=0(root)

# The default user 'ubuntu' has user and group ID of 1000

Add the Redis server to be installed and managed by faasd.

ubuntu@faasd:~$ sudo nano /var/lib/faasd/docker-compose.yaml

# Add this to the file
  redis:
    image: docker.io/library/redis:6.2.1-alpine
    volumes:
      - type: bind
        source: ./redis/data
        target: /data
    cap_add:
      - CAP_NET_RAW
    entrypoint: /usr/local/bin/redis-server --appendonly yes
    user: "1000"
    ports:
      - "9988:6379"

I will be exposing the Redis server on port 9988 from the faasd instance (the multipass VM).

Restart faasd and verify

ubuntu@faasd:~$ sudo systemctl daemon-reload && sudo systemctl restart faasd

# Tail (follow) the redis logs
ubuntu@faasd:~$ sudo journalctl -f -t openfaas:redis

# On my Mac I checked if the faasd VM port is exposed
$ nmap -Pn -p9988 192.168.64.4
...
PORT     STATE SERVICE
9988/tcp open  nsesrvr

Installing Python RQ (Redis Queue)

I will first be installing RQ locally and testing it from my Mac before adding to the OpenFaaS project.

$ cd learn-openfaas/legodb
$ source legodb/venv/bin/activate

(venv)$ pip install rq
(venv)$ pip freeze > legodb/requirements.txt

I have one terminal window open to monitor the Redis server logs.

ubuntu@faasd:~$ sudo journalctl -f -t openfaas:redis

Create a worker

We will need to start a worker that will accept tasks to be processed.

# Need to figure out how to connect to the Redis server (and not just localhost)
(venv)$ rq worker --help
...
-u, --url TEXT                  URL describing Redis connection details.

# Ok lets spin up a worker
(venv)$ rq worker -u 'redis://192.168.64.4:9988/0' johnny5

20:49:06 Worker rq:worker:c0b23f5bf6e14800bbcb83b5a87775ca: started, version 1.7.0
20:49:06 Subscribing to channel rq:pubsub:c0b23f5bf6e14800bbcb83b5a87775ca
20:49:06 *** Listening on johnny5...
20:49:06 Cleaning registries for queue: johnny5

Create a task and test it works

Add a new file named tasks.py in the same directory where handler.py lives. To test that Redis and RQ works, I will be using the example from Flask Mega Tutorial.

# tasks.py
import time

def example_task(seconds):
    print('Starting task')
    for i in range(seconds):
        print(i)
        time.sleep(1)
    print('Task completed')

I modified my indev.py file and ran it

# indev.py
from redis import Redis
import rq

if __name__ == "__main__":
    # Test RQ and Redis is working
    queue = rq.Queue('johnny5', connection=Redis.from_url('redis://192.168.64.4:9988/0'))
    job = queue.enqueue('legodb.tasks.example_task', 42)
    print(job.get_id())
(venv)$ python indev.py
b147c143-53b1-469c-ba90-d264a1b8935f
# ^^ the job.get_id()

The output from the worker

20:57:35 johnny5: legodb.tasks.example_task(42) (b147c143-53b1-469c-ba90-d264a1b8935f)
Starting task
0
1
2
...
41
Task completed
20:58:17 johnny5: Job OK (b147c143-53b1-469c-ba90-d264a1b8935f)
20:58:17 Result is kept for 500 seconds

Nice one! We have a worker running locally that can process tasks.

Is this really using the faasd Redis server?

I noticed that monitoring the openfaas:redis logs had nothing about the worker or tasks and thus I was wondering if this is actually working the way I think it was. So I decided to do some testing.

# Stop faasd
ubuntu@faasd:~$ sudo systemctl stop faasd

Not long after stopping faasd, the RQ worker logged the following.

...
21:06:52 Worker rq:worker:c0b23f5bf6e14800bbcb83b5a87775ca: found an unhandled exception, quitting...
...
redis.exceptions.ConnectionError: Connection closed by server.
Error 61 connecting to 192.168.64.4:9988. Connection refused.

Ok so the worker is indeed using Redis from our faasd instance. It also did stop running.

Started it all up again and did another test run.

ubuntu@faasd:~$ sudo systemctl start faasd

(venv)$ rq worker -u 'redis://192.168.64.4:9988/0' johnny5
...
21:16:37 *** Listening on johnny5...
21:17:24 johnny5: legodb.tasks.example_task(42) (d65a042d-b188-4f70-878e-c132cfb34b6c)
Starting task
...

What happens when the task does not exist? I changed the name being requested to "legodb.tasks.example_task_fail" and ran it.

(venv)$ python indev.py
8a093e7a-a6d6-4746-98d1-9bdec7cce074
# ^^ Interestingly we did get a job id

# Worker output
21:18:11 johnny5: legodb.tasks.example_task_fail(42) (8a093e7a-a6d6-4746-98d1-9bdec7cce074)
... throws a big exception
AttributeError: module 'legodb.tasks' has no attribute 'example_task_fail'

Ok at least the worker didn’t just quit.

What happens when there is no worker with the specified name? I changed the name being requested to "johnny5_fail" and ran it.

(venv)$ python indev.py
3a50f84f-47db-400e-8b6a-41ab5d8c2bd5

Ok this is interesting. So there is no worker with that specified name but we did schedule the job to be run. My guess is the task is waiting in Redis until a worker with that name shows up for work.

Let’s clock that worker bee in.

(venv)$ rq worker -u 'redis://192.168.64.4:9988/0' johnny5_fail
...
21:25:09 johnny5_fail: legodb.tasks.example_task(42) (3a50f84f-47db-400e-8b6a-41ab5d8c2bd5)
Starting task
0
1
...

Nice! Ok so you can schedule tasks even if no worker is running to pick up the tasks.

Next actions

Tomorrow I will start hooking this up to my OpenFaaS function and also see how we can start workers on the faasd instance reliably.


Leave a comment

Your email address will not be published. Required fields are marked *