{"id":322,"date":"2021-03-27T21:43:24","date_gmt":"2021-03-27T21:43:24","guid":{"rendered":"https:\/\/andrejacobs.org\/?p=322"},"modified":"2022-04-11T20:24:23","modified_gmt":"2022-04-11T20:24:23","slug":"100-days-of-learning-day-19-installing-redis-on-faasd-and-using-python-rq-as-a-task-queue","status":"publish","type":"post","link":"https:\/\/andrejacobs.org\/100-days-challenge\/100-days-of-learning-day-19-installing-redis-on-faasd-and-using-python-rq-as-a-task-queue\/","title":{"rendered":"100 Days of Learning: Day 19 \u2013 Installing Redis on faasd and using Python RQ as a task queue"},"content":{"rendered":"\n
Photo by Eden Constantino<\/a> on Unsplash<\/a><\/p>\n\n\n\n Here is my Log book<\/a><\/p>\n Git repository<\/a> for the functions.<\/p>\n The next part in my Lego database learning project will be to schedule tasks that will go and download the Images. In order to do this I will need a task queue system.<\/p>\n In the past I have used Celery but I came across RQ<\/a> (Redis Queue) while reading the Flask Mega Tutorial<\/a> and decided since I like using Redis this will be a natural fit.<\/p>\n Redis<\/a> is an in-memory data structure store, used as a database, cache, and message broker.<\/p>\n As luck would have it, the book Serverless for Everyone Else<\/a> covers how to install redis directly on your faasd instance.<\/p>\n Learning action point:<\/strong> Ideally all the extra services will be installed and configured from a single config file and not needing me to manually do it. I am guessing cloud-init can take care of that.<\/p>\n SSH into the faasd instance.<\/p>\n Create the directory in which the Redis data will be stored.<\/p>\n Who is user and group with ID 1000?<\/p>\n Add the Redis server to be installed and managed by faasd.<\/p>\n I will be exposing the Redis server on port 9988 from the faasd instance (the multipass VM).<\/p>\n Restart faasd and verify<\/p>\n I will first be installing RQ locally and testing it from my Mac before adding to the OpenFaaS project.<\/p>\n I have one terminal window open to monitor the Redis server logs.<\/p>\n We will need to start a worker that will accept tasks to be processed.<\/p>\nInstalling Redis on faasd instance<\/h2>\n
# SSH into the faasd instance\n$ ssh -i ~\/.ssh\/id_rsa -o PreferredAuthentications=publickey ubuntu@192.168.64.4\n...\nubuntu@faasd:~$\n<\/code><\/pre>\n
ubuntu@faasd:~$ sudo mkdir -p \/var\/lib\/faasd\/redis\/data\nubuntu@faasd:~$ sudo chown -R 1000:1000 \/var\/lib\/faasd\/redis\/data\n\n# Confirm\nubuntu@faasd:~$ sudo ls -la \/var\/lib\/faasd\/redis\ntotal 12\ndrwxr-xr-x 3 root root 4096 Mar 27 20:05 .\ndrw-r--r-- 4 root root 4096 Mar 27 20:05 ..\ndrwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 20:05 data\n<\/code><\/pre>\n
ubuntu@faasd:~$ whoami\nubuntu\nubuntu@faasd:~$ id\nuid=1000(ubuntu) gid=1000(ubuntu) groups=1000(ubuntu),4(adm),20(dialout),24(cdrom),25(floppy),27(sudo),29(audio),30(dip),44(video),46(plugdev),117(netdev),118(lxd)\nubuntu@faasd:~$ id root\nuid=0(root) gid=0(root) groups=0(root)\n\n# The default user 'ubuntu' has user and group ID of 1000\n<\/code><\/pre>\n
ubuntu@faasd:~$ sudo nano \/var\/lib\/faasd\/docker-compose.yaml\n\n# Add this to the file\n redis:\n image: docker.io\/library\/redis:6.2.1-alpine\n volumes:\n - type: bind\n source: .\/redis\/data\n target: \/data\n cap_add:\n - CAP_NET_RAW\n entrypoint: \/usr\/local\/bin\/redis-server --appendonly yes\n user: "1000"\n ports:\n - "9988:6379"\n<\/code><\/pre>\n
ubuntu@faasd:~$ sudo systemctl daemon-reload && sudo systemctl restart faasd\n\n# Tail (follow) the redis logs\nubuntu@faasd:~$ sudo journalctl -f -t openfaas:redis\n\n# On my Mac I checked if the faasd VM port is exposed\n$ nmap -Pn -p9988 192.168.64.4\n...\nPORT STATE SERVICE\n9988\/tcp open nsesrvr\n<\/code><\/pre>\n
Installing Python RQ (Redis Queue)<\/h2>\n
$ cd learn-openfaas\/legodb\n$ source legodb\/venv\/bin\/activate\n\n(venv)$ pip install rq\n(venv)$ pip freeze > legodb\/requirements.txt\n<\/code><\/pre>\n
ubuntu@faasd:~$ sudo journalctl -f -t openfaas:redis\n<\/code><\/pre>\n
Create a worker<\/h3>\n
# Need to figure out how to connect to the Redis server (and not just localhost)\n(venv)$ rq worker --help\n...\n-u, --url TEXT URL describing Redis connection details.\n\n# Ok lets spin up a worker\n(venv)$ rq worker -u 'redis:\/\/192.168.64.4:9988\/0' johnny5\n\n20:49:06 Worker rq:worker:c0b23f5bf6e14800bbcb83b5a87775ca: started, version 1.7.0\n20:49:06 Subscribing to channel rq:pubsub:c0b23f5bf6e14800bbcb83b5a87775ca\n20:49:06 *** Listening on johnny5...\n20:49:06 Cleaning registries for queue: johnny5\n<\/code><\/pre>\n
Create a task and test it works<\/h3>\n