100 Days of Learning: Day 5 – Invoking an OpenFaaS function asynchronously

by Posted byAndré Jacobs

Photo by Lennart Schneider on Unsplash

Here is my Log book

Quick recap on some things for faasd

It can become an easy distant memory of how all the things work and where what lives. Especially when you end up using faas-cli to do like 99% of the interaction while you are developing functions.

Note: I am mainly referencing the setup I have installed on my Mac. See my Day 0 entry for more details.

  • An Ubuntu VM deployed using Multipass. I named my instance "faasd"
  • Currently this instance is accessible by the IP address: 192.168.64.4 (use multipass list to details)
  • The default user is ubuntu
  • SSH into the box using "ssh ubuntu@192.168.64.4"
  • faasd was cloned to "/root/faasd" because I used the root user ("sudo -i").
  • However the actual installed location for the configuration files is located at "/var/lib/faasd"
  • The main configuration file is located at: "/var/lib/faasd/docker-compose.yaml"

Explore docker-compose.yaml

How easy would it be to change the port at which we access the UI and functions from?

Edit /var/lib/faasd/docker-compose.yaml Note: root owns all the files in my setup (so I am running these commands as root)

# docker-compose.yaml
...
  gateway:
    ...
    ports:
      - "8080:8080"

My docker knowledge is super rusty but if I recall correctly then the above ports value means, map the docker host (which would be my Ubuntu faasd instance) port 8080 to the container (gateway) port 8080.

I am going to see if I can map the "external" end point to be port 9999.

    ports:
      - "9999:8080"

Restart faasd

root@faasd:~# sudo systemctl daemon-reload && sudo systemctl restart faasd

Verify from my local machine. Note at this point just running "faas-cli list" shows it can no longer connect to port 8080. So first I need to remap OPENFAAS_URL.

# For now just make changes in the current terminal session
$ export OPENFAAS_URL=http://192.168.64.4:9999

$ faas-cli list
Unauthorized access, run "faas-cli login" to setup authentication for this server

# Following my steps from Day 1 entry
╭ ~/Learning/faasd                                                                       [16:40:16]
╰ $ cat password.txt | faas-cli login --username admin --password-stdin
Calling the OpenFaaS server to validate the credentials...
WARNING! You are not using an encrypted connection to the gateway, consider using HTTPS.
credentials saved for admin http://192.168.64.4:9999

$ faas-cli list
Function                      	Invocations    	Replicas
iss-location                  	0              	1
nodeinfo                      	0              	1
cows                          	0              	1
expose                        	0              	1
helloworld                    	0              	1

# Boom!

Try and call a function.

$ curl http://192.168.64.4:9999/function/expose
Honeypot

Open the UI at http://192.168.64.4:9999

It actually worked!

Asynchronous result

OpenFaaS functions can be called asynchronously

I am going to try and write 2 functions. The first function "ping" will simulate a long running task. The second function "pong" will be called by the first function and then send out an email with the result from the first function.

Creating the ping function

$ mkdir ping
$ cd ping
$ faas-cli new --lang python3 ping
$ mate ping/handler.py

import random
import time

def handle(req):
    quotes = [
        'Neo: We need guns. Lots of guns.',
        'Switch: Great, the digital pimp at work.',
        'Trinity: Dodge this.',
        "Neo: I don't like the idea that I'm not in control of my life.",
        'Neo: There is no spoon!',
        'Cypher: Ignorance is bliss.'
    ]
    
    time.sleep(random.randint(10,30))
    
    return random.choice(quotes)

$ faas-cli up -f ping.yml
...
Deployed. 200 OK.

$ faas-cli describe ping
...
URL:                 http://192.168.64.4:9999/function/ping
Async URL:           http://192.168.64.4:9999/async-function/ping

You will notice that two URLs are available for calling the function.

First lets call it synchronously.

$ curl -i http://192.168.64.4:9999/function/ping
HTTP/1.1 500 Internal Server Error
Content-Length: 31
Content-Type: text/plain; charset=utf-8
Date: Sat, 13 Mar 2021 19:58:00 GMT
X-Call-Id: eb77497d-42bd-4a2a-bfb5-90061b884dbc
X-Content-Type-Options: nosniff
X-Start-Time: 1615665458133139591

Can't reach service for: ping.

Ouch! I am guessing this is to do with the default OpenFaaS timeouts.

Checking your function’s log: faas-cli logs FUNCTION_NAME

$ faas-cli logs ping
2021-03-13T19:57:22Z 2021/03/13 19:57:22 SIGTERM received.. shutting down server in 5s
...
2021-03-13T19:57:34Z 2021/03/13 19:57:34 Timeouts: read: 5s, write: 5s hard: 0s.
2021-03-13T19:57:34Z 2021/03/13 19:57:34 Listening on port: 8080

Ok so it looks like there is a timeout of 5 seconds for both reading and writing.

A quick search later and I found this example (and ironically it is in Python and uses sleep).

Edit ping.yml and set all timeouts to be 1 minute.

version: 1.0
provider:
  name: openfaas
  gateway: http://192.168.64.4:9999
functions:
  ping:
    lang: python3
    handler: ./ping
    image: dockername/ping:latest

    environment:
      read_timeout: "1m"
      write_timeout: "1m"
      exec_timeout: "1m"

Try calling ping synchronously again

$ faas-cli up -f ping.yml
$ curl -i http://192.168.64.4:9999/function/ping

HTTP/1.1 200 OK
Content-Length: 33
Content-Type: text/plain; charset=utf-8
Date: Sat, 13 Mar 2021 20:11:01 GMT
X-Call-Id: 83abfa46-012b-4fb3-b1c5-8399910d5669
X-Duration-Seconds: 10.168427
X-Start-Time: 1615666251217315311

Neo: We need guns. Lots of guns.

Ok our very slow function is working and holding up the terminal session until it completes. Take note of the X-Duration-Seconds header. This indicates how long the function took to complete.

Lets call the asynchronous endpoint and see what happens.

$ curl -i http://192.168.64.4:9999/async-function/ping
HTTP/1.1 405 Method Not Allowed
Date: Sat, 13 Mar 2021 20:13:17 GMT
Content-Length: 0

I read somewhere in the OpenFaaS documentation or even in the eBook that async invocations can’t be a GET request and must be POST

$ curl -d "" -i http://192.168.64.4:9999/async-function/ping
HTTP/1.1 202 Accepted
X-Call-Id: 4698d8b3-34b6-44e2-9f14-851abfeef447
X-Start-Time: 1615666738440754292
Date: Sat, 13 Mar 2021 20:18:58 GMT
Content-Length: 0

Ok this time by POSTing data to the function (-d "") we get a 202 response.

Now what? Where is the response?

In Severless for Everyone Else Alex shows us that you will need to check the logs for the queue worker on the server and use the X-Call-Id you received when invoking the function.

ubuntu@faasd:~$ sudo journalctl -t openfaas:queue-worker | grep '4698d8b3-34b6-44e2-9f14-851abfeef447'
Mar 13 20:18:58 faasd openfaas:queue-worker[14557]: [#1] Received on [faas-request]: 'sequence:1 subject:"faas-request" data:"{\"Header\":{\"Accept\":[\"*/*\"],\"Content-Length\":[\"0\"],\"Content-Type\":[\"application/x-www-form-urlencoded\"],\"User-Agent\":[\"curl/7.64.1\"],\"X-Call-Id\":[\"4698d8b3-34b6-44e2-9f14-851abfeef447\"],\"X-Start-Time\":[\"1615666738440754292\"]},\"Host\":\"192.168.64.4:9999\",\"Body\":\"\",\"Method\":\"POST\",\"Path\":\"\",\"QueryString\":\"\",\"Function\":\"ping\",\"QueueName\":\"\",\"CallbackUrl\":null}" timestamp:1615666738477080588 '

I don’t see the returned response, but there sure is a lot of relevant info if I ever wanted to see how a function was invoked.

Time to move onto phase 2 and that is to have the response from ping posted to a callback endpoint using the header X-Callback-Url

Creating the pong function

I am going to be installing sendmail directly on the Ubuntu instance and use that for email delivery from python.

ubuntu@faasd:~$ sudo apt install sendmail
ubuntu@faasd:~$ sudo sendmailconfig
# Answer y to everything
# Send a test email
ubuntu@faasd:~$ echo "Hello world!" | sendmail -v email@address.com

Surprisingly that actually worked first time!

Until I wrote the function and then couldn’t get it to connect to the sendmail server. To fix this I needed to let the sendmail server listen on all IP addresses. I could have binded on 10.62.0.1:25 but I chose to let it listen on all interfaces.

ubuntu@faasd:~$ sudo nano /etc/mail/sendmail.mc
# Comment out the 2 lines where 127.0.0.1 is being used (by adding dnl)

dnl DAEMON_OPTIONS(`Family=inet,  Name=MTA-v4, Port=smtp, Addr=127.0.0.1')dnl
dnl DAEMON_OPTIONS(`Family=inet,  Name=MSP-v4, Port=submission, M=Ea, Addr=127.0.0.1')dnl

# Save and exit
# Generate config file
ubuntu@faasd:~$ sudo m4 /etc/mail/sendmail.mc > /etc/sendmail.cf
# Restart sendmail
ubuntu@faasd:~$ sudo systemctl restart sendmail

ubuntu@faasd:~$ netstat -tuna  # <-- I love this, easy to remember tuna
# Install netstat using: sudo apt install net-tools
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN

At this stage the function reported the following error: … Relaying denied. IP name lookup failed [10.62.0.55].

ubuntu@faasd:~$ sudo nano /etc/mail/access

# Uncomment the line (this will relay 10.x.x.x)
Connect:10                              RELAY

ubuntu@faasd:~$ sudo -i
root@faasd:~# sudo makemap hash /etc/mail/access.db < /etc/mail/access
root@faasd:~# systemctl restart sendmail

Create the new pong function.

$ mkdir pong
$ cd pong
$ faas-cli new --lang python3 pong
$ mate pong/handler.py

Edit handler.py

import smtplib
from email.message import EmailMessage

def handle(req):
    msg = EmailMessage()
    msg.set_content(req)
    msg['Subject'] = 'ping result'
    msg['From'] = 'ubuntu@faasd'
    msg['To'] = 'you@youremail.com' # This should be a secret

    s = smtplib.SMTP('10.62.0.1') # This should be an environment variable
    s.send_message(msg)
    s.quit()

    return "done"

Deploy and do a test to see that email is sent.

$ faas-cli up -f pong.yml
$ curl -d "Hello world" http://192.168.64.4:9999/function/pong
done

Amazing! You won’t believe how much trouble I had to just try and send an email from the function. Lesson learned next time I will just deploy an out of the box email sending Docker image.

Playing ping pong

I should now be able to call the ping function asynchronously and tell it to callback on the pong function to email me the results.

$ curl -d "" -i http://192.168.64.4:9999/async-function/ping \
--header "X-Callback-Url: http://10.62.0.1:9999/function/pong"

HTTP/1.1 202 Accepted
X-Call-Id: 6286dcb7-46f7-4dfd-a8e7-d6c495d99db6
X-Start-Time: 1615684430227839637
Date: Sun, 14 Mar 2021 01:13:50 GMT
Content-Length: 0

# Repeated this a few times

First mistake was to use the 192.168.x.x address for the callback URL which the ping function don’t have access to. After changing it to the openfaas0 bridge (10.62.0.1) I could see from the logs that both functions appear to be working, but I don’t seem to receive the emails anymore … (drum roll) … they landed in my spam folder now 🙂

Conclusion

I think the hardest thing to wrap your brain around when working with VMs, containers, docker etc. is to mentally keep track of where in all this virtual space and networks the services live and what they can have access to and how data gets to them and back out to other services.

Initially I thought this would be a quick job of writing two functions, but instead I faced issues all over the place. At one point I even gave up on trying to send an email and instead just tried to save to /var/log/something and that failed because of permissions.

But I persevered and learned a ton of new things along the way.

Something that came in very handy was being able to check the logs of the functions and the queue worker.

ubuntu@faasd:~$ sudo journalctl -t openfaas-fn:FUNCTION_NAME
ubuntu@faasd:~$ sudo journalctl -t openfaas:queue-worker

If you are going to be playing around with faasd then I highly recommend you buy a copy of Severless for Everyone Else. I find myself coming back to the book time and time again.

1 comment on “100 Days of Learning: Day 5 – Invoking an OpenFaaS function asynchronously”

  1. Pingback: 100 Days of Learning: Day 20 & 21 – You win some, you lose some - André Jacobs

Comments are closed.