Photo by Jordan Harrison on Unsplash
Here is my Log book
Visual Studio Code or PyCharm?
Yesterday I said that I will be using PyCharm as my Python IDE, however one part of this challenge is to get out of my comfort zone. So I have decided to give VS Code a try as my main Python IDE for a bit.
I have used VS Code here and there before but never in all out mode and certainly not for Python.
The Python support needs to be installed additionally.
Create new project
Up to this point none of the OpenFaaS functions I have created thus far has been stored in a git repository. I decided I will create a repo on GitHub for this new lego database project as well as any other OpenFaaS functions I work on during the challenge.
I created the repo https://github.com/andrejacobs/learn-openfaas
$ git clone git@github.com:andrejacobs/learn-openfaas.git
$ cd learn-openfaas
Build a Flask microservice with OpenFaaS
As luck would have it, OpenFaaS already has a tutorial written on how to setup Flask using the template as well as how to deploy an existing Flask based app.
https://www.openfaas.com/blog/openfaas-flask/
Learning action point: There is a free course by the LinuxFoundation.
This made me LOLed "That’s right folks. Serverless has servers."
The template repo has good info as well.
Initial scouting of the above tutorial and reading the template docs makes me think I want to use the python3-http template. The nice thing about the template is that the Content-Type will be set to Content-Type: application/json if a dictionary is returned.
The tutorial mentions that you can have RESTfull style URLs and gives a short example, but where there is a gap in my knowledge and the tutorial is where do you create this mapping? I dug a bit deeper into the linked documentation and it seems that I would need to create and deploy 1 function per API end point and would then create this mapping file to bind them all. For now this is too much trouble for what I want to achieve. (p.s. I could very easily be wrong here)
However the tutorial also gives an example of how to deploy an existing Flask app and it shows how you can have multiple API end points from the same "function". I started going down this route but then realised there must be an easier option.
Detour: Let’s explore faas-cli templates a bit
Recap, faas-cli comes with standard templates like python3 and it will pull these from the git repo when a new function is started with faas-cli new.
Other templates like python3-flask need to be first pulled from the template store.
For example asking for a new function based of the python3-http template inside of an empty directory will result in an error.
$ cd ~/temp/template-test
$ ls -la
... nothing
$ faas-cli new --lang python3-http myfunction
2021/03/17 19:21:51 No templates found in current directory.
2021/03/17 19:21:51 Attempting to expand templates from https://github.com/openfaas/templates.git
2021/03/17 19:21:52 Fetched 13 template(s) : [csharp dockerfile go java11 java11-vert-x node node12 node14 php7 python python3 python3-debian ruby] from https://github.com/openfaas/templates.git
Python3-http is unavailable or not supported
$ ls template
csharp/ java11/ node12/ python/ ruby/
dockerfile/ java11-vert-x/ node14/ python3/
go/ node/ php7/ python3-debian/
faas-cli did however pull the standard templates into the current directory.
To pull a "batteries-not-included" template.
$ faas-cli template store pull python3-flask
...
2021/03/17 19:27:49 Fetched 7 template(s) : [python27-flask python3-flask python3-flask-armhf python3-flask-debian python3-http python3-http-armhf python3-http-debian] from https://github.com/openfaas/python-flask-template
$ ls template
csharp/ node/ python27-flask/ python3-flask-debian/
dockerfile/ node12/ python3/ python3-http/
go/ node14/ python3-debian/ python3-http-armhf/
java11/ php7/ python3-flask/ python3-http-debian/
java11-vert-x/ python/ python3-flask-armhf/ ruby/
# Create the new function from the template
$ faas-cli new --lang python3-http myfunction
...
Function created in folder: myfunction
Stack file written: myfunction.yml
Ok so, what happens when we run faas-cli up?
faas-cli up is a combination of "build", "push" and "deploy"
$ ls
myfunction/ myfunction.yml template/
$ faas-cli up -f myfunction.yml
$ ls
build/ myfunction/ myfunction.yml template/
Notice that a build directory was created.
If you open the myfunction directory you will notice that you only have the handler.py and requirements.txt files. Neither of these contain the scaffolding code, in this case where is all the Flask setup and run code coming from?
$ ls template/python3-http
Dockerfile function/ index.py requirements.txt template.yml
The code you are looking for (SW for the win) is located in index.py.
Revelation: The template directory is instrumental during the build phase. Meaning it is not just a one off clone template and modify it.
# Let's proof ^^
$ mv template/python3-http template/not-the-droids-you-are-looking-for
$ faas-cli up -f myfunction.yml
[0] > Building myfunction.
[0] < Building myfunction done in 0.00s.
[0] Worker done.
Total build time: 0.00s
Errors received during build:
- language template: python3-http not supported, build a custom Dockerfile
$ cat myfunction.yml
...
functions:
myfunction:
lang: python3-http
Thus faas-cli uses the lang value to determine which template to use during the build phase.
Back on tour: Creating a new function using the python3-http template
After some more exploring of how I would like to build the REST apis for my learning project and lots of reading, I ended up going the path of least resistance to get my MVP out.
Programmer trap: I was already dreaming about amazing but time consuming ways to go ahead and build a new shiny thing. But the reality is I need to focus on what I actually wanted here and that is to make my OpenFaaS function work with a database (hosted in AWS) and explore more of the features. My focus is not to build a new amazing scaffold or to write production level code (I do enough of that on the day job).
Thus I will be using the python3-http template as is and just do all my API routing from the handler.py.
Alex’s suggested to just use the path and also gave a good Go example.
I would however have loved to something like this in handler.py, using Flask’s routing etc.
# def handle(event, context):
# return {
# "statusCode": 200,
# "body": "Hello from OpenFaaS!"
# }
@app.route('/')
def index():
return 'index'
@app.route('/login')
def login():
return 'login'
Maybe that is a contribution project for the future?
Right, time to crack on with it.
$ mkdir legodb
$ cd legodb
$ faas-cli template store pull python3-flask
$ faas-cli new --lang python3-http legodb
...
___ _____ ____
/ _ \ _ __ ___ _ __ | ___|_ _ __ _/ ___|
| | | | '_ \ / _ \ '_ \| |_ / _` |/ _` \___ \
| |_| | |_) | __/ | | | _| (_| | (_| |___) |
\___/| .__/ \___|_| |_|_| \__,_|\__,_|____/
|_|
Function created in folder: legodb
Stack file written: legodb.yml
$ faas-cli up -f legodb.yml
...
Deployed. 200 OK.
URL: http://192.168.64.4:9999/function/legodb
$ curl -i http://192.168.64.4:9999/function/legodb
HTTP/1.1 200 OK
Content-Length: 20
Content-Type: text/html; charset=utf-8
Date: Wed, 17 Mar 2021 16:58:47 GMT
Server: waitress
X-Call-Id: 137d08b0-7b90-4a87-94ea-994e24ed0a4d
X-Duration-Seconds: 0.002660
X-Start-Time: 1616000327192323869
Hello from OpenFaaS!
One nice thing I noticed was that running the template actually added a .gitignore file and I don’t have to figure out what files to commit and which to ignore.
Edit "legodb/Dockerfile". I am shortcutting this a bit by looking at the python3-http template instead of just blindly following the tutorial.
Initial printf style debugging
In order to understand what our handler.py function is given when a request comes in, I will be returning some values as a response (ok so I lied about printf).
Looking inside of the template’s index.py you will see how the event instance is created.
# Our handler.py
import json
def handle(event, context):
return {
'statusCode': 200,
'body': {
'headers': str(event.headers),
'method': str(event.method),
'path': str(event.path),
'query': str(event.query),
'body': str(event.body)
}
}
Explorer mode activated.
$ faas-cli up -f legodb.yml
$ curl -s http://192.168.64.4:9999/function/legodb | jq
{
"body": "b''",
"headers": "Host: 10.62.0.68:8080\r\nUser-Agent: curl/7.64.1\r\nAccept: */*\r\nAccept-Encoding: gzip\r\nX-Call-Id: 7f167e06-d757-43b1-9164-c4802d5d37bd\r\nX-Forwarded-For: 10.62.0.1:56786\r\nX-Forwarded-Host: 192.168.64.4:9999\r\nX-Start-Time: 1616010611817968042\r\n\r\n",
"method": "GET",
"path": "/",
"query": "ImmutableMultiDict([])"
}
# Path and Query
$ curl -s "http://192.168.64.4:9999/function/legodb/abc/xyz?param1=value1" | jq
...
"path": "/abc/xyz",
"query": "ImmutableMultiDict([('param1', 'value1')])"
# Post JSON data
$ curl -s -H "Content-Type: application/json" -d '{"a":"apple"}' \
"http://192.168.64.4:9999/function/legodb/abc/xyz?param1=value1" | jq
{
"body": "b'{\"a\":\"apple\"}'",
"headers": "... Content-Type: application/json ...",
"method": "POST",
"path": "/abc/xyz",
"query": "ImmutableMultiDict([('param1', 'value1')])"
}
If it is a post, can we decode from JSON to a Python dictionary? (We will need this for adding a new entry to our lego database).
import json
def handle(event, context):
postJSON = None
if event.method == 'POST':
inputJSON = event.body.decode('utf8').replace("'", '"')
postJSON = json.loads(inputJSON)
return {
'statusCode': 200,
'body': {
'headers': str(event.headers),
'method': str(event.method),
'path': str(event.path),
'query': str(event.query),
'body': str(event.body),
'postJSON': postJSON
}
}
$ faas-cli up -f legodb.yml
$ curl -s -H "Content-Type: application/json" -d '{"a":"apple", "meaning":42}' \
"http://192.168.64.4:9999/function/legodb/abc/xyz?param1=value1" | jq
{
"body": "b'{\"a\":\"apple\", \"meaning\":42}'",
"headers": ...
"method": "POST",
"path": "/abc/xyz",
"postJSON": {
"a": "apple",
"meaning": 42
},
"query": "ImmutableMultiDict([('param1', 'value1')])"
}
Ok we are in a good enough place now to write the initial API handler.
Hacked Handling
Hacked? why hacked? Well I am not planning on writing production level code here but just enough code to hopefully not blow up.
Here is the first pass of having a mocked API.
# handler.py
import json
# GET /legosets : Returns the list of lego sets
# POST /legoset : Add a new lego set to the database
def handle(event, context):
response = {'statusCode': 400}
if event.method == 'GET':
if event.path == '/legosets':
legosets = get_list_of_legosets()
response = {'statusCode': 200,
'body': legosets,
'headers': {'Content-Type': 'application/json'}
}
elif event.method == 'POST':
if event.path == '/legoset':
response = add_new_legoset(event.body)
return response
def get_list_of_legosets():
set1 = {'LegoID': 21322,
'Description': 'Pirates of Barracuda Bay',
'ProductURL':'',
'ImageURL': ''
}
return {"sets": [set1]}
def add_new_legoset(body):
response = None
try:
inputJSON = json.loads(body.decode('utf8').replace("'", '"'))
response = {
'statusCode': 200,
'body': {'received': inputJSON},
'headers': {'Content-Type': 'application/json'}
}
except ValueError:
response = {
'statusCode': 400,
'body': {'reason': 'Invalid JSON'},
'headers': {'Content-Type': 'application/json'}
}
return response
Deploy and test various scenarios.
$ faas-cli up -f legodb.yml
# Invalid path test
$ curl -i http://192.168.64.4:9999/function/legodb/invalid/path
HTTP/1.1 400 Bad Request
...
# Unsupported verb test
$ curl -i -X "DELETE" http://192.168.64.4:9999/function/legodb/legoset/42
HTTP/1.1 400 Bad Request
# Not production level worries test
# Path is correct but has the trailing /
$ curl -i http://192.168.64.4:9999/function/legodb/legosets/
HTTP/1.1 400 Bad Request
# Valid GET test
$ curl -s http://192.168.64.4:9999/function/legodb/legosets | jq
{
"sets": [
{
"Description": "Pirates of Barracuda Bay",
"ImageURL": "",
"LegoID": 21322,
"ProductURL": ""
}
]
}
# Post of invalid JSON test
$ curl -s -H "Content-Type: application/json" -d 'this is not json' "http://192.168.64.4:9999/function/legodb/legoset" | jq
{
"reason": "Invalid JSON"
}
# Valid POST of some JSON input test
$ curl -s -H "Content-Type: application/json" -d '{"a":"apple"}' \
"http://192.168.64.4:9999/function/legodb/legoset" | jq
{
"received": {
"a": "apple"
}
}
Learning action point: I recall Alex had an example in Serverless for Everyone Else for unit-testing a function. Need to see how I can unit-test the python based functions.
Cleaning up some of the unused functions
My faasd instance is starting to get a bit crowded and I guess now is a good time to learn how to remove unused functions.
$ faas-cli list
Function Invocations Replicas
cows 1 1
expose 1 1
flasktest 9 0
helloworld 0 1
iss-location 0 1
legodb 58 1
myfunction 0 1
nodeinfo 0 1
ping 15 1
pong 18 1
In Chapter 11 of Serverless for Everyone Else there is very good information on how to clean up old docker images on your instance, how to check the resource usage of faasd, functions as well as a pruning script that you can schedule on cron.
Deleting a function
$ faas-cli remove cows
Deleting: cows.
Removing old function.
...
$ faas-cli list
Function Invocations Replicas
legodb 58 1
nodeinfo 0 1
Next plans
Tomorrow I will start looking at what it would take to get SQLalchemy setup as a dependency and connected to the MariaDB running on AWS RDS. Will also need to install mysql-connector.