Photo by benjamin lehman<\/a> on Unsplash<\/a><\/p>\n\n\n\n
Here is my Log book<\/a>.<\/p>\n
Git repository<\/a> for the functions.<\/p>\n
Continuing on from yesterday where I had a mocked API in place, the next phase is to add storage for the lego database.<\/p>\n
I will be using SQLAlchemy<\/a> as the ORM layer and this will connect to the MariaDB instance I have setup in AWS RDS. In other to connect to MariaDB I will need to use mysql-connector<\/s> MariaDB connector<\/a><\/p>\n
Instead of doing trial and error development, I want to develop and test the Python code locally on my machine first and once I know it works it can be transferred to the OpenFaaS code.<\/p>\n
$ cd legodb\/legodb\n$ ls\nhandler.py requirements.txt\n\n# Install and activate a virtual environment\n$ python3 -m venv .\/venv\n$ source venv\/bin\/activate\n(venv)$\n\n(venv)$ python3 -m pip install --upgrade pip\n<\/code><\/pre>\n
Installing mariadb-connector-c and mariadb (connector)<\/h3>\n
(venv)$ brew install mariadb-connector-c\n(venv)$ pip install mariadb\n\nCollecting mariadb\n Using cached mariadb-1.0.6.tar.gz (67 kB)\nUsing legacy 'setup.py install' for mariadb, since package 'wheel' is not installed.\nInstalling collected packages: mariadb\n Running setup.py install for mariadb ... done\nSuccessfully installed mariadb-1.0.6\n<\/code><\/pre>\n
The MariaDB connector for Python requires the MariaDB connector for C to be installed first. This is going to be a learning curve later to see how I get the faas-cli + template + faasd to install these dependencies (including C ones on basically an Ubuntu image)<\/p>\n
Make a test connection<\/h3>\n
First I am testing to see if the connector is working and that I can connect to AWS database.<\/p>\n
(venv)$ python\n>>> import mariadb\n>>> connection = mariadb.connect(user="admin", \\\npassword="DA_PASSWORD", \\\nhost="xyzabc.rds.amazonaws.com", \\\nport=3306,database="openfaasdb")\n>>> cursor = connection.cursor()\n>>> cursor.execute("SELECT Description FROM openfaasdb.legosets")\n>>> for description in cursor:\n... print(description)\n...\n('Pirates of Barracuda Bay',)\n('Medieval Blacksmith',)\n>>> exit() # This is a bit like the "how do you quit vim?"\n<\/code><\/pre>\n
Nice! Ok next test is to add it to the handler.py and see if we can get the dependencies to install correctly on faasd.<\/p>\n
Configure function and dependencies<\/h2>\n
# Update the requirments.txt to include the packages we have installed locally\n(venv)$ pip freeze > requirements.txt\n(venv)$ cat requirements.txt\nmariadb==1.0.6\n<\/code><\/pre>\n
I am 100% sure that running faas-cli up will explode because we need a way to also specify that mariadb-connector-c be installed. If faas-cli doesn’t explode then faasd surely will.<\/p>\n
$ faas-cli up -f legodb.yml\n...\n#22 [stage-1 14\/18] COPY function\/requirements.txt\t.\n#22 sha256:333b1ca8a3738bc696305e1b571339f0dd690a63f9211340f23111e9be70c1ed\n#22 DONE 0.2s\n\n#23 [stage-1 15\/18] RUN pip install --user -r requirements.txt\n#23 sha256:bc0ee19013bf0423e61eb77d7d2f6e9ae7fc90b25b9557e7d451250819bb3c0d\n#23 1.848 Collecting mariadb==1.0.6\n#23 2.017 Downloading mariadb-1.0.6.tar.gz (67 kB)\n#23 2.286 ERROR: Command errored out with exit status 1:\n...\n#23 2.286 OSError: mariadb_config not found.\n#23 2.286\n#23 2.286 Please make sure, that MariaDB Connector\/C is installed on your system.\n#\n<\/code><\/pre>\n
Ok that confirms at least that during faas-cli build it will explode.<\/p>\n
Down the rabbit hole<\/h3>\n
My suspicion earlier was spot on that it was going to be a learning curve to figure out how to get other non-python dependencies working.<\/p>\n
I read a number of docs and tutorials and even came across a github issue for OpenFaaS that described you can pass values to the template’s Dockerfile from you yaml file. But sometimes you will read a lot of things and the brain has not yet made all the required connections.<\/p>\n
Below I will describe the process I went through to figure this out.<\/p>\n
# 1st attempt, I edited template.yml\nbuild_options:\n - name: dev\n packages:\n ... \n - mariadb-connector-c\n# Failed in the same way about not being able to find mariadb_config\n\n# 2nd attempt, I edited legodb.yml\nfunctions:\n legodb:\n ...\n build_args:\n # ADDITIONAL_PACKAGE is used in the Dockerfile\n ADDITIONAL_PACKAGE: mariadb-connector-c\n# Failed in the same way\n\n# 3rd attempt, I found a StackOverflow post about that you might also need mariadb-dev\n ADDITIONAL_PACKAGE: mariadb-dev mariadb-connector-c\n# Ok we are getting somewhere, the error is now about not being able to find gcc\n\n# 4th attempt\n ADDITIONAL_PACKAGE: gcc mariadb-dev mariadb-connector-c\n# Now the error is about gcc not being able to find limits.h header file\n<\/code><\/pre>\n
Clearly I need help here. So I joined the OpenFaaS Slack<\/a> and asked for help on this. Also I called it quits at this stage for the night.<\/p>\n
Next day I got a reply from Richard Gee<\/a> saying that the build_options would be a better place for this and pointed me to the docs<\/a>.<\/p>\n
# template.yml\nlanguage: python3-http\nfprocess: python index.py\nbuild_options:\n - name: dev\n packages: \n # ... Added the following 2 packages\n - mariadb-dev\n - mariadb-connector-c\n\n$ faas-cli up --build-option dev -f legodb.yml\n# This worked!\n<\/code><\/pre>\n
Getting the list of Lego sets from MariaDB<\/h2>\n
First we need to start using secrets and environment variables to pass the database details to the OpenFaaS function.<\/p>\n
version: 1.0\nprovider:\n name: openfaas\n gateway: http:\/\/192.168.64.4:9999\nfunctions:\n legodb:\n lang: python3-http\n handler: .\/legodb\n image: andrejacobs42\/legodb:latest\n environment:\n database-name: openfaasdb\n database-port: 3306\n secrets:\n - database-host\n - database-user\n - database-password\n<\/code><\/pre>\n
Supply the secrets to the faasd instance.<\/p>\n
# Copy the hostname and then create secret\n$ pbpaste | faas-cli secret create database-host\nCreating secret: database-host\nCreated: 200 OK\n\n# Copy the username and then create secret\n$ pbpaste | faas-cli secret create database-user\n\n# Copy the password and then create secret\n$ pbpaste | faas-cli secret create database-password\n<\/code><\/pre>\n
I will be using MariaDB directly to query the database.<\/p>\n
import os\nimport json\nimport mariadb\n\n# GET \/legosets : Returns the list of lego sets\n# POST \/legoset : Add a new lego set to the database\n\ndef handle(event, context):\n response = {'statusCode': 400}\n\n if event.method == 'GET':\n if event.path == '\/legosets':\n legosets = get_list_of_legosets()\n response = {'statusCode': 200, \n 'body': legosets,\n 'headers': {'Content-Type': 'application\/json'}\n }\n elif event.method == 'POST':\n if event.path == '\/legoset':\n response = add_new_legoset(event.body)\n \n return response\n\ndef load_secret(name):\n filepath = os.path.join('\/var\/openfaas\/secrets\/', name)\n with open(filepath) as f:\n secret = f.read()\n return secret\n\ndef database_connection():\n host = load_secret('database-host')\n user = load_secret('database-user')\n password = load_secret('database-password')\n database_name = os.environ.get('database-name')\n database_port = int(os.environ.get('database-port', '3306'))\n\n try:\n connection = mariadb.connect(\n user=user,\n password=password,\n host=host,\n port=database_port,\n database=database_name\n )\n return connection\n except mariadb.Error as error:\n print(f"Error: {error}")\n return None\n\ndef get_list_of_legosets():\n try:\n connection = database_connection()\n cursor = connection.cursor()\n cursor.execute("SELECT LegoID, Description FROM legosets")\n result = []\n for legoID, description in cursor:\n result.append({ "legoID": legoID, "description": description})\n return {"sets": result}\n except mariadb.Error as error:\n print(f"Error{error}")\n return {"error": f"{error}"}\n\ndef add_new_legoset(body):\n response = None\n try:\n inputJSON = json.loads(body.decode('utf8').replace("'", '"'))\n response = {\n 'statusCode': 200,\n 'body': {'received': inputJSON},\n 'headers': {'Content-Type': 'application\/json'}\n }\n except ValueError:\n response = {\n 'statusCode': 400,\n 'body': {'reason': 'Invalid JSON'},\n 'headers': {'Content-Type': 'application\/json'}\n }\n return response\n<\/code><\/pre>\n
Build and test<\/p>\n
$ faas-cli up --build-option dev -f legodb.yml\n$ curl -s http:\/\/192.168.64.4:9999\/function\/legodb\/legosets | jq\n{\n "sets": [\n {\n "description": "Pirates of Barracuda Bay",\n "legoID": 21322\n },\n {\n "description": "Medieval Blacksmith",\n "legoID": 21325\n }\n ]\n}\n<\/code><\/pre>\n
<\/p>\n
Next actions<\/h3>\n
Tomorrow I will start exploring the use of SQLAlchemy<\/a>.<\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"