{"id":540,"date":"2022-03-26T20:29:22","date_gmt":"2022-03-26T20:29:22","guid":{"rendered":"https:\/\/andrejacobs.org\/?p=540"},"modified":"2022-04-11T20:22:58","modified_gmt":"2022-04-11T20:22:58","slug":"installing-prometheus-on-ubuntu-20-04","status":"publish","type":"post","link":"https:\/\/andrejacobs.org\/linux\/installing-prometheus-on-ubuntu-20-04\/","title":{"rendered":"Installing Prometheus on Ubuntu 20.04"},"content":{"rendered":"\n<div class=\"wp-block-jetpack-markdown\"><p>I <a href=\"https:\/\/andrejacobs.org\/linux\/spinning-down-seagate-hard-drives-in-ubuntu-20-04\/\">noticed that my hard drives<\/a> in my Ubuntu server was not spinning down after being idle and thus I needed a way to monitor their power states and to verify through trial and error if any of my attempts is working or not. Enter Prometheus to the rescue!<\/p>\n<p>I will be following this <a href=\"https:\/\/linoxide.com\/how-to-install-prometheus-on-ubuntu\/\">guide<\/a> to start with.<\/p>\n<p>This article will cover the following:<\/p>\n<ul>\n<li>Install Prometheus on Ubuntu 20.04.<\/li>\n<li>Configure Prometheus to be run as a systemd service.<\/li>\n<li>Securing Prometheus with basic authentication and TLS.<\/li>\n<li>Install Prometheus Node Exporter and run it as a systemd service.<\/li>\n<li>Enable the textfile collector and write our first script to output metrics.<\/li>\n<li>Use the community provided smartmon script. [I ended up not trusting this in the end].<\/li>\n<li>Roll my own script that uses openSeaChest to output metrics.<\/li>\n<\/ul>\n<h2>Prerequisites<\/h2>\n<ul>\n<li>Update system.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ sudo apt update &amp;&amp; sudo apt upgrade\n# Optionally remove packages no longer used\n$ sudo apt autoremove\n<\/code><\/pre>\n<ul>\n<li>Create required directories.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ sudo mkdir -p \/etc\/prometheus\n$ sudo mkdir -p \/var\/lib\/prometheus\n<\/code><\/pre>\n<ul>\n<li>Check what is the latest Linux release from: <a href=\"https:\/\/prometheus.io\/download\/\">https:\/\/prometheus.io\/download\/<\/a> and copy the link.<\/li>\n<\/ul>\n<h2>Download and Install<\/h2>\n<ul>\n<li>Download and extract.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ wget https:\/\/github.com\/prometheus\/prometheus\/releases\/download\/v2.34.0\/prometheus-2.34.0.linux-amd64.tar.gz\n$ tar -xvf prometheus-2.34.0.linux-amd64.tar.gz\n<\/code><\/pre>\n<ul>\n<li>Copy the binaries and other files into place.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ cd prometheus-2.34.0.linux-amd64\n$ sudo cp prometheus promtool \/usr\/local\/bin\/\n$ sudo cp -r consoles\/ console_libraries\/ \/etc\/prometheus\/\n$ sudo cp prometheus.yml \/etc\/prometheus\/prometheus.yml\n<\/code><\/pre>\n<ul>\n<li>Verify.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ cd ~\n$ prometheus --version\n\nprometheus, version 2.34.0 (branch: HEAD, revision: 881111fec4332c33094a6fb2680c71fffc427275)\n  build user:       root@121ad7ea5487\n  build date:       20220315-15:18:00\n  go version:       go1.17.8\n  platform:         linux\/amd64\n\n$ promtool --version\n\npromtool, version 2.34.0 (branch: HEAD, revision: 881111fec4332c33094a6fb2680c71fffc427275)\n  build user:       root@121ad7ea5487\n  build date:       20220315-15:18:00\n  go version:       go1.17.8\n  platform:         linux\/amd64\n<\/code><\/pre>\n<h2>Configure permissions<\/h2>\n<ul>\n<li>Create a group and user specifically to be used by Prometheus<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ sudo groupadd --system prometheus\n$ sudo useradd -s \/sbin\/nologin --system -g prometheus prometheus\n<\/code><\/pre>\n<ul>\n<li>Set permissions.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ sudo chown -R prometheus:prometheus \/etc\/prometheus\/ \/var\/lib\/prometheus\/\n$ sudo chmod -R 775 \/etc\/prometheus\/ \/var\/lib\/prometheus\/\n<\/code><\/pre>\n<h2>Configure Prometheus to be run as a systemd service<\/h2>\n<ul>\n<li>Create the .service file<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ sudo vi \/etc\/systemd\/system\/prometheus.service\n\n# Add the following\n[Unit]\nDescription=Prometheus\nWants=network-online.target\nAfter=network-online.target\n\n[Service]\nUser=prometheus\nGroup=prometheus\nRestart=always\nType=simple\nExecStart=\/usr\/local\/bin\/prometheus \\\n    --config.file=\/etc\/prometheus\/prometheus.yml \\\n    --storage.tsdb.path=\/var\/lib\/prometheus\/ \\\n    --web.console.templates=\/etc\/prometheus\/consoles \\\n    --web.console.libraries=\/etc\/prometheus\/console_libraries \\\n    --web.listen-address=0.0.0.0:9090\n\n[Install]\nWantedBy=multi-user.target\n<\/code><\/pre>\n<p>I would recommend you first check which ports are already in use and assign maybe a less obvious port number. To see what ports are being used to listen on, run <code>netstat -tulpn | grep LISTEN<\/code><\/p>\n<ul>\n<li>Edit the file <code>\/etc\/prometheus\/prometheus.yml<\/code> to suit your purposes.<\/li>\n<li>Enable the service to start at boot.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ sudo systemctl enable prometheus\n<\/code><\/pre>\n<ul>\n<li>Start Prometheus and verify.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ sudo systemctl start prometheus\n$ sudo systemctl status prometheus\n...\n\u25cf prometheus.service - Prometheus\n     Loaded: loaded (\/etc\/systemd\/system\/prometheus.service; enabled; vendor preset: enabled)\n     Active: active (running) since Tue 2022-03-22 16:34:37 UTC; 15s ago\n   Main PID: 57191 (prometheus)\n      Tasks: 10 (limit: 76945)\n     Memory: 19.4M\n<\/code><\/pre>\n<h3>Allow access from firewall<\/h3>\n<p>I use UFW so will allow access to the service only from the specific NIC and ip-port.<\/p>\n<pre><code class=\"language-bash\"># Get the name of the NIC to use\n$ ip a\n...\n2: enp2s0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; ...\n\n# port 9090 here is what prometheus was configured on\n$ sudo ufw allow in on enp2s0 proto tcp to 192.168.x.x port 9090\n<\/code><\/pre>\n<ul>\n<li>Open the dashboard in your browser (e.g. http:\/\/192.168.x.x:9090)<\/li>\n<\/ul>\n<p><img decoding=\"async\" src=\"https:\/\/andrejacobs.org\/wp-content\/uploads\/2022\/03\/Untitled.png\" alt=\"\"><\/p>\n<h2>Securing Prometheus<\/h2>\n<p>Next I will be setting up Basic Auth following the <a href=\"https:\/\/prometheus.io\/docs\/guides\/basic-auth\/\">official guide<\/a>.<\/p>\n<ul>\n<li>Generate a password hash using bcrypt. For this we need to install python3-bcrypt.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ sudo apt install python3-bcrypt\n<\/code><\/pre>\n<ul>\n<li>Create a script called <code>gen-bcrypt-password.py<\/code>.<\/li>\n<\/ul>\n<pre><code class=\"language-python\">import getpass\nimport bcrypt\n\npassword = getpass.getpass(&quot;password: &quot;)\nhashed_password = bcrypt.hashpw(password.encode(&quot;utf-8&quot;), bcrypt.gensalt())\nprint(hashed_password.decode())\n<\/code><\/pre>\n<ul>\n<li>Run the script and enter a password to be hashed.<\/li>\n<\/ul>\n<pre><code class=\"language-python\">$ python3 gen-bcrypt-password.py\n\npassword:\n$2b$12$z2shPn1IGtuyj7CL3E1Tb.dnci.HA7KXxFFysng3rWViTpaZK0LMS\n<\/code><\/pre>\n<ul>\n<li>Create the <code>\/etc\/prometheus\/web.yml<\/code> config file. The username in the example is admin.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ sudo vi \/etc\/prometheus\/web.yml\n<\/code><\/pre>\n<pre><code class=\"language-yaml\">basic_auth_users:\n    admin: $2b$12$z2shPn1IGtuyj7CL3E1Tb.dnci.HA7KXxFFysng3rWViTpaZK0LMS\n<\/code><\/pre>\n<ul>\n<li>Verify the config file.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\"># Change the permissions\n$ sudo chown prometheus:prometheus \/etc\/prometheus\/web.yml\n$ sudo chmod 640 \/etc\/prometheus\/web.yml\n\n$ sudo promtool check web-config \/etc\/prometheus\/web.yml\n\/etc\/prometheus\/web.yml SUCCESS\n<\/code><\/pre>\n<ul>\n<li>Change the systemd service to start Prometheus using this new web.yml config file.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ sudo vi \/etc\/systemd\/system\/prometheus.service\n\n# Modify this setting to include\nExecStart=\n...\n# Add this bit (ensure you add \\ to the previous line)\n    --web.config.file=\/etc\/prometheus\/web.yml\n\n# Restart the service\n$ sudo systemctl daemon-reload\n$ sudo systemctl restart prometheus\n$ sudo systemctl status prometheus\n<\/code><\/pre>\n<ul>\n<li>Verify.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ curl --head http:\/\/localhost:9090\/graph\nHTTP\/1.1 401 Unauthorized\n\n# Open the dashboard in your browser again and you should be\n# asked for username and password.\n<\/code><\/pre>\n<h3>Enable TLS<\/h3>\n<p>I will be using a self signed certificate for my server since it is only accessible from the local network.<\/p>\n<p>I found this <a href=\"https:\/\/medium.com\/@antelle\/how-to-generate-a-self-signed-ssl-certificate-for-an-ip-address-f0dd8dddf754\">article<\/a> showing how you can generate a self signed certificate using an IP address.<\/p>\n<ul>\n<li>Create a script called <code>generate-ip-cert.sh<\/code>.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">#!\/bin\/sh\n# Generate a self signed certificate using the IP\n# Based on: https:\/\/raw.githubusercontent.com\/antelle\/generate-ip-cert\/master\/generate-ip-cert.sh\n\nIP=$(echo $1 | egrep -o &quot;^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}$&quot;)\n\nif [ ! $IP ]\nthen\n    echo &quot;Usage: generate-ip-cert.sh 127.0.0.1&quot;\n    exit 1\nfi\n\necho &quot;[req]\ndefault_bits = 4096\ndistinguished_name = req_distinguished_name\nreq_extensions = req_ext\nx509_extensions = v3_req\nprompt = no\n\n[req_distinguished_name]\ncountryName = XX\nstateOrProvinceName = N\/A\nlocalityName = N\/A\norganizationName = Self-signed certificate\ncommonName = $IP: Self-signed certificate\n\n[req_ext]\nsubjectAltName = @alt_names\n\n[v3_req]\nsubjectAltName = @alt_names\n\n[alt_names]\nIP.1 = $IP\n&quot; &gt; san.cnf\n\nopenssl req -x509 -nodes -days 3650 -newkey rsa:4096 -keyout key.pem -out cert.pem -config san.cnf\nrm san.cnf\n<\/code><\/pre>\n<ul>\n<li>Run the script and move the files to \/etc\/prometheus.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ .\/generate-ip-cert.sh 192.168.x.x\n\n$ sudo mv cert.pem key.pem \/etc\/prometheus\n$ sudo chown prometheus:prometheus \/etc\/prometheus\/*.pem\n<\/code><\/pre>\n<ul>\n<li>Edit the \/etc\/prometheus\/web.yml file and add the following at the beginning.<\/li>\n<\/ul>\n<pre><code class=\"language-yaml\">tls_server_config:\n  cert_file: \/etc\/prometheus\/cert.pem\n  key_file: \/etc\/prometheus\/key.pem\n<\/code><\/pre>\n<ul>\n<li>Restart prometheus and verify.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ sudo systemctl daemon-reload\n$ sudo systemctl restart prometheus\n<\/code><\/pre>\n<p>You will have issues getting your browser to trust the self signed certificate. What I have done is to copy the cert.pem file over to my Mac and import it into Keychain. Then \u201cGet Info\u201d on the certificate and change the trust settings to \u201cAlways Trust\u201d.<\/p>\n<p>Now I can open <a href=\"https:\/\/192.168.x.x:9090\">https:\/\/192.168.x.x:9090<\/a> in my browser and I am asked for the username and password and the site shows that it is protected by TLS.<\/p>\n<h2>Gather the first metrics<\/h2>\n<p>To gather some system metrics I will be installing and using the Prometheus Node Exporter following the <a href=\"https:\/\/prometheus.io\/docs\/guides\/node-exporter\/\">official guide<\/a>.<\/p>\n<ul>\n<li>Copy the link to the latest release from <a href=\"https:\/\/prometheus.io\/download\/#node_exporter\">here<\/a>.<\/li>\n<li>Download and extract.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ wget https:\/\/github.com\/prometheus\/node_exporter\/releases\/download\/v1.3.1\/node_exporter-1.3.1.linux-amd64.tar.gz\n$ tar xvfz node_exporter-*.*-amd64.tar.gz\n<\/code><\/pre>\n<ul>\n<li>Do a test run.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ cd node_exporter-*.*-amd64\n$ .\/node_exporter\n...\n# See some output.\n\n# Start a new SSH session and get the metrics from this new service\n$ curl http:\/\/localhost:9100\/metrics\n...\n# See some metric output.\n<\/code><\/pre>\n<ul>\n<li>Install node_exporter similar to how I installed prometheus.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ sudo cp node_exporter \/usr\/local\/bin\/\n$ sudo vi \/etc\/systemd\/system\/node_exporter.service\n<\/code><\/pre>\n<pre><code class=\"language-bash\">[Unit]\nDescription=Node Exporter\nAfter=network.target\n\n[Service]\nUser=prometheus\nGroup=prometheus\nType=simple\nExecStart=\/usr\/local\/bin\/node_exporter\n\n[Install]\nWantedBy=multi-user.target\n<\/code><\/pre>\n<ul>\n<li>Start the service.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ sudo systemctl daemon-reload\n$ sudo systemctl start node_exporter\n# Enable the service to start at boot\n$ sudo systemctl enable node_exporter\n\n# Check status\n$ sudo systemctl status node_exporter\n<\/code><\/pre>\n<ul>\n<li>Configure Prometheus to gather the metrics from node_exporter.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ sudo vi \/etc\/prometheus\/prometheus.yml\n\n# Add the following into the scape_configs section\n  - job_name: 'node_exporter_metrics'\n    scrape_interval: 1m\n    static_configs:\n      - targets: ['localhost:9100']\n\n$ sudo systemctl daemon-reload\n$ sudo systemctl restart prometheus\n<\/code><\/pre>\n<ul>\n<li>Open the Prometheus dashboard and try a query. For example: node_hwmon_temp_celsius gives me the temperature sensors.<\/li>\n<li>I don\u2019t need to gather all the default enabled metrics so instead I will disable all the collectors and only turn on the ones I am interested in. To do so you modify the systemd <code>node_exporter.service<\/code> file.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\"># Example of disabling all collectors and then specify each collector individually\nExecStart=\/usr\/local\/bin\/node_exporter \\\n    --collector.disable-defaults \\\n    --collector.&lt;name&gt;\n<\/code><\/pre>\n<h2>\u201cReal world\u201d usage: Check how often my drives are being spin up<\/h2>\n<p>While trying to figure out why my drives are no longer being put to sleep while idle, I came across <a href=\"https:\/\/www.taricorp.net\/2021\/hdd-parking-monitoring\/\">this article<\/a> from Peter Marheine which is what lead me to install and use Prometheus in the first place.<\/p>\n<p>In order to gather metrics from smartmon I will need to use the <code>textfile<\/code> collector from <code>node_exporter<\/code>.<\/p>\n<ul>\n<li>Create the location for where the textfile collector will read .prom files from.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ sudo mkdir -p \/var\/lib\/prometheus\/textfiles\n$ sudo chown prometheus:prometheus \/var\/lib\/prometheus\/textfiles\n<\/code><\/pre>\n<ul>\n<li>Enable the textfile collector and configure the directory to use.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ sudo vi \/etc\/systemd\/system\/node_exporter.service\n\n# Modify this value to include the textfile collector if needed\n# and you do need to set the textfile.directory\nExecStart=\/usr\/local\/bin\/node_exporter \\\n    --collector.textfile \\\n    --collector.textfile.directory=\/var\/lib\/prometheus\/textfiles\n\n# Reload service\n$ sudo systemctl daemon-reload\n$ sudo systemctl restart node_exporter\n$ sudo systemctl status node_exporter\n<\/code><\/pre>\n<ul>\n<li>Create the directory in which we will store the scripts\/binaries that produce textfiles.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ sudo mkdir -p \/usr\/local\/bin\/node_exporter_textfile_collector\n<\/code><\/pre>\n<h3>Trial run<\/h3>\n<ul>\n<li>Lets create a test script and see that the metrics are picked up. The script will generate a random number between 0 and 10 and store it under the name \u201ctest_hello_world_number\u201d.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ sudo vi \/usr\/local\/bin\/node_exporter_textfile_collector\/hello_world.sh\n\n#!\/bin\/bash\nMAGIC=`shuf -i 0-10 -n 1`\necho '# HELP test_hello_world_number Used for testing that the textfile collector is working'\necho '# TYPE test_hello_world_number gauge'\necho &quot;test_hello_world_number ${MAGIC}&quot;\n\n# Save and change permissions\n$ sudo chmod +x \/usr\/local\/bin\/node_exporter_textfile_collector\/hello_world.sh\n# Run the script a couple of times and you should see output like this\n\n# HELP test_hello_world_number Used for testing that the textfile collector is working\n# TYPE test_hello_world_number gauge\ntest_hello_world_number 4\n<\/code><\/pre>\n<ul>\n<li>Since we will be using the smartmon script provided by the <a href=\"https:\/\/github.com\/prometheus-community\/node-exporter-textfile-collector-scripts\">community<\/a> they recommend you use <code>sponge<\/code> to write the output files. Looking into this it is because you want to ensure a textfile\u2019s metrics is written completely before node_exporter starts reading metrics. I.e. atomically. To install sponge <code>sudo apt install moreutils<\/code>.<\/li>\n<li>Run the test script a couple of times and check the dashboard for the metrics. My Prometheus is configured to scrape from node_exporter once every minute, so I will have to wait at least a minute before results start showing up.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ sudo sh -c '\/usr\/local\/bin\/node_exporter_textfile_collector\/hello_world.sh | sponge \\\n\/var\/lib\/prometheus\/textfiles\/test_hello_world.prom'\n<\/code><\/pre>\n<p><img decoding=\"async\" src=\"https:\/\/andrejacobs.org\/wp-content\/uploads\/2022\/03\/Untitled-1.png\" alt=\"\"><\/p>\n<h3>Smartmon monitoring<\/h3>\n<ul>\n<li>Download the <a href=\"https:\/\/raw.githubusercontent.com\/prometheus-community\/node-exporter-textfile-collector-scripts\/master\/smartmon.sh\">community smartmon.sh<\/a> and store it as <code>\/usr\/local\/bin\/node_exporter_textfile_collector\/smartmon.sh<\/code>. Make it executable <code>sudo chmod +x<\/code><\/li>\n<li>Run the script as root to ensure smartmon metrics are reported.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ sudo \/usr\/local\/bin\/node_exporter_textfile_collector\/smartmon.sh\n...\n# Expect to see a lot of stats here. If you only get the version then the\n# user running the script does not have enough permissions for smartctlCreate a systemd timer to run the script every 5 minutes and to produce a textfile for node_exporter to pick up.\n<\/code><\/pre>\n<ul>\n<li>I will be using cron to schedule the script to be run every 5 minutes.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ sudo crontab -e\n\n# Export smartctl metrics to Prometheus\n*\/5 * * * * \/usr\/local\/bin\/node_exporter_textfile_collector\/smartmon.sh | sponge \/var\/lib\/prometheus\/textfiles\/smartmon.prom\n\n<\/code><\/pre>\n<ul>\n<li>Although this script captures a treasure trove of information it is however ironically keeping my disks active. Maybe a bug with Seagate drives and this script?<\/li>\n<\/ul>\n<h3>Time to roll my own script<\/h3>\n<p>See my other <a href=\"https:\/\/andrejacobs.org\/linux\/spinning-down-seagate-hard-drives-in-ubuntu-20-04\/\">article about openSeaChest<\/a> for more details on using Seagate\u2019s openSeaChest tools.<\/p>\n<p>Since I am using openSeaChest to correctly control my Seagate drives I will use <code>openSeaChest_PowerControl<\/code> to report on the power status and capture that to a textfile for node_exporter to pick up.<\/p>\n<ul>\n<li>First ensure that the binary can be accessed correctly.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ sudo cd openSeaChest\/builddir\n$ sudo mkdir -p \/usr\/local\/bin\/openSeaChest\n$ sudo cp openSeaChest_* \/usr\/local\/bin\/openSeaChest\/\n<\/code><\/pre>\n<ul>\n<li>By running openSeaChest_PowerControl, I discovered all the states I am interested in.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ sudo \/usr\/local\/bin\/openSeaChest\/openSeaChest_PowerControl \\\n-q -d \/dev\/... --checkPowerMode\n\n# This is the list of the various states\n\nDevice is in the PM0: Active state or PM1: Idle State\nDevice is in the PM2: Standby state and device is in the Standby_z power condition\nDevice is in the PM2: Standby state.\nDevice is in the PM1: Idle state and the device is in the Idle_a power condition\nDevice is in the PM1: Idle state and the device is in the Idle_b power condition\nDevice is in the PM1: Idle state and the device is in the Idle_c power condition\n<\/code><\/pre>\n<ul>\n<li>Create the script <code>\/usr\/local\/bin\/node_exporter_textfile_collector\/drive_powermode.sh<\/code><\/li>\n<\/ul>\n<pre><code class=\"language-bash\">#!\/bin\/bash\n# Check drive states and report it as metrics for the node exporter textfile collector\n\necho '# HELP andre_drive_powermode Report the power mode of the \/dev\/sd? drives'\necho '# TYPE andre_drive_powermode gauge'\n\nfunction checkPowerMode() {\n    local guage=1.0\n    local powerMode=$(\/usr\/local\/bin\/openSeaChest\/openSeaChest_PowerControl -q -d $1 --checkPowerMode)\n    \n    if echo $powerMode | grep -q -w 'Active'; then\n        guage=1.0\n    elif echo $powerMode | grep -q -w 'Standby'; then\n        guage=0.0\n    elif echo $powerMode | grep -q -w 'Idle_c'; then\n        guage=0.25\n    elif echo $powerMode | grep -q -w 'Idle_b'; then\n        guage=0.5\n    elif echo $powerMode | grep -q -w 'Idle_a'; then\n        guage=0.75\n    fi\n        \n    echo &quot;andre_drive_powermode{dev=\\&quot;$1\\&quot;} $guage&quot;\n}\n\nfor drive in \/dev\/sd? ; do\n    checkPowerMode $drive\ndone\n<\/code><\/pre>\n<ul>\n<li>Schedule the script to be run every 5 minutes and to produce the .prom file.<\/li>\n<\/ul>\n<pre><code class=\"language-bash\">$ sudo crontab -e\n\n# Export drive power mode metrics to Prometheus\n*\/5 * * * * \/usr\/local\/bin\/node_exporter_textfile_collector\/drive_powermode.sh | sponge \/var\/lib\/prometheus\/textfiles\/drive_powermode.prom\n<\/code><\/pre>\n<p>On the Prometheus dashboard I can already see that drives have started going into the various idle states.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/andrejacobs.org\/wp-content\/uploads\/2022\/03\/Untitled-2.png\" alt=\"\"><\/p>\n<p>After two days of trial and error I have finally managed to get the drives to spin down after about 25 minutes of inactivity.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/andrejacobs.org\/wp-content\/uploads\/2022\/03\/Untitled-3.png\" alt=\"\"><\/p>\n<h2>References:<\/h2>\n<p><a href=\"https:\/\/linoxide.com\/how-to-install-prometheus-on-ubuntu\/\">https:\/\/linoxide.com\/how-to-install-prometheus-on-ubuntu\/<\/a><\/p>\n<p><a href=\"https:\/\/medium.com\/@antelle\/how-to-generate-a-self-signed-ssl-certificate-for-an-ip-address-f0dd8dddf754\">https:\/\/medium.com\/@antelle\/how-to-generate-a-self-signed-ssl-certificate-for-an-ip-address-f0dd8dddf754<\/a><\/p>\n<p><a href=\"https:\/\/www.taricorp.net\/2021\/hdd-parking-monitoring\/\">https:\/\/www.taricorp.net\/2021\/hdd-parking-monitoring\/<\/a><\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p> [&#8230;]<\/p>\n<p><a class=\"understrap-read-more-link\" href=\"https:\/\/andrejacobs.org\/linux\/installing-prometheus-on-ubuntu-20-04\/\">Read more &rarr;<\/a><\/p>\n","protected":false},"author":2,"featured_media":538,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_mi_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[5],"tags":[49,50,47],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v19.13 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Installing Prometheus on Ubuntu 20.04 - Andr\u00e9 Jacobs<\/title>\n<meta name=\"description\" content=\"Let me show you how to install and secure Prometheus on Ubuntu 20.04 to gather analytics and monitor your server.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/andrejacobs.org\/linux\/installing-prometheus-on-ubuntu-20-04\/\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Installing Prometheus on Ubuntu 20.04 - Andr\u00e9 Jacobs\" \/>\n<meta property=\"og:description\" content=\"Let me show you how to install and secure Prometheus on Ubuntu 20.04 to gather analytics and monitor your server.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/andrejacobs.org\/linux\/installing-prometheus-on-ubuntu-20-04\/\" \/>\n<meta property=\"og:site_name\" content=\"Andr\u00e9 Jacobs\" \/>\n<meta property=\"article:published_time\" content=\"2022-03-26T20:29:22+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-04-11T20:22:58+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/andrejacobs.org\/wp-content\/uploads\/2022\/03\/Untitled-3.png\" \/>\n\t<meta property=\"og:image:width\" content=\"2796\" \/>\n\t<meta property=\"og:image:height\" content=\"1092\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Andr\u00e9 Jacobs\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@andrejacobs81\" \/>\n<meta name=\"twitter:site\" content=\"@andrejacobs81\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Andr\u00e9 Jacobs\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"11 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/andrejacobs.org\/linux\/installing-prometheus-on-ubuntu-20-04\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/andrejacobs.org\/linux\/installing-prometheus-on-ubuntu-20-04\/\"},\"author\":{\"name\":\"Andr\u00e9 Jacobs\",\"@id\":\"https:\/\/andrejacobs.org\/#\/schema\/person\/3d38360883015e883c80c2fb875c5a68\"},\"headline\":\"Installing Prometheus on Ubuntu 20.04\",\"datePublished\":\"2022-03-26T20:29:22+00:00\",\"dateModified\":\"2022-04-11T20:22:58+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/andrejacobs.org\/linux\/installing-prometheus-on-ubuntu-20-04\/\"},\"wordCount\":1061,\"publisher\":{\"@id\":\"https:\/\/andrejacobs.org\/#\/schema\/person\/3d38360883015e883c80c2fb875c5a68\"},\"keywords\":[\"Analytics\",\"Prometheus\",\"Ubuntu\"],\"articleSection\":[\"Linux\"],\"inLanguage\":\"en-GB\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/andrejacobs.org\/linux\/installing-prometheus-on-ubuntu-20-04\/\",\"url\":\"https:\/\/andrejacobs.org\/linux\/installing-prometheus-on-ubuntu-20-04\/\",\"name\":\"Installing Prometheus on Ubuntu 20.04 - Andr\u00e9 Jacobs\",\"isPartOf\":{\"@id\":\"https:\/\/andrejacobs.org\/#website\"},\"datePublished\":\"2022-03-26T20:29:22+00:00\",\"dateModified\":\"2022-04-11T20:22:58+00:00\",\"description\":\"Let me show you how to install and secure Prometheus on Ubuntu 20.04 to gather analytics and monitor your server.\",\"breadcrumb\":{\"@id\":\"https:\/\/andrejacobs.org\/linux\/installing-prometheus-on-ubuntu-20-04\/#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/andrejacobs.org\/linux\/installing-prometheus-on-ubuntu-20-04\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/andrejacobs.org\/linux\/installing-prometheus-on-ubuntu-20-04\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/andrejacobs.org\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Installing Prometheus on Ubuntu 20.04\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/andrejacobs.org\/#website\",\"url\":\"https:\/\/andrejacobs.org\/\",\"name\":\"Andr\u00e9 Jacobs\",\"description\":\"iOS Development, Electronics, Robotics, Artificial Intelligence, Business and Health\",\"publisher\":{\"@id\":\"https:\/\/andrejacobs.org\/#\/schema\/person\/3d38360883015e883c80c2fb875c5a68\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/andrejacobs.org\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-GB\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/andrejacobs.org\/#\/schema\/person\/3d38360883015e883c80c2fb875c5a68\",\"name\":\"Andr\u00e9 Jacobs\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/andrejacobs.org\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/andrejacobs.org\/wp-content\/uploads\/2019\/03\/andre_jacobs.jpg\",\"contentUrl\":\"https:\/\/andrejacobs.org\/wp-content\/uploads\/2019\/03\/andre_jacobs.jpg\",\"width\":450,\"height\":450,\"caption\":\"Andr\u00e9 Jacobs\"},\"logo\":{\"@id\":\"https:\/\/andrejacobs.org\/#\/schema\/person\/image\/\"},\"sameAs\":[\"https:\/\/andrejacobs.org\",\"https:\/\/www.youtube.com\/channel\/UCzqXvnd_UJ3sAzQkQBD-IVw\"],\"url\":\"https:\/\/andrejacobs.org\/author\/andre\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Installing Prometheus on Ubuntu 20.04 - Andr\u00e9 Jacobs","description":"Let me show you how to install and secure Prometheus on Ubuntu 20.04 to gather analytics and monitor your server.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/andrejacobs.org\/linux\/installing-prometheus-on-ubuntu-20-04\/","og_locale":"en_GB","og_type":"article","og_title":"Installing Prometheus on Ubuntu 20.04 - Andr\u00e9 Jacobs","og_description":"Let me show you how to install and secure Prometheus on Ubuntu 20.04 to gather analytics and monitor your server.","og_url":"https:\/\/andrejacobs.org\/linux\/installing-prometheus-on-ubuntu-20-04\/","og_site_name":"Andr\u00e9 Jacobs","article_published_time":"2022-03-26T20:29:22+00:00","article_modified_time":"2022-04-11T20:22:58+00:00","og_image":[{"width":2796,"height":1092,"url":"https:\/\/andrejacobs.org\/wp-content\/uploads\/2022\/03\/Untitled-3.png","type":"image\/png"}],"author":"Andr\u00e9 Jacobs","twitter_card":"summary_large_image","twitter_creator":"@andrejacobs81","twitter_site":"@andrejacobs81","twitter_misc":{"Written by":"Andr\u00e9 Jacobs","Estimated reading time":"11 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/andrejacobs.org\/linux\/installing-prometheus-on-ubuntu-20-04\/#article","isPartOf":{"@id":"https:\/\/andrejacobs.org\/linux\/installing-prometheus-on-ubuntu-20-04\/"},"author":{"name":"Andr\u00e9 Jacobs","@id":"https:\/\/andrejacobs.org\/#\/schema\/person\/3d38360883015e883c80c2fb875c5a68"},"headline":"Installing Prometheus on Ubuntu 20.04","datePublished":"2022-03-26T20:29:22+00:00","dateModified":"2022-04-11T20:22:58+00:00","mainEntityOfPage":{"@id":"https:\/\/andrejacobs.org\/linux\/installing-prometheus-on-ubuntu-20-04\/"},"wordCount":1061,"publisher":{"@id":"https:\/\/andrejacobs.org\/#\/schema\/person\/3d38360883015e883c80c2fb875c5a68"},"keywords":["Analytics","Prometheus","Ubuntu"],"articleSection":["Linux"],"inLanguage":"en-GB"},{"@type":"WebPage","@id":"https:\/\/andrejacobs.org\/linux\/installing-prometheus-on-ubuntu-20-04\/","url":"https:\/\/andrejacobs.org\/linux\/installing-prometheus-on-ubuntu-20-04\/","name":"Installing Prometheus on Ubuntu 20.04 - Andr\u00e9 Jacobs","isPartOf":{"@id":"https:\/\/andrejacobs.org\/#website"},"datePublished":"2022-03-26T20:29:22+00:00","dateModified":"2022-04-11T20:22:58+00:00","description":"Let me show you how to install and secure Prometheus on Ubuntu 20.04 to gather analytics and monitor your server.","breadcrumb":{"@id":"https:\/\/andrejacobs.org\/linux\/installing-prometheus-on-ubuntu-20-04\/#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/andrejacobs.org\/linux\/installing-prometheus-on-ubuntu-20-04\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/andrejacobs.org\/linux\/installing-prometheus-on-ubuntu-20-04\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/andrejacobs.org\/"},{"@type":"ListItem","position":2,"name":"Installing Prometheus on Ubuntu 20.04"}]},{"@type":"WebSite","@id":"https:\/\/andrejacobs.org\/#website","url":"https:\/\/andrejacobs.org\/","name":"Andr\u00e9 Jacobs","description":"iOS Development, Electronics, Robotics, Artificial Intelligence, Business and Health","publisher":{"@id":"https:\/\/andrejacobs.org\/#\/schema\/person\/3d38360883015e883c80c2fb875c5a68"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/andrejacobs.org\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-GB"},{"@type":["Person","Organization"],"@id":"https:\/\/andrejacobs.org\/#\/schema\/person\/3d38360883015e883c80c2fb875c5a68","name":"Andr\u00e9 Jacobs","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/andrejacobs.org\/#\/schema\/person\/image\/","url":"https:\/\/andrejacobs.org\/wp-content\/uploads\/2019\/03\/andre_jacobs.jpg","contentUrl":"https:\/\/andrejacobs.org\/wp-content\/uploads\/2019\/03\/andre_jacobs.jpg","width":450,"height":450,"caption":"Andr\u00e9 Jacobs"},"logo":{"@id":"https:\/\/andrejacobs.org\/#\/schema\/person\/image\/"},"sameAs":["https:\/\/andrejacobs.org","https:\/\/www.youtube.com\/channel\/UCzqXvnd_UJ3sAzQkQBD-IVw"],"url":"https:\/\/andrejacobs.org\/author\/andre\/"}]}},"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"https:\/\/andrejacobs.org\/wp-content\/uploads\/2022\/03\/Untitled-3.png","_links":{"self":[{"href":"https:\/\/andrejacobs.org\/wp-json\/wp\/v2\/posts\/540"}],"collection":[{"href":"https:\/\/andrejacobs.org\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/andrejacobs.org\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/andrejacobs.org\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/andrejacobs.org\/wp-json\/wp\/v2\/comments?post=540"}],"version-history":[{"count":2,"href":"https:\/\/andrejacobs.org\/wp-json\/wp\/v2\/posts\/540\/revisions"}],"predecessor-version":[{"id":542,"href":"https:\/\/andrejacobs.org\/wp-json\/wp\/v2\/posts\/540\/revisions\/542"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/andrejacobs.org\/wp-json\/wp\/v2\/media\/538"}],"wp:attachment":[{"href":"https:\/\/andrejacobs.org\/wp-json\/wp\/v2\/media?parent=540"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/andrejacobs.org\/wp-json\/wp\/v2\/categories?post=540"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/andrejacobs.org\/wp-json\/wp\/v2\/tags?post=540"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}