I’ve been playing with Prometheus monitoring lately. It is fairly new software that is getting popular. Prometheus works using a pull architecture. A central server connects to each thing you want to monitor every few seconds and grabs stats from it.
In the simplest case you run the node_exporter on each machine which gathers about 600-800 (!) metrics such as load, disk space and interface stats. This exporter listens on port 9100 and effectively works as an http server that responds to “GET /metrics HTTP/1.1” and spits several hundred lines of:
node_forks 7916 node_intr 3.8090539e+07 node_load1 0.47 node_load15 0.21 node_load5 0.31 node_memory_Active 6.23935488e+08
Other exporters listen on different ports and export stats for apache or mysql while more complicated ones will act as proxies for outgoing tests (via snmp, icmp, http). The full list of them is on the Prometheus website.
So my problem was that I wanted to check my virtual machine that is on Linode. The machine only has a public IP and I didn’t want to:
- Allow random people to check my servers stats
- Have to setup some sort of VPN.
So I decided that the best way was to just use put a user/password on the exporter.
However the node_exporter does not implement authentication itself since the authors wanted the avoid maintaining lots of security code. So I decided to put it behind a reverse proxy using apache mod_proxy.
Step 1 – Install node_exporter
Node_exporter is a single binary that I started via an upstart script. As part of the upstart script I told it to listen on localhost port 19100 instead of port 9100 on all interfaces
# cat /etc/init/prometheus_node_exporter.conf description "Prometheus Node Exporter" start on startup chdir /home/prometheus/ script /home/prometheus/node_exporter -web.listen-address 127.0.0.1:19100 end script
Once I start the exporter a simple “curl 127.0.0.1:19100/metrics” makes sure it is working and returning data.
Step 2 – Add Apache proxy entry
First make sure apache is listening on port 9100 . On Ubuntu edit the /etc/apache2/ports.conf file and add the line:
Listen 9100
Next create a simple apache proxy without authentication (don’t forget to enable mod_proxy too):
# more /etc/apache2/sites-available/prometheus.conf <VirtualHost *:9100> ServerName prometheus CustomLog /var/log/apache2/prometheus_access.log combined ErrorLog /var/log/apache2/prometheus_error.log ProxyRequests Off <Proxy *> Allow from all </Proxy> ProxyErrorOverride On ProxyPass / http://127.0.0.1:19100/ ProxyPassReverse / http://127.0.0.1:19100/ </VirtualHost>
This simply takes requests on port 9100 and forwards them to localhost port 19100 . Now reload apache and test via curl to port 9100. You can also use netstat to see what is listening on which ports:
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.1:19100 0.0.0.0:* LISTEN 8416/node_exporter tcp6 0 0 :::9100 :::* LISTEN 8725/apache2
Step 3 – Get Prometheus working
I’ll assume at this point you have other servers working. What you need to do now is add the following entries for you server in you prometheus.yml file.
First add basic_auth into your scape config for “node” and then add your servers, eg:
- job_name: 'node'
scrape_interval: 15s
basic_auth:
username: prom
password: mypassword
static_configs:
- targets: ['myserver.example.com:9100']
labels:
group: 'servers'
alias: 'myserver'
Now restart Prometheus and make sure it is working. You should see the following lines in your apache logs plus stats for the server should start appearing:
10.212.62.207 - - [31/Jul/2016:11:31:38 +0000] "GET /metrics HTTP/1.1" 200 11377 "-" "Go-http-client/1.1" 10.212.62.207 - - [31/Jul/2016:11:31:53 +0000] "GET /metrics HTTP/1.1" 200 11398 "-" "Go-http-client/1.1" 10.212.62.207 - - [31/Jul/2016:11:32:08 +0000] "GET /metrics HTTP/1.1" 200 11377 "-" "Go-http-client/1.1"
Notice that connections are 15 seconds apart, get http code 200 and are 11k in size. The Prometheus server is using Authentication but apache doesn’t need it yet.
Step 4 – Enable Authentication.
Now create an apache password file:
htpasswd -cb /home/prometheus/passwd prom mypassword
and update your apache entry to the followign to enable authentication:
# more /etc/apache2/sites-available/prometheus.conf <VirtualHost *:9100> ServerName prometheus CustomLog /var/log/apache2/prometheus_access.log combined ErrorLog /var/log/apache2/prometheus_error.log ProxyRequests Off <Proxy *> Order deny,allow Allow from all # AuthType Basic AuthName "Password Required" AuthBasicProvider file AuthUserFile "/home/prometheus/passwd" Require valid-user </Proxy> ProxyErrorOverride On ProxyPass / http://127.0.0.1:19100/ ProxyPassReverse / http://127.0.0.1:19100/ </VirtualHost>
After you reload apache you should see the following:
10.212.56.135 - prom [01/Aug/2016:04:42:08 +0000] "GET /metrics HTTP/1.1" 200 11394 "-" "Go-http-client/1.1" 10.212.56.135 - prom [01/Aug/2016:04:42:23 +0000] "GET /metrics HTTP/1.1" 200 11392 "-" "Go-http-client/1.1" 10.212.56.135 - prom [01/Aug/2016:04:42:38 +0000] "GET /metrics HTTP/1.1" 200 11391 "-" "Go-http-client/1.1"
Note that the “prom” in field 3 indicates that we are logging in for each connection. If you try to connect to the port without authentication you will get:
Unauthorized This server could not verify that you are authorized to access the document requested. Either you supplied the wrong credentials (e.g., bad password), or your browser doesn't understand how to supply the credentials required.
That is pretty much it. Note that will need to add additional Virtualhost entries for more ports if you run other exporters on the server.
