anynines website



Michael Both

Published at 21.12.2020

How-To’s & Tutorials

Prometheus Pushgateway on Cloud Foundry With Basic Authentication

Authentication methods are currently not supported in Prometheus, nor its related components like the Pushgateway.

However, this can be added via reverse proxies. Pushing it as an app to Cloud Foundry with neither the go-buildpack nor the binary-buildpack will give you such functionality. The staticfile-buildpack supports configuring basic authentication.

This article describes how to push the Pushgateway to Cloud Foundry using the staticfile-buildpack with basic authentication.

Table of Contents

Create Working Directories

First, let’s create a workspace directory for our project and change into it. Every file we create will be placed into that directory. The rest of the tutorial is using paths relative to that directory, so you may change it to your liking.

Code Example

1$ mkdir -p ~/workspace/apps/pushgateway
2$ cd ~/workspace/apps/pushgateway
Download and Extract Pushgateway Binary

We will use the binary release of the Pushgateway, so head over to the Github release page and download the respective Linux-amd64 archive of the version compatible with your Prometheus installation. For our purpose, we’ll be using the current version 1.0.0.

Code Example

1$ wget
2$ wget
3$ sha256sum -c <(grep pushgateway-1.0.0.linux-amd64.tar.gz sha256sums.txt)
4pushgateway-1.0.0.linux-amd64.tar.gz: OK

If the shasum check is OK, extract the binary and remove what is not needed anymore.

On Linux (GNU tar installed):

Code Example

1$ tar xzf pushgateway-1.0.0.linux-amd64.tar.gz --wildcards --strip-components=1 */pushgateway
2$ rm pushgateway-1.0.0.linux-amd64.tar.gz sha256sums.txt

For MacOS users (BSD tar installed):

Code Example

1$ tar xzf pushgateway-1.0.0.linux-amd64.tar.gz --strip-components=1 */pushgateway
2$ rm pushgateway-1.0.0.linux-amd64.tar.gz sha256sums.txt
Add Basic Authentication

The staticfile-buildpack is designed to serve files like HTML or CSS via an integrated NGINX server. Basic authentication is set using a special file Staticfile.auth that the buildpack will read at container start and setup NGINX accordingly.

The file is in the typical .htpasswd file format and can be created using the following command (I will use pushgateway as the username, but you may choose whatever suits you):

Code Example

1$ echo "pushgateway:$(openssl passwd -1)" > Staticfile.auth

Set the password and confirm it and you’re good to go. 

Create an Application Manifest

Everything except for the manifest.yml and files/folders listed in a .cfignore will be integrated to /home/vcap/app in the container.

NGINX is started by default via app/ on container start. But since we also want to start the pushgateway process, this needs to be changed.

Create the following manifest.yml file your workspace folder:

Code Example

2name: pushgateway
3disk_quota: 256M
4memory: 256M
5command: exec /home/vcap/app/
6buildpack: staticfile_buildpack

This will change the command executed at container start to /home/vcap/app/, which has yet to be created by us.

Create the Start Script

All you need to do is to start the pushgateway in the background and continue with the default startup.

Code Example

3/home/vcap/app/pushgateway --web.listen-address &

However, this does not redirect requests to yet. Instead, it serves all files in the workspace directory to the world, which is not what we want.

Add a Custom Location

To change this, the buildpack offers a way to customize the location block. We have to set root and location_include in a file named Staticfile:

Code Example

1root: htdocs
2location_include: includes/*.conf

I have chosen htdocs to serve as the root location, but you can use whatever you like. But if you leave it like that, you will get a staging error because the root directory is empty.

Code Example

1-----> Root folder /tmp/app/htdocs
2       **ERROR** Invalid root directory: the application Staticfile specifies a root directory htdocs that does not exist

All you need to do is add a dummy index.html file

Code Example

1$ mkdir -p htdocs
2$ echo "<h1>Nothing here</h1>" > htdocs/index.html

It doesn’t matter what content you add here, because it should actually never be displayed.

The location_include directive is relative to the /home/vcap/app/nginx/conf directory. NGINX will load any *.conf file there which we will use to configure the location. But as the workspace directory is added to /home/vcap/app, we need to create additional directories.

Code Example

1$ mkdir -p nginx/conf/includes

Then add the following content to nginx/conf/includes/pushgateway.conf with your favorite editor:

Code Example

2proxy_set_header	Host            $host;
3proxy_set_header	X-Real-IP       $remote_addr;
4proxy_set_header	X-Forwarded-for $remote_addr;
Adjust Scrape Configuration

Your scrape config must be adjusted to scrape with basic authentication (see Prometheus Configuration).

If you deploy Prometheus with prometheus-boshrelease, the internal pushgateway does not have basic-auth support at the moment and there are no ops files to scrape an external pushgateway. But you can add a custom ops file like the following to the deployment to make up for it.

Code Example

1- type: replace
2  path: /instance_groups/name=prometheus2/jobs/name=prometheus2/properties?/prometheus/scrape_configs/-
3  value:
4    job_name: pushgateway
5    static_configs:
6    - targets:
7      - ((pushgateway_host))
8    scheme: https
9    basic_auth:
10      username: ((pushgateway_basicauth.username))
11      password: ((pushgateway_basicauth.password))

If you use credhub to store your deployment secrets (recommended), you need to set the pushgateway_basicauth user credentials and pushgateway_host.

For example using the credhub CLI (replace the paths and values with respect to your environment):

Code Example

1$ credhub set -n /bosh/prometheus/pushgateway_host -t value -v
2$ credhub set -n /bosh/prometheus/pushgateway_basicauth -t user --username=pushgateway --password="REPLACE_ME"

In case you already have an external pushgateway setup without authentication, you can deploy this change without breaking metrics scraping. This is how we transitioned to an external pushgateway with basic authentication smoothly. That is because a pushgateway without an authentication set will just ignore the additional credentials. In the next step, you can then push the basic-auth enabled pushgateway to Cloud Foundry.

The only thing that might break is if your configuration so far does not use HTTPS, but you could also just remove the scheme line from the ops file above to use HTTP (not recommended).

Push the App

The final step is just to push the pushgateway to Cloud Foundry by simply using

Code Example

1$ cf push

after you logged in and chose the org/space where you want it to be deployed.


Finally, let’s test pushing metrics and reading them from the pushgateway.

Should fail:

Code Example

1$ metrics="# TYPE test_metric gauge
2test_metric{job=\"test\"} 0"
3$ curl -i --data-binary <(echo "$metrics")
4HTTP/1.1 401 Unauthorized
7$ curl -i
8HTTP/1.1 401 Unauthorized

Should work:

Code Example

1$ metrics="# TYPE test_metric gauge
2test_metric{job=\"test\"} 0"
3$ curl -i --user "pushgateway:REPLACE_ME" --data-binary @<(echo "$metrics")
4HTTP/1.1 202 Accepted
7$ curl -i --user "pushgateway:REPLACE_ME"
8HTTP/1.1 200 OK
Final Thoughts

In this tutorial, we have seen how to use staticfile-buildpack to deploy the Prometheus Pushgateway to Cloud Foundry with basic authentication. It is a rather long procedure and there were some traps on the way because the staticfile-buildpack probably isn’t directly meant for this kind of usage, but once done even updates are rather simple by simply exchanging the Pushgateway binary.

One thing, however, is not covered here and should be improved upon, and that is health monitoring/restarting of the pushgateway process in case it crashes. But this is for another time.

© anynines GmbH 2024


Privacy Policy


© anynines GmbH 2024