How to Push Apps to Cloud Foundry With a Sidecar to Stream Metrics to Prometheus

Introduction

Retrieving application metrics is usually not difficult. Simply implement a metrics endpoint and let Prometheus, for example, query that endpoint. But why is it the problem when you deploy an application on Cloud Foundry?

If you deploy your application to a platform like Cloud Foundry, you will eventually run into problems. Most likely the endpoint cannot be easily retrieved from a running Prometheus instance to be scraped.

Reasons could be e.g. the endpoint is not reachable from the network where the Prometheus instance is running also there are most probably multiple instances of your application running and each time Prometheus retrieves the application, you are redirected to a different instance via the Gorouter as he acts like a load balancer.

Although Prometheus does not work this way, there is a way around it. Instead of having Prometheus scrape the metrics endpoint, the metrics can be sent for example to a Graphite Exporter endpoint, which is then scraped by Prometheus.

Assuming that a Prometheus service instance is deployed with a Graphite Exporter you only need a sidecar deployed in your application to stream the metrics to the Graphite Exporter.

In this example, a simple application that provides a metrics endpoint is deployed to Cloud Foundry. The sidecar is packaged in a buildpack and deployed in parallel with the application via the application manifest. The actual application is not changed in the process. Thus, the effort is reduced to customizing the manifest. Only a few lines are added.

Let’s take a look at what we have.

The App

The application is written in Go and provides basic metrics in addition to a static page via a separate metrics endpoint using client libraries provided by Prometheus.

The Sidecar

For the sidecar, this example uses the telegraf-buildpack. It includes Telegraf which is a plugin-driven server agent for collecting and sending metrics. It is coupled to your application, scraps the metrics endpoint, and forwards them to a configured Graphite Exporter endpoint. In addition to the application metrics, the telegraf-buildpack is configured to let Telegraf also forward the system metrics from your application container. Collecting system metrics is a separate topic that is worth discussing. I recommend checking out this video.

Prerequisites

To deploy the application we need a platform like Cloud Foundry and a Prometheus instance. The Cloud Foundry installation we want to use is a9s Public PaaS. It offers us the necessary flexibility and we can deploy a Prometheus instance here.

The Prometheus service is the a9s Prometheus which contains Prometheus, the Prometheus Alertmanager, a Graphite Exporter and also a Grafana dashboard connected to Prometheus.

a9s Prometheus is not enabled in all accounts by default, please ask our support to get a preview.

Deploy a9s Prometheus

a9s Prometheus does not provide the ability to display application metrics by default. This is because the Graphite Exporter has a mapping configuration that transfers Graphite-compliant metrics into Prometheus metrics that are enriched with tags. By default, the Graphite Exporter is set to strictly adhere to these mappings and reject others. However, you can change this configuration by setting the mapping_strict_match property to false.

$ cf create-service a9s-prometheus prometheus-single-medium prometheus -c ‘{ “mapping_strict_match”: false }’

Before we can push the application, we have to wait until the service is deployed. You can use this command to query the status.

$ cf service prometheus

Showing info of service prometheus in org demonstrations / space demonstrations as phartz@anynines.com…

1name:             prometheus
2
3service:          a9s-prometheus
4
5tags:
6
7plan:             prometheus-single-medium
8
9description:      This is a service creating and managing dedicated Prometheus to monitor applications and service instances, powered by the anynines Service Framework
10
11documentation:    https://www.anynines.com
12
13dashboard:        https://a9s-dashboard.de.a9s.eu/service-instances/8f866593-b6bc-4dac-9caf-902677d83b04
14
15service broker:   a9s-prometheus

This service is not currently shared.

Showing status of last operation from service prometheus…

1status:    create succeeded
2
3message:
4
5started:   2022-01-31T11:10:38Z
6
7updated:   2022-01-31T11:12:50Z
8

Upgrades are not supported by this broker.

The status should be create succeeded like in the example above. Check out the Sample Application

While waiting we can check out the application with git clone and have a look at the application manifest.

$ git clone https://github.com/anynines/a9s-metrics-sample.git

Change into the directory.

$ cd a9s-metrics-sample The Application Manifest File

The repository already contains the manifest file. There are three important parts.

1. In the buildpacks section, the telegraf-buildpack is already at the beginning of the list. Do not change the order, as this may cause a deployment error.

https://gitlabci.exxeta.com/paas_buildpacks/telegraf-buildpack.git

2. The buildpack is designed to automatically use a bound a9s Prometheus instance. Therefore, you must define the service to bind in the services section. Cloud Foundry and the a9s Service Broker will then automatically take care of the necessary parameters to connect to the Prometheus instance.

3. If you use another Prometheus offering or the instance name in VCAP_SERVICES or your application differs from a9s-prometheus you have to set the env variables GRAPHITE_HOST and GRAPHITE_PORT.

1env:
2
3  GRAPHITE_HOST: phd31bf28-prometheus.node.dc1.consul.dsf2
4
5  GRAPHITE_PORT: 9109
6

Push the App

When the service instance is ready, simply push the application with the cf push command.

$ cf push

Example Dashboard

When the application is pushed, we can open the Grafana dashboard in our service instance and use the example dashboard provided in the repository to see the metrics. Just copy the JSON data into a new dashboard.

That’s it.