This page talks about the steps I took to configure HPA on Browserless pods using Queue length
Summary
We collect metrics from Browserless pods by deploying a container that runs Browserless and a Browserless-metrics-exporter for every pod. The Browserless-metrics-exporter scrapes metrics from the Browserless deployment and changes the metrics into Prometheus format. We enable Prometheus scraper using Sumologic Helm chart and tell it to scrape an endpoint exposed by by the Browserless-metrics-exporter that returns metrics in Prometheus format. Now Prometheus is able to scrape metrics from Browserless at regular intervals. Next, we deploy a prometheus-adapter using a prometheus adapter Helm chart which converts the scraped metrics to be accessible by Kubernetes. Lastly, the Horizontal Pod Autoscaler is able to query for a desired metrics and control the deployment of Browserless Pods . And by setting some scaling policies with the queried metric we have configured Horizontal Pod Autoscaling for Browserless Pods .
Steps
We collect metrics using Prometheus. Prometheus is a tool used for metrics collections. So first we need to enable Prometheus for the current Kubernetes pods we are running. To do this we enable the kube-prometheus-stack: in values.yaml file in the infra repo and enable a service monitor. Service monitor is used to define an application you wish to scrape metrics from within Kubernetes pods.
kube-prometheus-stack:
prometheus:
additionalServiceMonitors:
- name: browserless
endpoints:
- port: metrics
path: /metrics
namespaceSelector:
any: true
selector:
matchLabels:
name: browserless
Now we can move on to browserless pods and their metrics.
Browserless gives us the metrics we need.
To get these metrics, first let’s get all the browserless pods.
ishan@Ishans-MBP-2 infra % kubectl --context head --namespace browserless get pods
ishan@Ishans-MBP-2 infra % kubectl --context head --namespace browserless port-forward browserless-pri-689f9f7687-4bjrf 3000:3000
Then, we can port forward any one of the browserless pods
ishan@Ishans-MBP-2 infra % kubectl --context head --namespace browserless port-forward browserless-pri-689f9f7687-4bjrf 3000:3000
and when we go to
http://127.0.0.1:3000/pressure
there are other ends as well - /metrics , /prometheus and lots more in the Github link mentioned above.
We get this JSON response. We can use these metrics to scale the broweserless pods. However, rather than doing this same for all browserless pods, and and varying time intervals, and congregating that information we can use Prometheus.
Prometheus is a metrics collection service that scraps information from any endpoint. So we can tell prometheus to scrape infromation from localhost:3000/pressure it will collect these metrics periodically and will let you run queries and different things on it. The only thing is that prometheus can scrape information only a particular format and not a Json blob.
So if I tell it go scrape localhost:3000/pressure it will error.
Adding an additional endpoint.
So we nee an additional endpoint. This additional endpoint will request localhost:3000/pressure and turn the JSON into prometheus format and will return it. So we have 2 endpoints - browerlessPort, metricsExporterPort. Prometheus goes to metricsExporterPort to scrape. This port will go to browerlessPort to collect metrics from Browserless, switch the format and return to Prometheus through metricsExporterPort. With these metrics we can use an off the shelf autoscaler to scale the browserlesspods.
We can also look at the metrics when we port forward
kubectl port-forward --context head --namespace kube-system pod/prometheus-sumologic-kube-prometheus-prometheus-0 9090:9090
9090 is a promethues port for their metrics web dashboard
We need do a couple things for this to be done.
So first we need to enable Prometheus for the current Kubernetes pods we are running. To do this we enable the kube-prometheus-stack: in values.yaml file in the infra repo and enable a service monitor.
but disable exporter Nodes because they are useless.
Set up a Service Monitor for prometheus where we define what port to scrape.
Service monitor is used to define an application you wish to scrape metrics from within Kubernetes pods.
kube-prometheus-stack:
prometheus:
additionalServiceMonitors:
- name: browserless
endpoints:
- port: metrics
path: /metrics
namespaceSelector:
any: true
selector:
matchLabels:
name: browserless
The port is a name. Metrics is set to 3000 and /metrics the path where I get promethues metrics. (In my implementation) So I am essentially telling prometheus to scrape localhost:3000/metrics
Like this we have connect our Kubernetes pods with Promethues and have all its features. However, above we saying that only enable the service monitor for browserless pods(using matchLabels). Moreover, we probably don’t want a lot of Promethues features like having exporter pods for everything. We disable this for ease.
Set up a an additional Websever. I make an express-app in the headless-chrome repo.
This is the JSON we get from the '/pressure' endpoint -
{
"pressure": {
"date": 1660153166465,
"reason": "",
"message": "",
"isAvailable": true,
"queued": 0,
"recentlyRejected": 0,
"running": 2,
"maxConcurrent": 2,
"maxQueued": 10,
"cpu": 1,
"memory": 4
}
}
From this we discard all non integer value (eg isAvailable, message).
This is changed into prometheus format -
"browserless_date 1660153166465
browserless_queued 0
browserless_recently_rejected 0
browserless_running 2
browserless_max_concurrent 2
browserless_max_queued 10
browserless_cpu 1
browserless_memory 4"
*/
I was initially deploying this locally on my machine. To have this express app run on Browserless pods one needs to make a Docker and Docker Compose file to run Browserless and my metrics exporter in one container. However, this is to only run the code locally, (which is not that useful but good for testing).
To run it on cloud pods I have to upload a docker image for Metrics-Exporter to ECR. The ECR already has an image of Browserless chrome to run Browserless on pods
Its annoying that if you make a docker image on M1 mac, your image will be arm64 image. But the cloud is running x86 so there will be an issue. I had to ask Ryan, my internhost, upload images from his intel mac that runs x86. Alternatively, I made changes to the circle CI code that makes and uploads my ECR image directly. Circle CI runs whenever you push to github and runs some tests. So now it makes an image of the exporter and uploads it to EcR as well. Instructions to push to ECR is on AWS/ECR console where they give you push instructions.
I had to make changes to browserless_template to add/identify a new image.
Now if I go port forward the prometheus pods and go to the dash board I can see metrics being scraped.
Configuring HPA
To get prometheus metrics and use it to scale Kube pods we need a prometheus metrics adapter. Rather than writing my own prometheus adapter, I used a helm charts. Helm charts are basically pre written Kuberneters config files (things that can create deployments, pods, service monitors, etc) that you can customize to your need. I used the
helm chart which deployed a prometheus metrics adapter (and a bunch or extra things). And whatever changes we want to, we make it in a separate file (value.yaml in my case) that you pass when you want to install/resinstall(upgrade) the helm chart.
I added the following to my value.yaml file which basically sets the correct prometheus link and queries for the information I want. This is to run the prometheus adapter from headless-chrome.