Skip to content
Horizontal Pod AutoScaling for Browserless
Share
Explore
Horizontal Pod AutoScaling for Browserless

icon picker
Configuration HPA from scratch

This page talks about the steps I took to configure HPA on Browserless pods using Queue length
Screen Shot 2022-08-18 at 1.33.32 PM.png

Summary

We collect metrics from Browserless pods by deploying a container that runs Browserless and a Browserless-metrics-exporter for every pod. The Browserless-metrics-exporter scrapes metrics from the Browserless deployment and changes the metrics into Prometheus format. We enable Prometheus scraper using Sumologic Helm chart and tell it to scrape an endpoint exposed by by the Browserless-metrics-exporter that returns metrics in Prometheus format. Now Prometheus is able to scrape metrics from Browserless at regular intervals. Next, we deploy a prometheus-adapter using a prometheus adapter Helm chart which converts the scraped metrics to be accessible by Kubernetes. Lastly, the Horizontal Pod Autoscaler is able to query for a desired metrics and control the deployment of Browserless Pods . And by setting some scaling policies with the queried metric we have configured Horizontal Pod Autoscaling for Browserless Pods .

Steps

We collect metrics using Prometheus. Prometheus is a tool used for metrics collections. So first we need to enable Prometheus for the current Kubernetes pods we are running. To do this we enable the kube-prometheus-stack: in values.yaml file in the infra repo and enable a service monitor. Service monitor is used to define an application you wish to scrape metrics from within Kubernetes pods.
kube-prometheus-stack:prometheus:additionalServiceMonitors:- name: browserlessendpoints:- port: metricspath: /metricsnamespaceSelector:any: trueselector:matchLabels:name: browserless

Now we can move on to browserless pods and their metrics.

Browserless gives us the metrics we need.
To get these metrics, first let’s get all the browserless pods.
ishan@Ishans-MBP-2 infra % kubectl --context head --namespace browserless get podsNAME READY STATUS RESTARTS AGEbrowserless-7dcb8dc7f-mfx7g 1/1 Running 0 9m32sbrowserless-7dcb8dc7f-qhncr 1/1 Running 0 24mbrowserless-pri-689f9f7687-4bjrf 1/1 Running 0 8m32sbrowserless-pri-689f9f7687-q46jx 1/1 Running 0 24msafe-proxy-5f488cf9b6-l6fdt 1/1 Running 0 77dsafe-proxy-5f488cf9b6-tzwgw 1/1 Running 0 21hishan@Ishans-MBP-2 infra % kubectl --context head --namespace browserless port-forward browserless-pri-689f9f7687-4bjrf 3000:3000
Then, we can port forward any one of the browserless pods
ishan@Ishans-MBP-2 infra % kubectl --context head --namespace browserless port-forward browserless-pri-689f9f7687-4bjrf 3000:3000
and when we go to
http://127.0.0.1:3000/pressure
there are other ends as well - /metrics , /prometheus and lots more in the Github link mentioned above.

Screen Shot 2022-08-10 at 4.42.55 PM.png
Screen Shot 2022-06-30 at 12.40.55 PM.png
We get this JSON response. We can use these metrics to scale the broweserless pods. However, rather than doing this same for all browserless pods, and and varying time intervals, and congregating that information we can use Prometheus.
Prometheus is a metrics collection service that scraps information from any endpoint. So we can tell prometheus to scrape infromation from localhost:3000/pressure it will collect these metrics periodically and will let you run queries and different things on it. The only thing is that prometheus can scrape information only a particular format and not a Json blob.
So if I tell it go scrape localhost:3000/pressure it will error.

Adding an additional endpoint.

So we nee an additional endpoint. This additional endpoint will request localhost:3000/pressure and turn the JSON into prometheus format and will return it. So we have 2 endpoints - browerlessPort, metricsExporterPort. Prometheus goes to metricsExporterPort to scrape. This port will go to browerlessPort to collect metrics from Browserless, switch the format and return to Prometheus through metricsExporterPort. With these metrics we can use an off the shelf autoscaler to scale the browserlesspods.
We can also look at the metrics when we port forward
kubectl port-forward --context head --namespace kube-system pod/prometheus-sumologic-kube-prometheus-prometheus-0 9090:9090

9090 is a promethues port for their metrics web dashboard

We need do a couple things for this to be done.

So first we need to enable Prometheus for the current Kubernetes pods we are running. To do this we enable the kube-prometheus-stack: in values.yaml file in the infra repo and enable a service monitor.
Enable Prometheus Stack in but disable exporter Nodes because they are useless.
Set up a Service Monitor for prometheus where we define what port to scrape.
Service monitor is used to define an application you wish to scrape metrics from within Kubernetes pods.
kube-prometheus-stack:prometheus:additionalServiceMonitors:- name: browserlessendpoints:- port: metricspath: /metricsnamespaceSelector:any: trueselector:matchLabels:name: browserless
The port is a name. Metrics is set to 3000 and /metrics the path where I get promethues metrics. (In my implementation) So I am essentially telling prometheus to scrape localhost:3000/metrics
Like this we have connect our Kubernetes pods with Promethues and have all its features. However, above we saying that only enable the service monitor for browserless pods(using matchLabels). Moreover, we probably don’t want a lot of Promethues features like having exporter pods for everything. We disable this for ease.
Set up a an additional Websever. I make an express-app in the headless-chrome repo.
const prometheusFormat = Object.entries(responsePressureJson['pressure']).filter(([key, value]) => Number.isInteger(value)).map(([key, value]) => `browserless_${key} ${value}`).join('\n');

This is the JSON we get from the '/pressure' endpoint -{"pressure": {"date": 1660153166465,"reason": "","message": "","isAvailable": true,"queued": 0,"recentlyRejected": 0,"running": 2,"maxConcurrent": 2,"maxQueued": 10,"cpu": 1,"memory": 4}}

From this we discard all non integer value (eg isAvailable, message).This is changed into prometheus format -

"browserless_date 1660153166465browserless_queued 0browserless_recently_rejected 0browserless_running 2browserless_max_concurrent 2browserless_max_queued 10browserless_cpu 1browserless_memory 4"*/
I was initially deploying this locally on my machine. To have this express app run on Browserless pods one needs to make a Docker and Docker Compose file to run Browserless and my metrics exporter in one container. However, this is to only run the code locally, (which is not that useful but good for testing).
To run it on cloud pods I have to upload a docker image for Metrics-Exporter to ECR. The ECR already has an image of Browserless chrome to run Browserless on pods
Its annoying that if you make a docker image on M1 mac, your image will be arm64 image. But the cloud is running x86 so there will be an issue. I had to ask Ryan, my internhost, upload images from his intel mac that runs x86. Alternatively, I made changes to the circle CI code that makes and uploads my ECR image directly. Circle CI runs whenever you push to github and runs some tests. So now it makes an image of the exporter and uploads it to EcR as well. Instructions to push to ECR is on AWS/ECR console where they give you push instructions.
I had to make changes to browserless_template to add/identify a new image.
- env:- name: BROWSERLESS_HOSTvalue: "localhost"- name: BROWSERLESS_PORTvalue: "3000"- name: EXPORTER_PORTvalue: "8000"image: 560248293975.dkr.ecr.us-west-2.amazonaws.com/browserless-metrics-exporter:${BROWSERLESS_METRICS_IMAGE_TAG}imagePullPolicy: IfNotPresentlivenessProbe:failureThreshold: 3initialDelaySeconds: 5periodSeconds: 10successThreshold: 1tcpSocket:port: 8000timeoutSeconds: 20name: browserless-metrics-exporterports:- containerPort: 8000name: metricsprotocol: TCPresources:limits:cpu: "1"memory: 1Grequests:cpu: "1"memory: 512M
I also need to make a change to service monitors
apiVersion: v1kind: Servicemetadata:name: ${POOL_NAME}labels:name: ${POOL_NAME}spec:ports:- name: puppeteerprotocol: TCPport: 80targetPort: 3000- name: metricsprotocol: TCPport: 8000
Want to print your doc?
This is not the way.
Try clicking the ⋯ next to your doc name or using a keyboard shortcut (
CtrlP
) instead.