Skip to content
Horizontal Pod AutoScaling for Browserless
Share
Explore
Horizontal Pod AutoScaling for Browserless

icon picker
Testing and Troubleshooting

This page is about testing different parts of the configuration

Testing if metrics are flowing from into Prometheus.

Service Monitor

See if the additional service monitor is created or not:
kubectl --context head --namespace kube-system get servicemonitors
You should ideally get something like this and browserless is the service monitor we have created. If not then check the kube-prometheus-stack
ishan@Ishans-MBP-2 infra % kubectl --context head --namespace kube-system get servicemonitorsNAME AGEbrowserless 37dsumologic-kube-prometheus-apiserver 37dsumologic-kube-prometheus-coredns 37dsumologic-kube-prometheus-kube-controller-manager 37dsumologic-kube-prometheus-kube-etcd 37dsumologic-kube-prometheus-kube-proxy 37dsumologic-kube-prometheus-kube-scheduler 37dsumologic-kube-prometheus-kube-state-metrics 37dsumologic-kube-prometheus-kubelet 37dsumologic-kube-prometheus-node-exporter 50dsumologic-kube-prometheus-operator 37dsumologic-kube-prometheus-prometheus 37d

Ports and Endpoints

You need to specify correct ports at three locations.
When defining kube-prometheus-stack
In the browserless_template file
In the Service Monitor

Internal API

We port forward the prometheus ports
kubectl port-forward --context head --namespace kube-system pod/prometheus-sumologic-kube-prometheus-prometheus-0 9090:9090
and going to localhost:9090 and see if we see the browserless internal api is live or not. Looking at the error message can be helpful. Normally the error is in the Prometheus format, correct endpoints and ports.
image.png

X86 Images

To run it on cloud pods I have to upload a docker image for Metrics-Exporter to ECR. The ECR already has an image of Browserless chrome to run Browserless on pods
Its annoying that if you make a docker image on M1 mac, your image will be arm64 image. But the cloud is running x86 so there will be an issue. I had to ask Ryan, my internhost, upload images from his intel mac that runs x86. Alternatively, I made changes to the circle CI code that makes and uploads my ECR image directly. Circle CI runs whenever you push to github and runs some tests. So now it makes an image of the exporter and uploads it to EcR as well. Instructions to push to ECR is on AWS/ECR console where they give you push instructions.

Format

Check if the metrics are in correct prometheus format. Format is “browserless_{metric name} {value}”

Testing if metric are flowing from Prometheus to Kubernetes

Custom metrics API

kubectl --context head --namespace kube-system get apiservice
This should list the custom.metrics.api as available, like this.
v1beta1.batch Local True 3y250dv1beta1.certificates.k8s.io Local True 3y250dv1beta1.coordination.k8s.io Local True 3y115dv1beta1.custom.metrics.k8s.io kube-system/prometheus-adapter True 41hv1beta1.discovery.k8s.io Local True 2y30dv1beta1.elbv2.k8s.aws Local True 36dv1beta1.events.k8s.io Local True 3y250d
This would he deployed by helm however it takes 2-3 minutes for the API to get started. Also, if you helm install/upgrade charts a lot, things might get a bit jumbled so it would beneficial to helm uninstall and deploy. Interns do not have access to uninstall charts so one can ask appropriate person with permissions.

Metric URLs

There are 2 URLs that you can check and see if you are getting metrics correctly.
kubectl --context head --namespace kube-system get --raw "/apis/custom.metrics.k8s.io/v1beta1/" | jq .
This should ideally give you all the metrics that being scraped by prometheus.
Screen Shot 2022-08-18 at 10.21.51 AM.png
kubectl --context head --namespace kube-system get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/browserless/pods/*/browserless_queued" | jq .
This is give specific metrics and their values like this
ishan@Ishans-MBP-2 infra % kubectl --context head --namespace kube-system get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/browserless/pods/*/browserless_queued" | jq .{ "kind": "MetricValueList", "apiVersion": "custom.metrics.k8s.io/v1beta1", "metadata": {}, "items": [ { "describedObject": { "kind": "Pod", "namespace": "browserless", "name": "browserless-fd84df59-mf5pg", "apiVersion": "/v1" }, "metricName": "browserless_queued", "timestamp": "2022-08-18T17:23:05Z", "value": "0", "selector": null }, { "describedObject": { "kind": "Pod", "namespace": "browserless", "name": "browserless-fd84df59-pqsbg", "apiVersion": "/v1" }, "metricName": "browserless_queued", "timestamp": "2022-08-18T17:23:05Z", "value": "0", "selector": null } ]}

Troubleshooting with logs

Prometheus Adapter and HPA info

kubectl --context head --namespace browserless get pods
You can see if the prometheus adapter pods are being created and not crashing. Checking individual pod logs can be too excessive and won’t help a lot.
kubectl --context head --namespace browserless describe hpa/browserless
You can check the what the HPA is doing — ideally should look like this
ishan@Ishans-MBP-2 infra % kubectl --context head --namespace browserless describe hpa/browserlessName: browserlessNamespace: browserlessLabels: name=browserlessAnnotations: <none>CreationTimestamp: Wed, 05 Feb 2020 09:57:39 -0800Reference: Deployment/browserlessMetrics: ( current / target ) "browserless_queued" on pods: 0 / 1 resource cpu on pods (as a percentage of request): 0% (3m) / 35%Min replicas: 2Max replicas: 14Behavior: Scale Up: Stabilization Window: 0 seconds Select Policy: Max Policies: - Type: Percent Value: 900 Period: 5 seconds Scale Down: Select Policy: Max Policies: - Type: Pods Value: 1 Period: 60 secondsDeployment pods: 2 current / 2 desiredConditions: Type Status Reason Message ---- ------ ------ ------- AbleToScale True ScaleDownStabilized recent recommendations were higher than current one, applying the highest recent recommendation ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request) ScalingLimited True TooFewReplicas the desired replica count is less than the minimum replica countEvents: <none>ishan@Ishans-MBP-2 infra %
If not reading the message is useful. It will be mostly because the metrics are’t flowing in correctly to Kubernetes.



Want to print your doc?
This is not the way.
Try clicking the ⋯ next to your doc name or using a keyboard shortcut (
CtrlP
) instead.