Static Dask cluster in GCP with Helm

Get everything installed
First you need to get setup with a GCP account. There is a .
Download and install gcloud, the CLI for interacting with GCP. You will initialize gcloud with credentials, which may be your personal account directly or an IAM user.
(You can use gcloud config configurations list if you need to see what config is active.)
Download and install kubectl. This is the CLI for interacting with a running kubernetes API. In our case, gcloud will set up the kubernetes API for us, but when we want to tell kubernetes to do something directly we will need this tool.
Download and install helm. . Helm is a CLI for finding and deploying “helm charts”, which are packages of configuration for different kubernetes resources. In order to use a specific helm chart that has already been made by a third party, you need to add their helm repo. In our case we want to use the helm chart made by the Dask folks, so that means you need to run helm repo add https://helm.dask.org and then helm repo update to get local access to it.
.Deploy your Dask cluster
Start a GKE cluster. You will want to match your to the workload requirements you have.
gcloud container clusters create demo-cluster --region us-east1 --machine-type n1-standard-2
Install the helm chart
helm install dask/dask --autogenerate-name
Upgrade the helm chart with your config.yml (see my example below), which most importantly sets your worker pods to use an image that has ー or otherwise installs ー prefect in it. You can see what else can be edited in on the .
helm upgrade {name} dask/dask -f config.yml
Change the address in your flow configuration to the load balancer IP for the service in front of your scheduler.
Want to print your doc?
This is not the way.
Try clicking the ⋯ next to your doc name or using a keyboard shortcut (
CtrlP
) instead.