Not able to install GitPod 0-6-0 in a self hosted cluster

I have a 2 Nodes cluster, while trying to install gitpod-0.6.0, I get:

[root@Kubernetes-Master chart]# helm install gitpod gitpod.io/gitpod --debug
install.go:173: [debug] Original chart version: ""
install.go:190: [debug] CHART PATH: /root/.cache/helm/repository/gitpod-0.6.0.tgz

client.go:122: [debug] creating 128 resource(s)
client.go:282: [debug] Starting delete for "db-migrations" Job
client.go:122: [debug] creating 1 resource(s)
client.go:491: [debug] Watching for changes to Job db-migrations with timeout of 5m0s
client.go:519: [debug] Add/Modify event for db-migrations: ADDED
client.go:558: [debug] db-migrations: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:519: [debug] Add/Modify event for db-migrations: MODIFIED
client.go:558: [debug] db-migrations: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:282: [debug] Starting delete for "db-migrations" Job
Error: failed post-install: timed out waiting for the condition
helm.go:81: [debug] failed post-install: timed out waiting for the condition

I am having the same issue. When I run kubectl get pods --all-namespaces I see there are errors pulling the images:

default       registry-facade-tw77s                     0/1     ContainerCreating       0          3m19s
default       ws-daemon-mtdxp                           0/1     Init:0/2                0          3m19s
default       proxy-845d4497bb-57f86                    0/1     ContainerCreating       0          3m19s
default       ws-proxy-549f76f6bb-ssjlr                 0/1     ContainerCreating       0          3m18s
default       svclb-proxy-hhp7h                         2/2     Running                 0          3m19s
default       minio-84c5c6f694-wg25k                    1/1     Running                 0          3m18s
default       registry-6d64cf7dc9-wn9m2                 1/1     Running                 0          3m19s
default       mysql-7cbb9c9586-8nmdj                    1/1     Running                 0          3m19s
default       messagebus-7bbdb4ff6-s2ghf                0/1     ImagePullBackOff        0          3m19s
default       db-migrations-sfg28                       0/1     ImagePullBackOff        0          3m19s
default       blobserve-7d74c64667-hd2lv                0/1     ImagePullBackOff        0          3m18s
default       image-builder-6b55575cb4-8tm7d            1/2     ImagePullBackOff        0          3m19s
default       server-7848b59796-pwrg9                   0/1     Init:ErrImagePull       0          3m18s
default       content-service-648df67c69-cjw7g          0/1     ErrImagePull            0          3m19s
default       ws-manager-84bbd88967-xrks4               0/1     ErrImagePull            0          3m19s
default       ws-manager-bridge-569788bb89-z6rxl        0/1     Init:ImagePullBackOff   0          3m18s
default       dashboard-66cbdf4d7b-vb5rm                0/1     ErrImagePull            0          3m19s
default       ws-scheduler-6cb896f65d-4spzp             0/1     ErrImagePull            0          3m19s

I’m guessing that the gcr.io registry requires authentication for these images? I’ll keep looking into it and let you know if I find a solution.

It looks like my image-builder POD is not able to start due to:

[root@Kubernetes-Master chart]# kubectl logs image-builder-75689d6557-pbk6k service
{"interval":"6h0m0s","level":"info","message":"starting Docker ref pre-cache","refs":["gitpod/workspace-full:latest"],"serviceContext":{"service":"image-builder","version":""},"severity":"INFO","time":"2021-03-28T08:43:20Z"}
{"gitpodLayer":"/app/workspace-image-layer.tar.gz","hash":"ff3702e73ade2dfeb84d235f727e885b86bd1a9a40202e91e928dd65e77c7de3","level":"info","message":"computed Gitpod layer hash","serviceContext":{"service":"image-builder","version":""},"severity":"INFO","time":"2021-03-28T08:43:20Z"}
{"gitpodLayer":"/app/workspace-image-layer.tar.gz","level":"info","message":"running self-build","serviceContext":{"service":"image-builder","version":""},"severity":"INFO","time":"2021-03-28T08:43:20Z"}
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 10.96.0.10:53: read udp 10.32.0.2:41975-\u003e10.96.0.10:53: read: connection refused","level":"warning","message":"unable to precache reference","ref":"gitpod/workspace-full:latest","serviceContext":{"service":"image-builder","version":""},"severity":"WARNING","time":"2021-03-28T08:43:30Z"}
{"level":"debug","message":"self-build context sent","serviceContext":{"service":"image-builder","version":""},"severity":"DEBUG","time":"2021-03-28T08:43:30Z"}
{"level":"debug","message":"Step 1/7 : FROM alpine:3.9","severity":"DEBUG","time":"2021-03-28T08:43:30Z"}
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 10.96.0.10:53: read udp 10.32.0.2:55839-\u003e10.96.0.10:53: read: connection refused","level":"error","message":"self-build failed","serviceContext":{"service":"image-builder","version":""},"severity":"ERROR","time":"2021-03-28T08:43:40Z"}
{"level":"fatal","message":"Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 10.96.0.10:53: read udp 10.32.0.2:55839-\u003e10.96.0.10:53: read: connection refused","serviceContext":{"service":"image-builder","version":""},"severity":"CRITICAL","time":"2021-03-28T08:43:40Z"}

BTW, I’m following the installation instructions from: https://www.gitpod.io/docs/self-hosted/latest/install/install-on-kubernetes/

and when running kubectl logs image-builder-75689d6557-pbk6k dind I get:

time="2021-03-28T08:39:46.985677922Z" level=error msg="Handler for GET /v1.40/distribution/docker.io/gitpod/workspace-full:latest/json returned error: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 10.96.0.10:53: read udp 10.32.0.2:54252->10.96.0.10:53: read: connection refused"
time="2021-03-28T08:40:02.123302953Z" level=warning msg="Error getting v2 registry: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
time="2021-03-28T08:40:02.123336256Z" level=info msg="Attempting next endpoint for pull after error: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
time="2021-03-28T08:40:23.680391556Z" level=error msg="Handler for GET /v1.40/distribution/docker.io/gitpod/workspace-full:latest/json returned error: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 10.96.0.10:53: read udp 10.32.0.2:38219->10.96.0.10:53: read: connection refused"
time="2021-03-28T08:40:33.821106563Z" level=warning msg="Error getting v2 registry: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 10.96.0.10:53: read udp 10.32.0.2:36588->10.96.0.10:53: read: connection refused"
time="2021-03-28T08:40:33.821133005Z" level=info msg="Attempting next endpoint for pull after error: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 10.96.0.10:53: read udp 10.32.0.2:36588->10.96.0.10:53: read: connection refused"
time="2021-03-28T08:41:09.665935104Z" level=error msg="Handler for GET /v1.40/distribution/docker.io/gitpod/workspace-full:latest/json returned error: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 10.96.0.10:53: read udp 10.32.0.2:36422->10.96.0.10:53: read: connection refused"
time="2021-03-28T08:41:24.810236297Z" level=warning msg="Error getting v2 registry: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
time="2021-03-28T08:41:24.810260503Z" level=info msg="Attempting next endpoint for pull after error: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
time="2021-03-28T08:42:08.708295281Z" level=error msg="Handler for GET /v1.40/distribution/docker.io/gitpod/workspace-full:latest/json returned error: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 10.96.0.10:53: read udp 10.32.0.2:40779->10.96.0.10:53: i/o timeout"
time="2021-03-28T08:42:18.860332198Z" level=warning msg="Error getting v2 registry: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 10.96.0.10:53: read udp 10.32.0.2:46635->10.96.0.10:53: read: connection refused"
time="2021-03-28T08:42:18.860358809Z" level=info msg="Attempting next endpoint for pull after error: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 10.96.0.10:53: read udp 10.32.0.2:46635->10.96.0.10:53: read: connection refused"
time="2021-03-28T08:43:30.756327235Z" level=error msg="Handler for GET /v1.40/distribution/docker.io/gitpod/workspace-full:latest/json returned error: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 10.96.0.10:53: read udp 10.32.0.2:41975->10.96.0.10:53: read: connection refused"
time="2021-03-28T08:43:40.911100396Z" level=warning msg="Error getting v2 registry: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 10.96.0.10:53: read udp 10.32.0.2:55839->10.96.0.10:53: read: connection refused"
time="2021-03-28T08:43:40.911123461Z" level=info msg="Attempting next endpoint for pull after error: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 10.96.0.10:53: read udp 10.32.0.2:55839->10.96.0.10:53: read: connection refused"
time="2021-03-28T08:45:21.706780596Z" level=error msg="Handler for GET /v1.40/distribution/docker.io/gitpod/workspace-full:latest/json returned error: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 10.96.0.10:53: read udp 10.32.0.2:46360->10.96.0.10:53: read: connection refused"
time="2021-03-28T08:45:36.857207030Z" level=warning msg="Error getting v2 registry: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
time="2021-03-28T08:45:36.857243461Z" level=info msg="Attempting next endpoint for pull after error: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"

My issues turned out to be related to k3s and docker settings. I ended up getting it deployed under minikube instead.

I did notice that I was getting rate limited on pulls. Are you still having issues? If so, what k8s distribution are you using?

I was able to get it running under k3s as well, and I think I’ll stick with k3s for the time being.

Will, do you mind sharing how did you get GitPod 0-6-0 running in K8s?

I created my cluster by following: https://www.tecmint.com/install-a-kubernetes-cluster-on-centos-8/ and tried to get GitPod running by using: https://www.gitpod.io/docs/self-hosted/latest/install/install-on-kubernetes/

Here is a repository that I forked and updated/customized:

If you use a different DNS provider, then you’ll want to customize “create-certs.sh”, since it is currently set up for Cloudflare.

As far as prerequisites, you should just need Docker installed, and I did find that I had to set the host’s Docker daemon to use VFS instead of the overlay driver:

If I run into any problems with using VFS, then I’ll probably switch to k3s directly on the host next.