Self-hosted Kubernetes - cannot connect to ws-sync

I’ve been able to successfully get the Gitpod installation (chart v0.5.0) fully running on a self-hosted Kubernetes cluster. All services are able start successfully (thanks to the tip on manually removing node labels on chart upgrades via this issue. However when starting a workspace, I get this error:

cannot initialize workspace: cannot connect to ws-sync: cannot connect to ws-sync: cannot connect to workspace sync; last backup failed: cannot connect to ws-sync: cannot connect to workspace sync.

The logs for ws-sync and ws-manager don’t have any other clues it seems, except for the ECONNREFUSED 0.0.0.0:5778 issue that @thomashansen reported, which I’m not sure if that’s related.

I’m running on a single-node cluster and rely on ingress to connect to the proxy server. Is there some additional config that needs to be done to allow the connections to ws-sync to happen?

Documenting my solution here in case it’s helpful to anybody else. It turns out ws-sync binds to hostport 8080 (and not a nodeport or container port, as I had assumed). In my case this caused an issue because I am already running another service on port 8080 on the host. Changing the port number solved the issue!

2 Likes

Hi @abesnabe, cool that you were able to fix it so quickly, and many thanks for sharing your solution! :100::pray:

I believe this error message is generic and applies to several situations. I too have this error with a v0.5.0 deployment, however, it is due to the fact the workspace pod fails to achieve a healthy running state.

Readiness probe failed: Get http://10.42.2.63:22999/: dial tcp 10.42.2.63:22999: connect: connection refused

It appears the theia process starts then stops and never enters a running state on the workstation pod. What is causing the pod to instantly stop is still unknown.

I can confirm same error, different problem. In this case the DNS settings were default of 1.1.1.1 and 8.8.8.8 blocked by the corporate firewall. Changing the chart to use internal DNS servers fixed the problem.

3 Likes

Thank you it fixed my issue too!