Websocket Issue with Gitpod Self-Hosted

Hi all!

I’m attempting to deploy Gitpod Self Hosted on an internal company network. I’m currently trying to use the Chart for v0.8.0, however, I’ve had quite a few issues. Following the docs (from https://www.gitpod.io/docs/self-hosted/latest/install/install-on-kubernetes) the helm install resulted in a bunch of pods that were failing to initialize. The issue was regarding my default storage class, so I ended up running helm template so that I could edit the yaml for the PVC. What I ended up running looked like:

I generated a csr for gitpod-proxy . k8s .mycompany.com (ignore the extra spaces there, the forum is telling me I’m using too many links), signed it with my company’s CA, and loaded that into https-certificates (per docs/self-hosted/latest/install/configure-ingress)

Then I ran:

helm template --name-template gitpod gitpod-0.8.0.tgz --output-dir gitpod --namespace gitpod --values values.custom.yaml

with a values.custom.yaml that looks like:

hostname: k8s.mycompany.com

storageClassName: storage-class-name

certificatesSecret:
   secretName: https-certificates

components:
   wsDaemon:
      containerRuntime:
         nodeRoots:
         - /var/lib
         - /run/containerd/io.containerd.runtime.v1.linux/moby

minio:
   accessKey: accesskey
   secretKey: secretkey

Then I ran:

kubectl -n gitpod apply -R -f gitpod

All of the pods spun up and seem to be running. We use an NGINX load balancer in front of our kubernetes cluster with a wild card cert and DNS entry so that *.k8s.mycompany.com get’s routed to our load balancer and then onto the k8s cluster. So next I configured an ingress like this:

kind: Ingress
metadata:
<snip>
   name: gitpod-ingress
   namespace: gitpod
spec:
   rules:
   - host: proxy-gitpod.k8s.mycompany.com
     http:
       paths:
       - backend:
         serviceName: proxy
         servicePort: https

Now I can go to gitpod-proxy.k8s.mycompany.com in my browser and I get a black screen. After a while, I get an error that says “We are having trouble connecting to the server. Either you are offline or websocket connections are blocked.” So I bring up the Firefox developer tools and use the Network tab to look at the HTTP requests. Everything looks good (HTTP 200) except for the components hitting wss://proxy-gitpod.k8s.mycompany.com which returns an HTTP 403 Forbidden.

Do I need to configure an additional ingress or something to get the websocket traffic routed to the correct pod?

Thanks!
-b

Gitpod should have created a proxy service as loadbalancer , You need to add 3 A records like gitpod.k8s.mycompany.com, *.gitpod.k8s.mycompany.com *.ws.gitpod.k8s.mycompany.com with Same load balancer ip. This should fix any routing issue.

Incase you need to use nginx as ingress controller then in Ingress Yaml you need to use ingress class annotation and point it to nginx. Something like below.

Hope this helps.

metadata:
name: <<Name_Of_Ingress>>
namespace: <>
annotations:
kubernetes.io/ingress.class: nginx

Hi @chaitu454,

As I mentioned, we use an NGINX Webserver that is external to our Kubernetes Cluster as a Load Balancer. We have DNS configured such that *.k8s.mycompany.com routes to this NGINX Webserver which has a wildcard certificate to enable TLS termination. So I don’t think I need to add any A records as they will all resolve to my NGINX load balancer which forwards the traffic to all worker nodes in the kubernetes clusters.

I added the annotation that you recommended, but I still get 403 (Forbidden) errors for all HTTP requests to wss://proxy-gitpod.k8s.mycompany.com/api/gitpod

Also, if I used the proxy L4 balancer endpoint that is created by Gitpod (which ends up being something like https://10.1.2.200) I get the same 403 errors on the wss HTTP requests.

Thanks for your help!
-b

Hi again!

Does anyone else have any ideas how I can resolve the Websocket 403 Forbidden error?

Thanks!
-b

Ack! Nevermind. I think I figured out the websocket issue. Essentially, I have a mutating admission controller that adjusts my ingresses. @chaitu454 kind of pointed me in the right direction. Once I disabled the admission controller and added ingresses for *.gitpod and *.ws.gitpod, the websocket error went away. Now I’m trying to figure out why I’m getting an OAuth Error from GitLab