Request startWorkspace failed with message: 14 UNAVAILABLE: No connection established

Hi, i install gitpod in myself k8s cluster and once i try open a gitlab repo I get the following error:

Request startWorkspace failed with message: 14 UNAVAILABLE: No connection established

and server logs error like this

{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","serviceContext":{"service":"server","version":"0.10.0"},"stack_trace":"Error: 14 UNAVAILABLE: No connection established\n    at Object.callErrorFromStatus (/app/node_modules/@grpc/grpc-js/build/src/call.js:31:26)\n    at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/build/src/client.js:176:52)\n    at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:336:141)\n    at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:299:181)\n    at /app/node_modules/@grpc/grpc-js/build/src/call-stream.js:145:78\n    at processTicksAndRejections (internal/process/task_queues.js:79:11)","component":"server","severity":"ERROR","time":"2021-09-30T09:29:51.008Z","environment":"production","region":"local","message":"Request startWorkspace failed with internal server error","error":"Error: 14 UNAVAILABLE: No connection established\n    at Object.callErrorFromStatus (/app/node_modules/@grpc/grpc-js/build/src/call.js:31:26)\n    at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/build/src/client.js:176:52)\n    at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:336:141)\n    at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:299:181)\n    at /app/node_modules/@grpc/grpc-js/build/src/call-stream.js:145:78\n    at processTicksAndRejections (internal/process/task_queues.js:79:11)","payload":{"method":"startWorkspace","args":["moccasin-zebra-xb39y2np",{"forceDefaultImage":false},{"_isCancelled":false}]}}

is the ws-daemon can’t connect ? ws-daemon pod logged nothing.
the 8080 port is bind on localdns ip

> netstat -nlpt|grep 8080
tcp        0      0 169.254.20.10:8080      0.0.0.0:*               LISTEN      399911/node-cache

i can’t connect ws-daemon hostPort 8080 on k8s node .

➜  ~ telnet master1 8080
Trying 192.168.192.231...
telnet: Unable to connect to remote host: Connection refused

is this right?

my k8s cluster not have loadblance, so i change proxy serviceType to ClusterIP :

spec:
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 80
    - name: https
      protocol: TCP
      port: 443
      targetPort: 443
  selector:
    app: gitpod
    component: proxy
    kind: pod
    stage: production
  clusterIP: 10.96.26.220
  type: ClusterIP
  sessionAffinity: None

and use traefik to reverse proxy.
this is my traefik config:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: gitpod-dashboard
  namespace: kube-app
spec:
  entryPoints:
    - websecure
  routes:
  - match: Host(`gitpod.xxx.com`)  && PathPrefix("/")
    kind: Rule
    services:
    - name:  proxy
      port: 443
  tls:
    certResolver: default
---

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: gitpod-https-wss
  namespace: kube-app
spec:
  entryPoints:
    - websecure
  routes:
  - match: HostRegexp(`*\.*\.gitpod.xxx.com`)
    kind: Rule
    services:
    - name:  proxy
      port: 443
      passHostHeader: true
  tls:
    certResolver: default

am i wrong? please help me !

try kubectl get pods
kubectl describe pod xxx
kubectl logs xxx

that should give more information on what is going on

to make life easier download kubetail,

but all pod is running. no error logs.


just server pod have error log:

{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","serviceContext":{"service":"server","version":"0.10.0"},"stack_trace":"Error: 14 UNAVAILABLE: No connection established\n    at Object.callErrorFromStatus (/app/node_modules/@grpc/grpc-js/build/src/call.js:31:26)\n    at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/build/src/client.js:176:52)\n    at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:336:141)\n    at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:299:181)\n    at /app/node_modules/@grpc/grpc-js/build/src/call-stream.js:145:78\n    at processTicksAndRejections (internal/process/task_queues.js:79:11)","component":"server","severity":"ERROR","time":"2021-10-08T02:30:09.015Z","environment":"production","region":"local","message":"Request startWorkspace failed with internal server error","error":"Error: 14 UNAVAILABLE: No connection established\n    at Object.callErrorFromStatus (/app/node_modules/@grpc/grpc-js/build/src/call.js:31:26)\n    at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/build/src/client.js:176:52)\n    at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:336:141)\n    at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:299:181)\n    at /app/node_modules/@grpc/grpc-js/build/src/call-stream.js:145:78\n    at processTicksAndRejections (internal/process/task_queues.js:79:11)","payload":{"method":"startWorkspace","args":["moccasin-zebra-xb39y2np",{"forceDefaultImage":false},{"_isCancelled":false}]}}