Readiness probe failed

Hi
I set up the new 0.6.0 beta-4 on a self-hosted k3s machine. All pods are running but I get this warning from the workspace:

 Warning   Unhealthy                 pod/ws-32028f5a-c0c9-40be-b89d-598eb13f6506               Readiness probe failed: Get "http://10.42.0.32:22999/_supervisor/v1/status/content/wait/true": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

The workspace stucks in Acquiring Node (1/3), but according to the events it has been scheduled to my node

3m16s       Normal    Scheduled                 pod/ws-32028f5a-c0c9-40be-b89d-598eb13f6506               Placed pod [default/ws-32028f5a-c0c9-40be-b89d-598eb13f6506] 

I had the same behavior with older betas, except that the workspace stucks in step 3/3

Which version of k3s do you use?

I can reproduce this issue with v1.20.0+k3s2 but not with v1.18.13+k3s1. Don’t know yet what changed in k3s.

I use
k3s version v1.20.0+k3s2 (2ea6b163)
I will test older versions

I am doing exactly the same…
My cluster runs on Ubuntu 20.04 and k3s: v1.20.0+k3s2.

For dev i want to install a complete gitpod with registry and minio (both enabled).
After fixing some k3s issues, generating a wildcard ingress ! and setting up certs all pods came up proper.

i tried a lot different version:

imagePrefix: eu.gcr.io/gitpod-io/self-hosted/
v0.4.0
v0.6.0-beta1
v0.6.0-beta2
v0.6.0-beta3
v0.6.0-beta-

imagePrefix: eu.gcr.io/gitpod-core-dev/build/

master.327
master.333
master.340

Everytime the workspace pod came up its readyness probe fails… i wasnt able to fix it till now…

Any ideas what else i can do ?

Works with k3s v1.18.13+k3s1
Doesn’t work with v1.19.5+k3s1, v1.19.5+k3s2 and v1.20.0+k3s2
I did not test the k3s pre releases.

@barnybadzoo try to install the mentioned k3s version

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION={v1.18.13+k3s1}  --disable traefik" sh -s -

ok, wil give a try… but should work soon with 1.20…

One difference i found was under the wsdaemon: i needed to change the noderoot:
to /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io

in older chart this was mssing od not needed

so . for now i will setup my cluster with 1.18

The K3s version 1.18.13+k3s1 also didnt work for me…

mysetup: ubuntu 20.04 without most addons --disable=local-storage,servicelb,traefik
I use ingress-nginx, metallb and latest longhorn 1.10.

It tried different setups… also one with docker instead of containerd… but everytime after my setup is completly done i get the error like this issue:
https://github.com/k3s-io/k3s/issues/1948 but i am running at hyper-v not rasperry pi

I tested now a lot samples of k3s… None of them worked stable enough with longhorn… Got some issues with longhorn 1.10 which seams to crash after reboot so the cluster wont start anymore. (EXT4Fs- Cant find EXT$ FileSystem and corrupted superblocks at al virtualdisk …)

I open an issue for this: