Ws-deamon error on Workspace startup

With latest code (main). When the Workspsace is started with a repo, its able to build image and pull the image, but fails to start the workspace.

ws-deamon errors out with the following.

“error”:“context deadline exceeded”,“instanceId”:“xxxx”,“level”:“error”,“message”:“PrepareForUserNS: cannot find workspace container”

The workspace image is there in registry, there is no issues pulling the images. Is it because of the new user namespace feature ?

How do i disable that and check ?

Indeed this looks like an issue with the user namespace feature. There is no way to disable that feature though - at this point it’s an integral part of Gitpod.

Which container runtime, Kubernetes version, Linux distribution and Kernel version are you running?

Its AWS EKS 1.16. I guessed if Amazon Linux 2 which was causing it. It changed to Ubuntu

worker_ami_name_filter=“ubuntu-eks/k8s_${var.kubernetes.version}*”
worker_ami_owner_id=“099720109477”

Still same issue. Containerd is not able to find the workspace id

====== Daemon config ====
wsDaemon:
containerRuntime:
nodeRoots:
- /var/lib
- /run/containerd/io.containerd.runtime.v1.linux/moby
userNamespaces:
shiftfsModuleLoader:
enabled: true

The issue was Sandbox container was not ready. After seeting workspaceSizeLimit and blobQuota its moved to next stages

The new issue is

“error”:"cannot clean workspace folder:\n github.com/gitpod-io/gitpod/content-service/pkg/initiali
│ zer.InitializeWorkspace\n github.com/gitpod-io/gitpod/content-service@v0.0.0-00010101000000-000000000000/pkg/initializer/initializer.go:321\n - openfdat /dst/lost+found: permission denied

I was able to workaround this by Skipping this delete. But the issue still exists.

Containerd : “Version”:“1.3.3-0ubuntu1~18.04.4”