Cannot initialize workspace: content initializer failed (self-hosted))

After making a fresh self-hosted install, I try to create a workspace and I get

Any ideas? (I see some other forum posts about this but not clear they relate or if they have solutions.)

Are you using external S3 storage for your setup?
Check that the connections to this is working as it should.

@thomashansen I just followed the self-hosting guide which by default creates container-registry-gitpod-<accountId> bucket in S3. Looking in the bucket it looks like it has been able to create a bunch of files/folders

So bucket permissions don’t seem to be an issue.

Any other ideas to troubleshoot this?

Are you using the internal docker registry or an external one? That could be one of the causes of errors like this.

I set my system up with Rancher to manage the k3s cluster - as this makes it nice an easy to go through the various nodes and look for errors. You need to find the service that is causing error to get more detailed hints on what is going on.

I didn’t set up a special docker registry. In .env I left IMAGER_REGISTRY_WHITELIST` as blank.

I thought I should be able to open a workspace to one of the example workspaces.

I ran

kubectl logs ws-manager-65b8dc5b85-8tb7g -c ws-manager

and I get

{"instanceId":"1097d279-e72d-492d-806e-aba4d7fa1692","level":"error","message":"workspace failed","serviceContext":{"service":"ws-manager","version":""},"severity":"ERROR","status":{"id":"1097d279-e72d-492d-806e-aba4d7fa1692","metadata":{"owner":"a453d957-db87-4238-94e2-8ac3396b036a","meta_id":"tan-lamprey-pc01qqwi","started_at":{"seconds":1631655627}},"spec":{"workspace_image":"registry.default.svc.cluster.local/workspace-images:04fb4dc07a19ea36dd43fcc61467f05924c408ef4b4d5367c2012a75495da26d","ide_image":"eu.gcr.io/gitpod-core-dev/build/ide/code:aledbf-mk3.68","url":"https://tan-lamprey-pc01qqwi.ws.gitpod.mydomain.com","exposed_ports":[{"port":3000,"url":"https://3000-tan-lamprey-pc01qqwi.ws.gitpod.mydomain.com"}],"timeout":"30m"},"phase":6,"conditions":{"failed":"cannot initialize workspace: cannot initialize workspace: content initializer failed","service_exists":1,"deployed":1},"runtime":{"node_name":"ip-192-168-161-234.ec2.internal","pod_name":"ws-1097d279-e72d-492d-806e-aba4d7fa1692","node_ip":"192.168.161.234"},"auth":{"owner_token":"IHE8FEVkWL-Kp5OlSuStDkFz.TANsdlb"}},"time":"2021-09-14T21:46:28Z","userId":"a453d957-db87-4238-94e2-8ac3396b036a","workspaceId":"tan-lamprey-pc01qqwi"}
{"instanceId":"1097d279-e72d-492d-806e-aba4d7fa1692","level":"error","message":"workspace failed","serviceContext":{"service":"ws-manager","version":""},"severity":"ERROR","status":{"id":"1097d279-e72d-492d-806e-aba4d7fa1692","metadata":{"owner":"a453d957-db87-4238-94e2-8ac3396b036a","meta_id":"tan-lamprey-pc01qqwi","started_at":{"seconds":1631655627}},"spec":{"workspace_image":"registry.default.svc.cluster.local/workspace-images:04fb4dc07a19ea36dd43fcc61467f05924c408ef4b4d5367c2012a75495da26d","ide_image":"eu.gcr.io/gitpod-core-dev/build/ide/code:aledbf-mk3.68","url":"https://tan-lamprey-pc01qqwi.ws.gitpod.mydomain.com","timeout":"30m"},"phase":6,"conditions":{"failed":"cannot initialize workspace: cannot initialize workspace: content initializer failed","deployed":1},"runtime":{"node_name":"ip-192-168-161-234.ec2.internal","pod_name":"ws-1097d279-e72d-492d-806e-aba4d7fa1692","node_ip":"192.168.161.234"},"auth":{"owner_token":"IHE8FEVkWL-Kp5OlSuStDkFz.TANsdlb"}},"time":"2021-09-14T21:46:49Z","userId":"a453d957-db87-4238-94e2-8ac3396b036a","workspaceId":"tan-lamprey-pc01qqwi"}
{"instanceId":"1097d279-e72d-492d-806e-aba4d7fa1692","level":"error","message":"workspace failed","serviceContext":{"service":"ws-manager","version":""},"severity":"ERROR","status":{"id":"1097d279-e72d-492d-806e-aba4d7fa1692","metadata":{"owner":"a453d957-db87-4238-94e2-8ac3396b036a","meta_id":"tan-lamprey-pc01qqwi","started_at":{"seconds":1631655627}},"spec":{"workspace_image":"registry.default.svc.cluster.local/workspace-images:04fb4dc07a19ea36dd43fcc61467f05924c408ef4b4d5367c2012a75495da26d","ide_image":"eu.gcr.io/gitpod-core-dev/build/ide/code:aledbf-mk3.68","url":"https://tan-lamprey-pc01qqwi.ws.gitpod.mydomain.com","timeout":"30m"},"phase":6,"conditions":{"failed":"cannot initialize workspace: cannot initialize workspace: content initializer failed","deployed":1},"runtime":{"node_name":"ip-192-168-161-234.ec2.internal","pod_name":"ws-1097d279-e72d-492d-806e-aba4d7fa1692","node_ip":"192.168.161.234"},"auth":{"owner_token":"IHE8FEVkWL-Kp5OlSuStDkFz.TANsdlb"}},"time":"2021-09-14T21:46:50Z","userId":"a453d957-db87-4238-94e2-8ac3396b036a","workspaceId":"tan-lamprey-pc01qqwi"}
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"cannot find workspace 1097d279-e72d-492d-806e-aba4d7fa1692:\n    github.com/gitpod-io/gitpod/ws-manager/pkg/manager.(*Manager).modifyFinalizer.func1\n        github.com/gitpod-io/gitpod/ws-manager/pkg/manager/annotations.go:174\n  - pod for workspace 1097d279-e72d-492d-806e-aba4d7fa1692 not found","level":"error","message":"workspace monitor error","serviceContext":{"service":"ws-manager","version":""},"severity":"ERROR","time":"2021-09-14T21:46:54Z"}
{"instanceId":"1097d279-e72d-492d-806e-aba4d7fa1692","level":"error","message":"workspace failed","serviceContext":{"service":"ws-manager","version":""},"severity":"ERROR","status":{"id":"1097d279-e72d-492d-806e-aba4d7fa1692","metadata":{"owner":"a453d957-db87-4238-94e2-8ac3396b036a","meta_id":"tan-lamprey-pc01qqwi","started_at":{"seconds":1631655627}},"spec":{"workspace_image":"registry.default.svc.cluster.local/workspace-images:04fb4dc07a19ea36dd43fcc61467f05924c408ef4b4d5367c2012a75495da26d","ide_image":"eu.gcr.io/gitpod-core-dev/build/ide/code:aledbf-mk3.68","url":"https://tan-lamprey-pc01qqwi.ws.gitpod.mydomain.com","timeout":"30m"},"phase":6,"conditions":{"failed":"cannot initialize workspace: cannot initialize workspace: content initializer failed","deployed":1},"runtime":{"node_name":"ip-192-168-161-234.ec2.internal","pod_name":"ws-1097d279-e72d-492d-806e-aba4d7fa1692","node_ip":"192.168.161.234"},"auth":{"owner_token":"IHE8FEVkWL-Kp5OlSuStDkFz.TANsdlb"}},"time":"2021-09-14T21:46:54Z","userId":"a453d957-db87-4238-94e2-8ac3396b036a","workspaceId":"tan-lamprey-pc01qqwi"}

but that doesn’t help me understand the issue

Anyone have any other ideas?