Gitpod Self-Hosted pricing clarification? Limited to only 50 hours?

I’ve been running the most recent Docker images off of http://eu.gcr.io/gitpod-core-dev/build/gitpod-k3s successfully. But I’ve noticed in Settings > Plans there’s a countdown of 50 hours? It seems like the plans visible on this part of the UI are the SaaS plans and not the ones shown for the Self-Hosted version if I’m going by the official website (https://www.gitpod.io/self-hosted). I’d just like a little more clarification on the pricing. Is the self-hosted version of Gitpod also limited to 50 hours a month? If so, it’s not obvious from looking at the website.

It is not limited to 50 hours. We’ve done some refactorings and it seems like the payment wall for Saas is now active in self-hosted. It is a bug. Can you file one here: Issues · gitpod-io/gitpod · GitHub ?

Any idea if a bug was created for this issue? I can’t seem to find one.
-b

@bmnave can you please confirm if you are using v0.10.0 which was released last night. If not, please upgrade.

@ghuntley Tried upgrading. My workspaces immediately stop immediately after container is pulled. No logs, nothing. Extremely frustrating. Downgraded to 0.9.0, everything works fine.

This bug is also present on 0.10.0

Thanks for the update @cass. Raised Gitpod Self-Hosted v0.1.0 is limited to only 50 hours · Issue #4950 · gitpod-io/gitpod · GitHub with high development priority.

Hi @enriquecaballero ,

I’m looking at Gitpod Self-Hosted v0.1.0 is limited to only 50 hours · Issue #4950 · gitpod-io/gitpod · GitHub but could not find anything obvious in the code.

Just to clarify: You only see the 50h mentioned in the UI, but not enforced on workspace startup (it should show up as error on the start-workspace-page)?

Tried upgrading. My workspaces immediately stop immediately after container is pulled. No logs, nothing.

This is not a typical sign of the SaaS paywall, but sounds like a different error altogether. What do you see on /workspaces?

This bug is also present on 0.10.0

@caas Could you share what exactly your symptoms are?

Thx!

@geropl Hey. Right, it shows up in the UI. It might be that it just enables the UI but it doesn’t enforce the 50 hours behind the scenes?

Regarding the workspaces stopping, it seems to be a different issue altogether. Unfortunately it’s the only thing impeding me from using v0.10.0. What should /workspaces look like? I tried looking up logs but the ws pod terminates too fast for me to actually describe the pod or view its logs. FWIW, I’m using the docker install.

It might be that it just enables the UI but it doesn’t enforce the 50 hours behind the scenes?

Exactly! Not nice as well, but no the source of the workspace stopping which blocks you atm.

tried looking up logs but the ws pod terminates too fast for me to actually describe the pod or view its logs

It would help if you could share the logs of ws-manager and maybe ws-daemon if you’re using not more than a handful of nodes (one?).

@geropi Ah, good call. I’ll re-update tonight and check out the logs for those two.

To summarize, and close the issue this topic was originally created for:

  • Gitpod Self-Hosted is free to download, install and use
  • some features, however, (prebuilds, snapshots, admin dashboard, cmp. Self-Hosted) require paid a license
  • there is some UI that suggests you only have 50h left but that does not restrict functionality (leftover from gitpod.io)

@geropl Finally got around to looking at the logs of ws-manager. These are the logs of ws-manager when attempting to start a workspace.

{"instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"info","message":"starting new workspace","serviceContext":{"service":"ws-manager","version":""},"severity":"INFO","time":"2021-08-05T02:37:10Z","userId":"9fbb280a-8dc2-49c7-9a5c-044216fca6d4","workspaceId":"coral-aardvark-e60xvqcj"}
{"instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"error","message":"workspace failed","serviceContext":{"service":"ws-manager","version":""},"severity":"ERROR","status":{"id":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","metadata":{"owner":"9fbb280a-8dc2-49c7-9a5c-044216fca6d4","meta_id":"coral-aardvark-e60xvqcj","started_at":{"seconds":1628131030}},"spec":{"workspace_image":"registry.gitpod.caballero.dev/workspace-images:f17dfbd08c85a33a9c6aa628d41175a2779137b84ac9d21c88cfb03b2544befd","ide_image":"gcr.io/gitpod-io/self-hosted/ide/code:0.10.0","url":"https://coral-aardvark-e60xvqcj.ws.gitpod.caballero.dev","exposed_ports":[{"port":3000,"url":"https://3000-coral-aardvark-e60xvqcj.ws.gitpod.caballero.dev"}],"timeout":"30m"},"conditions":{"failed":"container workspace ran with an error: exit code 1","service_exists":1,"deployed":1},"runtime":{"node_name":"6bd36d02c0f9","pod_name":"ws-7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","node_ip":"172.18.0.9"},"auth":{"owner_token":"PnH6MEEYwFPut7TgKhR68wGIPxicFhv6"}},"time":"2021-08-05T02:37:11Z","userId":"9fbb280a-8dc2-49c7-9a5c-044216fca6d4","workspaceId":"coral-aardvark-e60xvqcj"}
{"instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"error","message":"workspace in UNKNOWN phase","serviceContext":{"service":"ws-manager","version":""},"severity":"ERROR","status":{"id":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","metadata":{"owner":"9fbb280a-8dc2-49c7-9a5c-044216fca6d4","meta_id":"coral-aardvark-e60xvqcj","started_at":{"seconds":1628131030}},"spec":{"workspace_image":"registry.gitpod.caballero.dev/workspace-images:f17dfbd08c85a33a9c6aa628d41175a2779137b84ac9d21c88cfb03b2544befd","ide_image":"gcr.io/gitpod-io/self-hosted/ide/code:0.10.0","url":"https://coral-aardvark-e60xvqcj.ws.gitpod.caballero.dev","exposed_ports":[{"port":3000,"url":"https://3000-coral-aardvark-e60xvqcj.ws.gitpod.caballero.dev"}],"timeout":"30m"},"conditions":{"failed":"container workspace ran with an error: exit code 1","service_exists":1,"deployed":1},"runtime":{"node_name":"6bd36d02c0f9","pod_name":"ws-7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","node_ip":"172.18.0.9"},"auth":{"owner_token":"PnH6MEEYwFPut7TgKhR68wGIPxicFhv6"}},"time":"2021-08-05T02:37:11Z","userId":"9fbb280a-8dc2-49c7-9a5c-044216fca6d4","workspaceId":"coral-aardvark-e60xvqcj"}
{"instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"error","message":"workspace failed","serviceContext":{"service":"ws-manager","version":""},"severity":"ERROR","status":{"id":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","metadata":{"owner":"9fbb280a-8dc2-49c7-9a5c-044216fca6d4","meta_id":"coral-aardvark-e60xvqcj","started_at":{"seconds":1628131030}},"spec":{"workspace_image":"registry.gitpod.caballero.dev/workspace-images:f17dfbd08c85a33a9c6aa628d41175a2779137b84ac9d21c88cfb03b2544befd","ide_image":"gcr.io/gitpod-io/self-hosted/ide/code:0.10.0","url":"https://coral-aardvark-e60xvqcj.ws.gitpod.caballero.dev","exposed_ports":[{"port":3000,"url":"https://3000-coral-aardvark-e60xvqcj.ws.gitpod.caballero.dev"}],"timeout":"30m"},"conditions":{"failed":"container workspace ran with an error: exit code 1","service_exists":1,"deployed":1},"runtime":{"node_name":"6bd36d02c0f9","pod_name":"ws-7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","node_ip":"172.18.0.9"},"auth":{"owner_token":"PnH6MEEYwFPut7TgKhR68wGIPxicFhv6"}},"time":"2021-08-05T02:37:11Z","userId":"9fbb280a-8dc2-49c7-9a5c-044216fca6d4","workspaceId":"coral-aardvark-e60xvqcj"}
{"instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"error","message":"workspace in UNKNOWN phase","serviceContext":{"service":"ws-manager","version":""},"severity":"ERROR","status":{"id":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","metadata":{"owner":"9fbb280a-8dc2-49c7-9a5c-044216fca6d4","meta_id":"coral-aardvark-e60xvqcj","started_at":{"seconds":1628131030}},"spec":{"workspace_image":"registry.gitpod.caballero.dev/workspace-images:f17dfbd08c85a33a9c6aa628d41175a2779137b84ac9d21c88cfb03b2544befd","ide_image":"gcr.io/gitpod-io/self-hosted/ide/code:0.10.0","url":"https://coral-aardvark-e60xvqcj.ws.gitpod.caballero.dev","exposed_ports":[{"port":3000,"url":"https://3000-coral-aardvark-e60xvqcj.ws.gitpod.caballero.dev"}],"timeout":"30m"},"conditions":{"failed":"container workspace ran with an error: exit code 1","service_exists":1,"deployed":1},"runtime":{"node_name":"6bd36d02c0f9","pod_name":"ws-7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","node_ip":"172.18.0.9"},"auth":{"owner_token":"PnH6MEEYwFPut7TgKhR68wGIPxicFhv6"}},"time":"2021-08-05T02:37:11Z","userId":"9fbb280a-8dc2-49c7-9a5c-044216fca6d4","workspaceId":"coral-aardvark-e60xvqcj"}
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"cannot mark workspace 7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd with +gitpod.io/disposalStatus:\n    github.com/gitpod-io/gitpod/ws-manager/pkg/manager.(*Manager).markWorkspace\n        github.com/gitpod-io/gitpod/ws-manager/pkg/manager/annotations.go:121\n  - cannot find workspace 7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd:\n    github.com/gitpod-io/gitpod/ws-manager/pkg/manager.(*Manager).markWorkspace.func1\n        github.com/gitpod-io/gitpod/ws-manager/pkg/manager/annotations.go:93\n  - pod for workspace 7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd not found","instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"error","message":"was unable to update pod's disposal state - this will break someone's experience","serviceContext":{"service":"ws-manager","version":""},"severity":"ERROR","time":"2021-08-05T02:37:11Z","userId":"9fbb280a-8dc2-49c7-9a5c-044216fca6d4","workspaceId":"coral-aardvark-e60xvqcj"}

And the ws-daemon logs:

{"instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"info","message":"InitWorkspace called","serviceContext":{"service":"ws-daemon","version":""},"severity":"INFO","time":"2021-08-05T02:37:10Z","userId":"9fbb280a-8dc2-49c7-9a5c-044216fca6d4","workspaceId":"coral-aardvark-e60xvqcj"}
{"hooks":2,"instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"debug","message":"running lifecycle hooks","serviceContext":{"service":"ws-daemon","version":""},"severity":"DEBUG","state":"initializing","time":"2021-08-05T02:37:10Z","userId":"9fbb280a-8dc2-49c7-9a5c-044216fca6d4","workspaceId":"coral-aardvark-e60xvqcj"}
{"instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"info","message":"established IWS server","serviceContext":{"service":"ws-daemon","version":""},"severity":"INFO","time":"2021-08-05T02:37:10Z","userId":"9fbb280a-8dc2-49c7-9a5c-044216fca6d4","workspaceId":"coral-aardvark-e60xvqcj"}
{"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec[25056]: =\u003e nsexec container setup","time":"2021-08-05T02:37:10Z"}
{"file":"github.com/opencontainers/runc/libcontainer/cgroups/fscommon/open.go:37","func":"github.com/opencontainers/runc/libcontainer/cgroups/fscommon.prepareOpenat2.func1","level":"debug","msg":"openat2 not available, falling back to securejoin","time":"2021-08-05T02:37:10Z"}
{"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-0[25056]: ~\u003e nsexec stage-0","time":"2021-08-05T02:37:10Z"}
{"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-0[25056]: spawn stage-1","time":"2021-08-05T02:37:10Z"}
{"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-0[25056]: -\u003e stage-1 synchronisation loop","time":"2021-08-05T02:37:10Z"}
{"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-1[25060]: ~\u003e nsexec stage-1","time":"2021-08-05T02:37:10Z"}
{"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-1[25060]: unshare remaining namespace (except cgroupns)","time":"2021-08-05T02:37:10Z"}
{"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-1[25060]: spawn stage-2","time":"2021-08-05T02:37:10Z"}
{"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-1[25060]: request stage-0 to forward stage-2 pid (25061)","time":"2021-08-05T02:37:10Z"}
{"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-2[1]: ~\u003e nsexec stage-2","time":"2021-08-05T02:37:10Z"}
{"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-0[25056]: stage-1 requested pid to be forwarded","time":"2021-08-05T02:37:10Z"}
{"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-0[25056]: forward stage-1 (25060) and stage-2 (25061) pids to runc","time":"2021-08-05T02:37:10Z"}
{"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-1[25060]: signal completion to stage-0","time":"2021-08-05T02:37:10Z"}
{"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-1[25060]: \u003c~ nsexec stage-1","time":"2021-08-05T02:37:10Z"}
{"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-0[25056]: stage-1 complete","time":"2021-08-05T02:37:10Z"}
{"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-0[25056]: \u003c- stage-1 synchronisation loop","time":"2021-08-05T02:37:10Z"}
{"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-0[25056]: -\u003e stage-2 synchronisation loop","time":"2021-08-05T02:37:10Z"}
{"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-0[25056]: signalling stage-2 to run","time":"2021-08-05T02:37:10Z"}
{"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-2[1]: signal completion to stage-0","time":"2021-08-05T02:37:10Z"}
{"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-2[1]: \u003c= nsexec container setup","time":"2021-08-05T02:37:10Z"}
{"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-0[25056]: stage-2 complete","time":"2021-08-05T02:37:10Z"}
{"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-2[1]: booting up go runtime ...","time":"2021-08-05T02:37:10Z"}
{"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-0[25056]: \u003c- stage-2 synchronisation loop","time":"2021-08-05T02:37:10Z"}
{"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-0[25056]: \u003c~ nsexec stage-0","time":"2021-08-05T02:37:10Z"}
{"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"child process in init()","time":"2021-08-05T02:37:10Z"}
{"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"init: closing the pipe to signal completion","time":"2021-08-05T02:37:10Z"}
{"file":"github.com/opencontainers/runc/signals.go:104","func":"main.(*signalHandler).forward","level":"debug","msg":"sending signal to process urgent I/O condition","time":"2021-08-05T02:37:10Z"}
{"level":"debug","location":"/dst","message":"using Git initializer","severity":"DEBUG","time":"2021-08-05T02:37:10Z"}
{"level":"debug","location":"/dst/assignment-1","message":"Running git clone on workspace","severity":"DEBUG","stage":"init","time":"2021-08-05T02:37:10Z"}
{"instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"debug","message":"found sandbox - adding to label cache","podname":"ws-7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","serviceContext":{"service":"ws-daemon","version":""},"severity":"DEBUG","time":"2021-08-05T02:37:10Z","userId":"9fbb280a-8dc2-49c7-9a5c-044216fca6d4","workspaceId":"coral-aardvark-e60xvqcj"}
{"level":"info","location":"/dst/assignment-1","message":"Git operations complete","severity":"INFO","stage":"init","time":"2021-08-05T02:37:10Z"}
{"file":"github.com/opencontainers/runc/signals.go:94","func":"main.(*signalHandler).forward","level":"debug","msg":"process exited","pid":25061,"status":0,"time":"2021-08-05T02:37:10Z"}
{"hooks":1,"instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"debug","message":"running lifecycle hooks","serviceContext":{"service":"ws-daemon","version":""},"severity":"DEBUG","state":"ready","time":"2021-08-05T02:37:10Z","userId":"9fbb280a-8dc2-49c7-9a5c-044216fca6d4","workspaceId":"coral-aardvark-e60xvqcj"}
{"ID":"681b1940ef24c0e8a3172094b6958d262b957be20223c784da0b0f926b51133f","instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"debug","message":"found workspace container - updating label cache","podname":"ws-7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","serviceContext":{"service":"ws-daemon","version":""},"severity":"DEBUG","time":"2021-08-05T02:37:11Z","userId":"9fbb280a-8dc2-49c7-9a5c-044216fca6d4","workspaceId":"coral-aardvark-e60xvqcj"}
{"cid":"681b1940ef24c0e8a3172094b6958d262b957be20223c784da0b0f926b51133f","instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"debug","message":"found task","rootfs":"","serviceContext":{"service":"ws-daemon","version":""},"severity":"DEBUG","time":"2021-08-05T02:37:11Z","upperdir":"","userId":"9fbb280a-8dc2-49c7-9a5c-044216fca6d4","workspaceId":"coral-aardvark-e60xvqcj"}
{"container":"681b1940ef24c0e8a3172094b6958d262b957be20223c784da0b0f926b51133f","instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"info","message":"dispatch found new workspace container","serviceContext":{"service":"ws-daemon","version":""},"severity":"INFO","time":"2021-08-05T02:37:11Z","userId":"9fbb280a-8dc2-49c7-9a5c-044216fca6d4","workspaceId":"coral-aardvark-e60xvqcj"}
{"containerID":"681b1940ef24c0e8a3172094b6958d262b957be20223c784da0b0f926b51133f","instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"info","message":"started new resource governer","serviceContext":{"service":"ws-daemon","version":""},"severity":"INFO","time":"2021-08-05T02:37:11Z","userId":"9fbb280a-8dc2-49c7-9a5c-044216fca6d4","workspaceId":"coral-aardvark-e60xvqcj"}
{"level":"debug","message":"FSShift","serviceContext":{"service":"ws-daemon","version":""},"severity":"DEBUG","time":"2021-08-05T02:37:11Z","type":1}
{"level":"INFO", "msg": "nsexec:121 nsexec started"}
{"level":"INFO", "msg": "nsexec:136 join mnt namespace: 5"}
{"level":"INFO", "msg": "nsexec:142 chroot: 3"}
{"level":"INFO", "msg": "nsexec:148 chcwd: 4"}
{"level":"INFO", "msg": "nsexec:121 nsexec started"}
{"level":"INFO", "msg": "nsexec:136 join mnt namespace: 5"}
{"level":"INFO", "msg": "nsexec:142 chroot: 3"}
{"level":"INFO", "msg": "nsexec:148 chcwd: 4"}
{"level":"INFO", "msg": "nsexec:121 nsexec started"}
{"level":"INFO", "msg": "nsexec:136 join mnt namespace: 5"}
{"level":"INFO", "msg": "nsexec:142 chroot: 3"}
{"level":"INFO", "msg": "nsexec:148 chcwd: 4"}
{"args":["/app/nsinsider","mount-fusefs-mark","--source","/run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/681b1940ef24c0e8a3172094b6958d262b957be20223c784da0b0f926b51133f/rootfs","--merged","/var/gitpod/workspaces/7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd-daemon/mark","--upper","/var/gitpod/workspaces/7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd-daemon/upper","--work","/var/gitpod/workspaces/7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd-daemon/work","--uidmapping","0:33333:1:1:100000:65534","--gidmapping","0:33333:1:1:100000:65534"],"instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"fatal","message":"fuse-overlayfs ([/run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/681b1940ef24c0e8a3172094b6958d262b957be20223c784da0b0f926b51133f/rootfs/.supervisor/fuse-overlayfs -o lowerdir=/run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/681b1940ef24c0e8a3172094b6958d262b957be20223c784da0b0f926b51133f/rootfs,upperdir=/var/gitpod/workspaces/7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd-daemon/upper,workdir=/var/gitpod/workspaces/7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd-daemon/work,uidmapping=0:33333:1:1:100000:65534,gidmapping=0:33333:1:1:100000:65534 none /var/gitpod/workspaces/7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd-daemon/mark]) failed: \"fuse: device not found, try 'modprobe fuse' first\\nfuse-overlayfs: cannot mount: No such file or directory\\n\"\nexit status 1","serviceContext":{"service":"nsinsider","version":""},"severity":"CRITICAL","time":"2021-08-05T02:37:11Z"}
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"cannot run nsinsider: exit status 1","level":"error","message":"cannot mount fusefs mark","rootfs":"/run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/681b1940ef24c0e8a3172094b6958d262b957be20223c784da0b0f926b51133f/rootfs","serviceContext":{"service":"ws-daemon","version":""},"severity":"ERROR","time":"2021-08-05T02:37:11Z"}
{"instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"debug","message":"DisposeWorkspace called","req":"id:\"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd\"","serviceContext":{"service":"ws-daemon","version":""},"severity":"DEBUG","time":"2021-08-05T02:37:11Z","userId":"","workspaceId":""}
{"hooks":1,"instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"debug","message":"running lifecycle hooks","serviceContext":{"service":"ws-daemon","version":""},"severity":"DEBUG","state":"disposing","time":"2021-08-05T02:37:11Z","userId":"9fbb280a-8dc2-49c7-9a5c-044216fca6d4","workspaceId":"coral-aardvark-e60xvqcj"}
{"instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"info","message":"stopped IWS server","serviceContext":{"service":"ws-daemon","version":""},"severity":"INFO","time":"2021-08-05T02:37:11Z","userId":"9fbb280a-8dc2-49c7-9a5c-044216fca6d4","workspaceId":"coral-aardvark-e60xvqcj"}
{"containerID":"681b1940ef24c0e8a3172094b6958d262b957be20223c784da0b0f926b51133f","instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"debug","message":"resource controller shutting down","serviceContext":{"service":"ws-daemon","version":""},"severity":"DEBUG","time":"2021-08-05T02:37:11Z","userId":"","workspaceId":"coral-aardvark-e60xvqcj"}
{"hooks":0,"instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"debug","message":"running lifecycle hooks","serviceContext":{"service":"ws-daemon","version":""},"severity":"DEBUG","state":"disposed","time":"2021-08-05T02:37:11Z","userId":"9fbb280a-8dc2-49c7-9a5c-044216fca6d4","workspaceId":"coral-aardvark-e60xvqcj"}
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","attempt":0,"containerID":"681b1940ef24c0e8a3172094b6958d262b957be20223c784da0b0f926b51133f","error":"pods \"ws-7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd\" not found","instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"warning","message":"cannot mark workspace container as gone","serviceContext":{"service":"ws-daemon","version":""},"severity":"WARNING","time":"2021-08-05T02:37:21Z","userId":"","workspaceId":"coral-aardvark-e60xvqcj"}
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","attempt":1,"containerID":"681b1940ef24c0e8a3172094b6958d262b957be20223c784da0b0f926b51133f","error":"pods \"ws-7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd\" not found","instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"warning","message":"cannot mark workspace container as gone","serviceContext":{"service":"ws-daemon","version":""},"severity":"WARNING","time":"2021-08-05T02:37:23Z","userId":"","workspaceId":"coral-aardvark-e60xvqcj"}
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","attempt":2,"containerID":"681b1940ef24c0e8a3172094b6958d262b957be20223c784da0b0f926b51133f","error":"pods \"ws-7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd\" not found","instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"warning","message":"cannot mark workspace container as gone","serviceContext":{"service":"ws-daemon","version":""},"severity":"WARNING","time":"2021-08-05T02:37:25Z","userId":"","workspaceId":"coral-aardvark-e60xvqcj"}
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","attempt":3,"containerID":"681b1940ef24c0e8a3172094b6958d262b957be20223c784da0b0f926b51133f","error":"pods \"ws-7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd\" not found","instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"warning","message":"cannot mark workspace container as gone","serviceContext":{"service":"ws-daemon","version":""},"severity":"WARNING","time":"2021-08-05T02:37:27Z","userId":"","workspaceId":"coral-aardvark-e60xvqcj"}
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","attempt":4,"containerID":"681b1940ef24c0e8a3172094b6958d262b957be20223c784da0b0f926b51133f","error":"pods \"ws-7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd\" not found","instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"warning","message":"cannot mark workspace container as gone","serviceContext":{"service":"ws-daemon","version":""},"severity":"WARNING","time":"2021-08-05T02:37:29Z","userId":"","workspaceId":"coral-aardvark-e60xvqcj"}
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","attempt":5,"containerID":"681b1940ef24c0e8a3172094b6958d262b957be20223c784da0b0f926b51133f","error":"pods \"ws-7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd\" not found","instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"warning","message":"cannot mark workspace container as gone","serviceContext":{"service":"ws-daemon","version":""},"severity":"WARNING","time":"2021-08-05T02:37:31Z","userId":"","workspaceId":"coral-aardvark-e60xvqcj"}
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","attempt":6,"containerID":"681b1940ef24c0e8a3172094b6958d262b957be20223c784da0b0f926b51133f","error":"pods \"ws-7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd\" not found","instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"warning","message":"cannot mark workspace container as gone","serviceContext":{"service":"ws-daemon","version":""},"severity":"WARNING","time":"2021-08-05T02:37:33Z","userId":"","workspaceId":"coral-aardvark-e60xvqcj"}
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","attempt":7,"containerID":"681b1940ef24c0e8a3172094b6958d262b957be20223c784da0b0f926b51133f","error":"pods \"ws-7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd\" not found","instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"warning","message":"cannot mark workspace container as gone","serviceContext":{"service":"ws-daemon","version":""},"severity":"WARNING","time":"2021-08-05T02:37:35Z","userId":"","workspaceId":"coral-aardvark-e60xvqcj"}
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","attempt":8,"containerID":"681b1940ef24c0e8a3172094b6958d262b957be20223c784da0b0f926b51133f","error":"pods \"ws-7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd\" not found","instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"warning","message":"cannot mark workspace container as gone","serviceContext":{"service":"ws-daemon","version":""},"severity":"WARNING","time":"2021-08-05T02:37:37Z","userId":"","workspaceId":"coral-aardvark-e60xvqcj"}
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","attempt":9,"containerID":"681b1940ef24c0e8a3172094b6958d262b957be20223c784da0b0f926b51133f","error":"pods \"ws-7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd\" not found","instanceId":"7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd","level":"warning","message":"cannot mark workspace container as gone","serviceContext":{"service":"ws-daemon","version":""},"severity":"WARNING","time":"2021-08-05T02:37:39Z","userId":"","workspaceId":"coral-aardvark-e60xvqcj"}
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"pods \"ws-7c5e3bf4-0e97-4123-ba58-bef5ff8f31bd\" not found","level":"error","message":"cannot ensure workspace pod gets deleted","serviceContext":{"service":"ws-daemon","version":""},"severity":"ERROR","time":"2021-08-05T02:37:39Z"}

@geropl I only run into this in 0.10.0. Works fine with 0.9.0.

@geropl Alright, so I fixed it. But I don’t know how or why. I had to add the following to my values.yaml:

wsDaemon:
  userNamespaces:
    fsShift: fuse 
    shiftfsModuleLoader:
      enabled: true
1 Like

For what it’s worth, this is my attempt at “Dockerizing” the gitpod-k3s Docker image a bit more: Container Images / gitpod-k3s · GitLab (caballero.dev)

2 Likes

That’s really cool @enriquecaballero. What license is that project released under? (the LICENSE.md is missing)

Woops. I’ll go ahead and add that. Completely open source.

Done.

Nice!

Will take that for a spin next week!