Workspace creation fails with http 400

Hi,

When I try to create a workspace, i got 400 error. (gitpod self-hosted 0.5.0 on EKS)

The pod appears in the cluster but it stuck in a pending state.
image

When I describe the pod I got this error:

Events:
Type Reason Age From Message


Warning FailedScheduling 27m workspace-scheduler no suitable node found: Zero nodes available!

Labels on the nodes:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=t3.large
beta.kubernetes.io/os=linux
eks.amazonaws.com/capacityType=ON_DEMAND
eks.amazonaws.com/nodegroup=clusterNodegroupDefaultCapa-aau3LmpJe3yJ
eks.amazonaws.com/nodegroup-image=ami-06cfd5b2a2d58e09a
failure-domain.beta.kubernetes.io/region=eu-central-1
failure-domain.beta.kubernetes.io/zone=eu-central-1a
gitpod.io/diskPressure=true
gitpod.io/theia.0.5.0=available
gitpod.io/ws-sync=available
kubernetes.io/arch=amd64
kubernetes.io/hostname=ip-10-0-165-151.eu-central-1.compute.internal
kubernetes.io/os=linux
node.kubernetes.io/instance-type=t3.large
topology.kubernetes.io/region=eu-central-1
topology.kubernetes.io/zone=eu-central-1a

Why the nodes are not fine for the pod?
Could you help to resolve this issue?

Thanks

Hi,

could you provide us the with more information. If you run kubectl describe pod <POD_ID> you get more info. In most cases there is some information in the log at the bottom or a volume mount that cannot be accessed.

$ kubectl describe pod/ws-78c698ea-2867-4314-8a40-734d93e0f375
Name: ws-78c698ea-2867-4314-8a40-734d93e0f375
Namespace: default
Priority: 0
Node: < none>
Labels: app=gitpod
component=workspace
gitpod.io/networkpolicy=default
gpwsman=true
headless=false
metaID=d0cb25ab-2cfd-4135-b6e9-fc14958e0201
owner=1905b794-84c7-4881-9fc4-9f30ae70b85a
workspaceID=78c698ea-2867-4314-8a40-734d93e0f375
workspaceType=regular
Annotations: gitpod/admission: admit_owner_only
gitpod/contentInitializer:
EpUBCihodHRwczovL2dpdGxhYi5jb20vc3plbGxnYWJvcjkwL2NpY2EuZ2l0GAIiBm1hc3RlcioEY2ljYTJZEAIqVWh0dHBzOi8vZ2Fib3JncC53ZWJpZGUuc2llbWVucy5jbG91ZC…
gitpod/customTimeout: 30m
gitpod/id: 78c698ea-2867-4314-8a40-734d93e0f375
gitpod/never-ready: true
gitpod/ownerToken: < token>
gitpod/servicePrefix: d0cb25ab-2cfd-4135-b6e9-fc14958e0201
gitpod/traceid: AAAAAAAAAAC0KRYZ54Bod2+CKXYJ8JWRWuhyhglzYvwBAAAAAA==
gitpod/url: https://< URL>/workspace/d0cb25ab-2cfd-4135-b6e9-fc14958e0201/
kubernetes.io/psp: eks.privileged
prometheus.io/path: /metrics
prometheus.io/port: 23000
prometheus.io/scrape: true
Status: Pending
IP:
Containers:
workspace:
Image: registry< URL>/workspace-images:835c95423ec4d3254bb7fa41910686e75630a97a4b74f8cc9accb9d01a8d4841
Port: 23000/TCP
Host Port: 0/TCP
Limits:
cpu: 5
memory: 4Gi
Requests:
cpu: 1m
ephemeral-storage: 4Gi
memory: 2304Mi
Readiness: http-get http://:22999/ delay=0s timeout=1s period=1s #success=1 #failure=600
Environment:
GITPOD_REPO_ROOT: /workspace/cica
GITPOD_CLI_APITOKEN: M_47Tk+Iu@NfDdH_5Gurk:w[$uzBAD@a
GITPOD_WORKSPACE_ID: d0cb25ab-2cfd-4135-b6e9-fc14958e0201
GITPOD_INSTANCE_ID: 78c698ea-2867-4314-8a40-734d93e0f375
GITPOD_THEIA_PORT: 23000
THEIA_WORKSPACE_ROOT: /workspace/cica
GITPOD_HOST: https://< URL>
GITPOD_WORKSPACE_URL: https://< URL>/workspace/d0cb25ab-2cfd-4135-b6e9-fc14958e0201/
THEIA_SUPERVISOR_ENDPOINT: :22999
THEIA_WEBVIEW_EXTERNAL_ENDPOINT: webview-{{hostname}}
GITPOD_GIT_USER_NAME: < USERNAME>
GITPOD_GIT_USER_EMAIL: < USERNAME>@gmail.com
GITPOD_TASKS: []
GITPOD_RESOLVED_EXTENSIONS: {“vscode.bat@1.44.2”:{“fullPluginName”:“vscode.bat@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.clojure@1.44.2”:{“fullPluginName”:“vscode.clojure@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.coffeescript@1.44.2”:{“fullPluginName”:“vscode.coffeescript@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.cpp@1.44.2”:{“fullPluginName”:“vscode.cpp@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.csharp@1.44.2”:{“fullPluginName”:“vscode.csharp@1.44.2”,“url”:“local”,“kind”:“builtin”},“llvm-vs-code-extensions.vscode-clangd@0.1.5”:{“fullPluginName”:“llvm-vs-code-extensions.vscode-clangd@0.1.5”,“url”:“local”,“kind”:“builtin”},“vscode.css@1.44.2”:{“fullPluginName”:“vscode.css@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.debug-auto-launch@1.44.2”:{“fullPluginName”:“vscode.debug-auto-launch@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.emmet@1.44.2”:{“fullPluginName”:“vscode.emmet@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.fsharp@1.44.2”:{“fullPluginName”:“vscode.fsharp@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.go@1.44.2”:{“fullPluginName”:“vscode.go@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.groovy@1.44.2”:{“fullPluginName”:“vscode.groovy@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.handlebars@1.44.2”:{“fullPluginName”:“vscode.handlebars@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.hlsl@1.44.2”:{“fullPluginName”:“vscode.hlsl@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.html@1.44.2”:{“fullPluginName”:“vscode.html@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.ini@1.44.2”:{“fullPluginName”:“vscode.ini@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.java@1.48.0”:{“fullPluginName”:“vscode.java@1.48.0”,“url”:“local”,“kind”:“builtin”},“vscode.javascript@1.44.2”:{“fullPluginName”:“vscode.javascript@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.json@1.44.2”:{“fullPluginName”:“vscode.json@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.json-language-features@1.46.1”:{“fullPluginName”:“vscode.json-language-features@1.46.1”,“url”:“local”,“kind”:“builtin”},“vscode.less@1.44.2”:{“fullPluginName”:“vscode.less@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.log@1.44.2”:{“fullPluginName”:“vscode.log@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.lua@1.44.2”:{“fullPluginName”:“vscode.lua@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.make@1.44.2”:{“fullPluginName”:“vscode.make@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.markdown@1.44.2”:{“fullPluginName”:“vscode.markdown@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.npm@1.39.1”:{“fullPluginName”:“vscode.npm@1.39.1”,“url”:“local”,“kind”:“builtin”},“vscode.objective-c@1.44.2”:{“fullPluginName”:“vscode.objective-c@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.perl@1.44.2”:{“fullPluginName”:“vscode.perl@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.php@1.44.2”:{“fullPluginName”:“vscode.php@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.powershell@1.44.2”:{“fullPluginName”:“vscode.powershell@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.pug@1.44.2”:{“fullPluginName”:“vscode.pug@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.python@1.47.3”:{“fullPluginName”:“vscode.python@1.47.3”,“url”:“local”,“kind”:“builtin”},“vscode.r@1.44.2”:{“fullPluginName”:“vscode.r@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.razor@1.44.2”:{“fullPluginName”:“vscode.razor@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.ruby@1.44.2”:{“fullPluginName”:“vscode.ruby@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.rust@1.44.2”:{“fullPluginName”:“vscode.rust@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.scss@1.44.2”:{“fullPluginName”:“vscode.scss@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.shaderlab@1.44.2”:{“fullPluginName”:“vscode.shaderlab@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.shellscript@1.44.2”:{“fullPluginName”:“vscode.shellscript@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.sql@1.44.2”:{“fullPluginName”:“vscode.sql@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.swift@1.44.2”:{“fullPluginName”:“vscode.swift@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.typescript@1.44.2”:{“fullPluginName”:“vscode.typescript@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.typescript-language-features@1.44.2”:{“fullPluginName”:“vscode.typescript-language-features@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.vb@1.44.2”:{“fullPluginName”:“vscode.vb@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.xml@1.44.2”:{“fullPluginName”:“vscode.xml@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.yaml@1.44.2”:{“fullPluginName”:“vscode.yaml@1.44.2”,“url”:“local”,“kind”:“builtin”},“redhat.java@0.65.0”:{“fullPluginName”:“redhat.java@0.65.0”,“url”:“local”,“kind”:“builtin”},“vscjava.vscode-java-debug@0.27.1”:{“fullPluginName”:“vscjava.vscode-java-debug@0.27.1”,“url”:“local”,“kind”:“builtin”},“vscjava.vscode-java-dependency@0.9.0”:{“fullPluginName”:“vscjava.vscode-java-dependency@0.9.0”,“url”:“local”,“kind”:“builtin”},“ms-vscode.node-debug@1.38.4”:{“fullPluginName”:“ms-vscode.node-debug@1.38.4”,“url”:“local”,“kind”:“builtin”},“ms-vscode.node-debug2@1.33.0”:{“fullPluginName”:“ms-vscode.node-debug2@1.33.0”,“url”:“local”,“kind”:“builtin”},“ms-python.python@2020.7.96456”:{“fullPluginName”:“ms-python.python@2020.7.96456”,“url”:“local”,“kind”:“builtin”},“ms-vscode.Go@0.14.3”:{“fullPluginName”:“ms-vscode.Go@0.14.3”,“url”:“local”,“kind”:“builtin”},“redhat.vscode-xml@0.11.0”:{“fullPluginName”:“redhat.vscode-xml@0.11.0”,“url”:“local”,“kind”:“builtin”},“redhat.vscode-yaml@0.8.0”:{“fullPluginName”:“redhat.vscode-yaml@0.8.0”,“url”:“local”,“kind”:“builtin”},“bmewburn.vscode-intelephense-client@1.4.0”:{“fullPluginName”:“bmewburn.vscode-intelephense-client@1.4.0”,“url”:“local”,“kind”:“builtin”},“felixfbecker.php-debug@1.13.0”:{“fullPluginName”:“felixfbecker.php-debug@1.13.0”,“url”:“local”,“kind”:“builtin”},“rust-lang.rust@0.7.8”:{“fullPluginName”:“rust-lang.rust@0.7.8”,“url”:“local”,“kind”:“builtin”},“vscode.theme-abyss@1.44.2”:{“fullPluginName”:“vscode.theme-abyss@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.theme-kimbie-dark@1.44.2”:{“fullPluginName”:“vscode.theme-kimbie-dark@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.theme-monokai@1.44.2”:{“fullPluginName”:“vscode.theme-monokai@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.theme-monokai-dimmed@1.44.2”:{“fullPluginName”:“vscode.theme-monokai-dimmed@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.theme-quietlight@1.44.2”:{“fullPluginName”:“vscode.theme-quietlight@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.theme-red@1.44.2”:{“fullPluginName”:“vscode.theme-red@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.theme-solarized-dark@1.44.2”:{“fullPluginName”:“vscode.theme-solarized-dark@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.theme-solarized-light@1.44.2”:{“fullPluginName”:“vscode.theme-solarized-light@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.theme-tomorrow-night-blue@1.44.2”:{“fullPluginName”:“vscode.theme-tomorrow-night-blue@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.vscode-theme-seti@1.44.2”:{“fullPluginName”:“vscode.vscode-theme-seti@1.44.2”,“url”:“local”,“kind”:“builtin”},“vscode.merge-conflict@1.44.2”:{“fullPluginName”:“vscode.merge-conflict@1.44.2”,“url”:“local”,“kind”:“builtin”},“ms-vscode.references-view@0.0.47”:{“fullPluginName”:“ms-vscode.references-view@0.0.47”,“url”:“local”,“kind”:“builtin”},“EditorConfig.EditorConfig@0.15.1”:{“fullPluginName”:“EditorConfig.EditorConfig@0.15.1”,“url”:“local”,“kind”:“builtin”},“vscode.docker@1.47.3”:{“fullPluginName”:“vscode.docker@1.47.3”,“url”:“local”,“kind”:“builtin”}}
GITPOD_INTERVAL: 30000
GITPOD_MEMORY: 2415
Mounts:
/theia from vol-this-theia (ro)
/workspace from vol-this-workspace (rw)
Volumes:
vol-this-theia:
Type: HostPath (bare host directory volume)
Path: /var/gitpod/theia/theia-0.5.0
HostPathType: Directory
vol-this-workspace:
Type: HostPath (bare host directory volume)
Path: /var/gitpod/workspaces/78c698ea-2867-4314-8a40-734d93e0f375
HostPathType: DirectoryOrCreate
QoS Class: Burstable
Node-Selectors: < none>
Tolerations: node.kubernetes.io/disk-pressure:NoExecute
node.kubernetes.io/memory-pressure:NoExecute
node.kubernetes.io/network-unavailable:NoExecute for 30s
node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: < none>

I’ll try to collect the logs too

ws-manager-node log:
{"@type":“type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent",“containerID”:“06498a09f16159889f5b24357b82525fdcf1cfca11c2bc82006b66f85d141899”,“containerImage”:"",“error”:"not found\ngithub.com/containerd/containerd/errdefs.init\n\t/workspace/go/pkg/mod/github.com/containerd/containerd@v1.3.2/errdefs/errors.go:45\nruntime.doInit\n\t/home/gitpod/sdk/go1.15/src/runtime/proc.go:5625\nruntime.doInit\n\t/home/gitpod/sdk/go1.15/src/runtime/proc.go:5620\nruntime.doInit\n\t/home/gitpod/sdk/go1.15/src/runtime/proc.go:5620\nruntime.doInit\n\t/home/gitpod/sdk/go1.15/src/runtime/proc.go:5620\nruntime.doInit\n\t/home/gitpod/sdk/go1.15/src/runtime/proc.go:5620\nruntime.doInit\n\t/home/gitpod/sdk/go1.15/src/runtime/proc.go:5620\nruntime.doInit\n\t/home/gitpod/sdk/go1.15/src/runtime/proc.go:5620\nruntime.main\n\t/home/gitpod/sdk/go1.15/src/runtime/proc.go:191\nruntime.goexit\n\t/home/gitpod/sdk/go1.15/src/runtime/asm_amd64.s:1374\ncontainer “06498a09f16159889f5b24357b82525fdcf1cfca11c2bc82006b66f85d141899” in namespace “k8s.io”\ngithub.com/containerd/containerd/errdefs.FromGRPC\n\t/workspace/go/pkg/mod/github.com/containerd/containerd@v1.3.2/errdefs/grpc.go:107\ngithub.com/containerd/containerd.(*remoteContainers).Get\n\t/workspace/go/pkg/mod/github.com/containerd/containerd@v1.3.2/containerstore.go:50\ngithub.com/gitpod-io/gitpod/ws-manager-node/pkg/resourcegov.(*WorkspaceDispatch).handleContainerdEvent\n\t/tmp/build/components-ws-manager-node–app.3d794f0228d862e5d84e30d75b432214ca40e938/pkg/resourcegov/dispatch.go:192\ngithub.com/gitpod-io/gitpod/ws-manager-node/pkg/resourcegov.(*WorkspaceDispatch).Start\n\t/tmp/build/components-ws-manager-node–app.3d794f0228d862e5d84e30d75b432214ca40e938/pkg/resourcegov/dispatch.go:176\nruntime.goexit\n\t/home/gitpod/sdk/go1.15/src/runtime/asm_amd64.s:1374”,“message”:“cannot find container we just received a create event for”,“serviceContext”:{“service”:“ws-manager-node”,“version”:""},“severity”:“warning”,“time”:“2020-11-03T11:10:17Z”}

ws-manager-bridge:

{“component”:“ws-manager-bridge”,“severity”:“DEBUG”,“time”:“2020-11-03T11:27:52.066Z”,“environment”:“production”,“region”:“local”,“message”:“controlling instances”,“payload”:{“installation”:“local-00”}}
{"@type":“type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent",“serviceContext”:{“service”:“ws-manager-bridge”,“version”:“0.5.0”},“component”:“ws-manager-bridge”,“severity”:“ERROR”,“time”:“2020-11-03T11:28:23.886Z”,“environment”:“production”,“region”:“local”,“message”:"Error in fetching sampling strategy: Error: connect ECONNREFUSED 0.0.0.0:5778.”,“loggedViaConsole”:true}
{"@type":“type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent",“serviceContext”:{“service”:“ws-manager-bridge”,“version”:“0.5.0”},“component”:“ws-manager-bridge”,“severity”:“ERROR”,“time”:“2020-11-03T11:28:35.797Z”,“environment”:“production”,“region”:“local”,“message”:"Error in fetching sampling strategy: Error: connect ECONNREFUSED 0.0.0.0:5778.”,“loggedViaConsole”:true}
{“component”:“ws-manager-bridge”,“severity”:“DEBUG”,“time”:“2020-11-03T11:28:52.067Z”,“environment”:“production”,“region”:“local”,“message”:“controlling instances”,“payload”:{“installation”:“local-00”}}
{"@type":“type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent",“serviceContext”:{“service”:“ws-manager-bridge”,“version”:“0.5.0”},“component”:“ws-manager-bridge”,“severity”:“ERROR”,“time”:“2020-11-03T11:29:23.886Z”,“environment”:“production”,“region”:“local”,“message”:"Error in fetching sampling strategy: Error: connect ECONNREFUSED 0.0.0.0:5778.”,“loggedViaConsole”:true}

node:
kubectl describe node ip-10-0-165-151.eu-central-1.compute.internal
Name: ip-10-0-165-151.eu-central-1.compute.internal
Roles:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=t3.large
beta.kubernetes.io/os=linux
eks.amazonaws.com/capacityType=ON_DEMAND
eks.amazonaws.com/nodegroup=clusterNodegroupDefaultCapa-aau3LmpJe3yJ
eks.amazonaws.com/nodegroup-image=ami-06cfd5b2a2d58e09a
failure-domain.beta.kubernetes.io/region=eu-central-1
failure-domain.beta.kubernetes.io/zone=eu-central-1a
gitpod.io/diskPressure=true
gitpod.io/theia.0.5.0=available
gitpod.io/ws-sync=available
kubernetes.io/arch=amd64
kubernetes.io/hostname=ip-10-0-165-151.eu-central-1.compute.internal
kubernetes.io/os=linux
node.kubernetes.io/instance-type=t3.large
topology.kubernetes.io/region=eu-central-1
topology.kubernetes.io/zone=eu-central-1a
Annotations: node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Tue, 03 Nov 2020 10:28:19 +0100
Taints:
Unschedulable: false
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message


MemoryPressure False Tue, 03 Nov 2020 12:30:54 +0100 Tue, 03 Nov 2020 10:28:19 +0100 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Tue, 03 Nov 2020 12:30:54 +0100 Tue, 03 Nov 2020 10:28:19 +0100 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Tue, 03 Nov 2020 12:30:54 +0100 Tue, 03 Nov 2020 10:28:19 +0100 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Tue, 03 Nov 2020 12:30:54 +0100 Tue, 03 Nov 2020 10:28:49 +0100 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 10.0.165.151
Hostname: ip-10-0-165-151.eu-central-1.compute.internal
InternalDNS: ip-10-0-165-151.eu-central-1.compute.internal
Capacity:
attachable-volumes-aws-ebs: 25
cpu: 2
ephemeral-storage: 20959212Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 8063660Ki
pods: 35
Allocatable:
attachable-volumes-aws-ebs: 25
cpu: 1930m
ephemeral-storage: 18242267924
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 7305900Ki
pods: 35
System Info:
Machine ID: ec2a0fdf8f35cdf039ad7328cfe9937f
System UUID: EC2A0FDF-8F35-CDF0-39AD-7328CFE9937F
Boot ID: 62b70ed9-a2a1-4d68-9130-3a3e971d9249
Kernel Version: 4.14.198-152.320.amzn2.x86_64
OS Image: Amazon Linux 2
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://19.3.6
Kubelet Version: v1.17.11-eks-cfdc40
Kube-Proxy Version: v1.17.11-eks-cfdc40
ProviderID: aws:///eu-central-1a/i-059c2bb27eb78c72b
Non-terminated Pods: (10 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE


default minio-6975cf7bb8-thwqx 250m (12%) 0 (0%) 256Mi (3%) 0 (0%) 124m
default node-daemon-hx7gw 5m (0%) 0 (0%) 250Mi (3%) 250Mi (3%) 124m
default registry-76dbddbf7d-vmmlx 0 (0%) 0 (0%) 0 (0%) 0 (0%) 124m
default theia-server-5b749ccf6d-87qjm 100m (5%) 0 (0%) 16Mi (0%) 0 (0%) 124m
default ws-manager-5b89b649b6-x7dtc 100m (5%) 0 (0%) 32Mi (0%) 0 (0%) 23m
default ws-manager-bridge-78cfb5ffc8-m2q7n 100m (5%) 0 (0%) 64Mi (0%) 0 (0%) 124m
default ws-manager-node-qbhpq 100m (5%) 0 (0%) 32Mi (0%) 0 (0%) 22m
default ws-sync-fhkfh 100m (5%) 0 (0%) 200Mi (2%) 0 (0%) 23m
kube-system aws-node-xnnxw 10m (0%) 0 (0%) 0 (0%) 0 (0%) 125m
kube-system kube-proxy-xnzhn 100m (5%) 0 (0%) 0 (0%) 0 (0%) 125m
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits


cpu 865m (44%) 0 (0%)
memory 850Mi (11%) 250Mi (3%)
ephemeral-storage 0 (0%) 0 (0%)
attachable-volumes-aws-ebs 0 0
Events:

I have not recognized this line earlier. Please check the size of your nodes and compare it with this requirements for the workspace pods.

workspaceSizing:  # for n1-standard-16 (16 vCPUs, 60 GB memory)
  ## deprecated settings
  # memoryRequest: 2150   # 2.25 GB in MiB
  # memoryLimit: 11444   # 12 GB in MiB
  #
  # new sizing settings
  requests:
    cpu: "1m"
    memory: "2.25Gi"
    storage: "5Gi"
  limits:
    cpu: "5"
    memory: "12Gi"

Consider that you need more resources on the node as there are more pods running on it than only the workspace pod.

I use this settings now:

workspaceSizing:
requests:
cpu: 1m
memory: 2.25Gi
storage: 4Gi #5Gi
limits:
cpu: “5”
memory: 4Gi #12Gi

I have 3 t3-large node in the cluster
Name vCPUs Memory (GiB) SSD (GiB)
t3.large 2 8.0 8.0

Do you think it isn’t enough? (it is just a POC cluster)

Hi
@wulfthimm

We found this label on the pods: gitpod.io/diskPressure
After a little reverse engineering, we realized need at least 20GB free storage.
We reduced the limit in the config :smiley: then the container could start
image
but we got the same 400 error on the UI

from the pod describe:
Events:
Type Reason Age From Message


Normal Scheduled 13m workspace-scheduler Placed pod [default/ws-8f3e46a5-0f4a-451b-b172-8536fae44e89] on ip-10-0-145-246.eu-central-1.compute.internal
Normal Pulling 13m kubelet, ip-10-0-145-246.eu-central-1.compute.internal Pulling image “registry.< URL>/workspace-images:835c95423ec4d3254bb7fa41910686e75630a97a4b74f8cc9accb9d01a8d4841”
Normal Pulled 12m kubelet, ip-10-0-145-246.eu-central-1.compute.internal Successfully pulled image “registry.gaborgp.webide.siemens.cloud/workspace-images:835c95423ec4d3254bb7fa41910686e75630a97a4b74f8cc9accb9d01a8d4841”
Normal Created 12m kubelet, ip-10-0-145-246.eu-central-1.compute.internal Created container workspace
Normal Started 12m kubelet, ip-10-0-145-246.eu-central-1.compute.internal Started container workspace

from proxy:

{“serverPort”:“443”,“requestScheme”:“https”,“requestTime”:“0.000”,“context”:{},“remotePort”:“31008”,“serverName”:"~^(webview-)?+(?[a-z][0-9a-z\-]+)\.ws(-[a-z0-9]+)?\.< URL>$",“connection”:“3262”,“responseHeaders”:{“transfer-encoding”:“chunked”,“x-gitpod-region”:“production.gitpod.local.00”,“content-type”:“application/octet-stream”,“connection”:“close”},“requestHeaders”:{“host”:“fa015e86-05f3-4e86-80f5-df48fe079d6e.ws.gaborgp.webide.siemens.cloud”,“accept-encoding”:“gzip”,“user-agent”:“Go-http-client/1.1”},“httpRequest”:{“latency”:“0.000”,“requestMethod”:“GET”,“serverIp”:“10.0.213.61”,“userAgent”:“Go-http-client/1.1”,“status”:“400”,“protocol”:“HTTP/1.1”,“remoteIp”:“10.0.231.34”,“requestSize”:“161”,“responseSize”:“243”,“requestUrl”:"/gitpod/ready"},“upstreamAddr”:“172.20.128.214:8080”,“requestHost”:“fa015e86-05f3-4e86-80f5-df48fe079d6e.< URL>”,“severity”:400}

What else can we check?

I feel I found the root cause, in the WebSocket messages.

It seems somehow when creating a workspace it tries to use the AWS VPC DNS server and not the internal one.

{jsonrpc: “2.0”, method: “onInstanceUpdate”,…}
jsonrpc: “2.0”
method: “onInstanceUpdate”
params: {id: “f6d7ffe0-372e-4f03-acde-8660a29984fa”, workspaceId: “d38fcaac-974c-4158-bbef-68e5d18f4ba0”,…}
creationTime: “2020-11-05T10:49:00.448Z”
deleted: false
deployedTime: “2020-11-05T10:49:00.840Z”
id: “f6d7ffe0-372e-4f03-acde-8660a29984fa”
ideUrl: “https://< URL>/workspace/d38fcaac-974c-4158-bbef-68e5d18f4ba0/”
phasePersisted: “stopped”
region: “kiscica”
status: {phase: “stopped”, message: “”, timeout: “30m”, conditions: {,…}, exposedPorts: []}
conditions: {,…}
deployed: false
failed: “cannot initialize workspace: cannot use configured storage: Get “http://minio:9000/gitpod-user-f36072bb-03e3-4644-894f-eb3da9b56032/?location=”: dial tcp: lookup minio on 10.0.0.2:53: no such host; last backup failed: workspace does not exist. Please contact support if you need the workspace data.”
pullingImages: false
serviceExists: false
timeout: “”
exposedPorts: []
message: “”
phase: “stopped”
timeout: “30m”
stoppedTime: “2020-11-05T10:49:10.615Z”

Could tell why or which component tries to call this?

Thanks

looks like its the minio issue

the issue has been discussed here https://github.com/gitpod-io/gitpod/issues/2067

have a look at manually patching chart/templates/server-deployment.yaml for the moment, issue is on post 1, later on we discuss issues with uploading extensions

Good day,

When can a new release with this hotfix be expected ?

Thanks

Best Regards,

Bjoern