/var/gitpod folder consume all my disk space 80G and all pods state change to evicted
Hi @suunavier! Oops, sorry that Gitpod is eating so much disk space. That’s definitely not expected / a bug.
Could you please try to figure out what is using up so much disk space? (For example, by running
du -hd 1 in
/var/gitpod, then in the biggest subdirectory, etc.)
Hi @jan, /var/gitpod/docker directory consume all my disk space. Thank for your response
Thanks for identifying this. When it’s the
docker/ directory that’s full, you may want to run
docker prune to free up some disk space again.
This can be achieved like so:
Switch to the appropriate Kubernetes context (
kubectxlists them, use
kubectl exec <image-builder-pod-name> -c dind -it sh \ export DOCKER_HOST=tcp://localhost:2375 docker system prune --filter "label!=gitpod.io/image-builder/protected"
Hint: As the garbage collector is very efficient in removing old workspaces, it is sufficient to run
docker system prunewithout
-all. In case all resources have to be removed, it may be neccessary to restart the
image-builderby deleting the pod with
kubectl delete image-builder-...to force the
image-builderto start from scratch and avoid running in a corrupted state.
Also, for completeness’ sake, here is how we deal with other directories if they use too much disk space (might not be applicable to you, but could be useful to others visiting this topic in the future):
workspaces/: if the workspace instance is still running: delete the contents; else: the whole folder (check with
kubectl get pods | grep <workspace_instance_id>)
sync-temp/: delete all contents, should be failsafe
theia/: delete older versions which are older than 1 day than the latest one (there might still be workspaces running depending on an older version!)
Hi @jan Thank for your response
I did like you mention but Only Total reclaimed space: 34.66MB, /var/gitpod/docker/vfs consume all disk space and all pods running state change to evicted
hi @suunavier! Can you run
docker images on that machine with the full disk and share the output with us? The should show which images docker has downloaded and how much disk space they use.
I solved no space left on device problem while building workspace image but now I have another problem that is gitpod stuck in acquiring node (1/3) and workspace state in gitpod is pending.