Self-hosted k3s requirements

Hi there,

I’ve spent most of this week trying to get Gitpod working on my k3s clusters launched with terraform on Hetzner Cloud. It has been a long process of trial and error, debugging and finding the missing pieces in a combination of the docs, this forum, issue and in some config files from the gitpod repo (especially the docker based installs have been useful) to try and come up with the requirements for setting everything up. Interesting, but not very effective use of time. I’d much rather be playing with Gitpod and working out how we can use it with my project :wink:

So… I would very much welcome a complete list of the requirements and limitations on Gitpod self hosted, as I am not seeing this in enough detail in the docs to be able to get a setup working.

My current PoC setup using the Helm installer (testing version 0.8.0. and 0.9.0-alpha1 right now), I have so far found that I need the following:

  • ubuntu 18.04 - this seems to be the only supported version of ubuntu at them moment,

  • k3s (looks like it can be the latest version judging by the docker based setups, although the docs say 1.17 is the highest version) - has to be configured correctly with Calico instead of Flannel which is the default on k3s.

I have found that there are issues with 0.9.0-alpha1 with a containerd higher than 1.2 - so sticking with the very old 1.1.7 for now.

  • Make sure your certs are created and that a kubernetes secret (https-certificates) for these has been created (manually in the cluster) before trying to do the helm install.

  • use the internal registry by default - using external registries seems limited. External registries have to support nested repositories (had an issue with quay.io not liking the way gitpod creates repositories).

  • use minio for storage. I’ve tried using external storage (Scaleway’s S3 compatible storage, but this gives errors. Unless you have tons of storage space on your nodes (500Gi which is the default for minio), then you will also need to fix the default minio PVC to be less than 500GiB.

After working out the various combinations of things that should work. I’ve been so close to getting things working - but keep running into problems when starting up a workspace.

This issue is the one that is giving problems now with both 0.8.0 and 0.9.0-alpha1:

Has anyone been able to create a fully working self-hosted setup recently?

I also spent a long time on this about a year ago as well - and came to the conclusion that self-hosted was just not ready for prime time then. Is this still the case?

1 Like

What do you know ? Just as I had posted this, I was able to get 0.8.0 to start a workspace - when I turned off the preview feature. :grin:

Something of a win - but I am still really frustrated over the overall setup experience - especially as I really do think Gitpod has huge promise. But we need to be able to integrate it with our own setups to get the full benefit of it when working with tech and languages that are not completely mainstream and where we will need some extensive customizations on workspace setups.

FYI I now have a working setup with 0.9.0-alpha3 using ubuntu 20.04 (without calico) and the latest k3s!
Using ubuntu/k3s instances spun up on Hetzner Cloud with terraform.

Hi @thomashansen ,

I tried on a local Proxmox VM with Ubuntu 20.04 and k3d but 0.9.0-alpha3 failed to install. Do I need multiple nodes? Maybe you can shed some light for me :slight_smile:

I have not tried with a single node or using k3d. I have been using terraform to spin up k3s clusters on a hosting provider (Hetzner).

I have also tried on civo, who use k3s, but have not yet managed to get this to work.

Right now 0.9.0-alpha3 will not work either - so we need to wait for another release that works…

my master k3s node is started with the following options from a script:

sudo apt install -y open-iscsi

export K3S_TOKEN=${secret}
export K3S_KUBECONFIG_MODE="644"
export INSTALL_K3S_EXEC="--cluster-init --disable=traefik --disable-network-policy --kubelet-arg=cloud-provider=external --node-label gitpod.io/main-node=true --node-label gitpod.io/workload_meta=true"
export K3S_NODE_NAME=main

curl -sfL https://get.k3s.io | sh -s -

and the worker nodes with:

sudo apt install -y open-iscsi

export K3S_TOKEN=${secret}
export K3S_KUBECONFIG_MODE="644"
export K3S_URL=https://${leader_ip}:6443
export INSTALL_K3S_EXEC="agent --node-label gitpod.io/workload_workspace=true"
2 Likes

Thx for the quick reply! I will try.

Try symlinking the K3s containerd.io [ /var/run/k3s/containerd/containerd.sock ] to the default containerd.socket path [ /var/run/containerd/containerd.sock. ]

ln /var/run/k3s/containerd.sock /var/run/containerd/containerd.sock

1 Like