Knative development on GitPod

I want to developing with knative and would like to have a minikube cluster on my container to be able for local development inside gitpod and testing.

When I run a minikube on gitpod without virtualization I get an error

$ minikube start -p knative --vm-driver=none
๐Ÿ˜„  [knative] minikube v1.17.1 on Ubuntu 20.04 (amd64)
โœจ  Using the none driver based on user configuration

๐Ÿคท  Exiting due to PROVIDER_NONE_NOT_FOUND: The 'none' provider was not found: running the 'none' driver as a regular user requires sudo permissions

Is there any way to work around this?

1 Like

Do I understand it right that you try to run minikube inside a Gitpod workspace? Since everything runs in a Kubernetes container with limited rights that probably wonโ€™t work.

Yeah, I try to get some kubernetes dev environment set up on my GitPod. I activated experimental features, to get support for the docker daemon and improved the situation compared to my first steps. But Iโ€™m still stuckโ€ฆ

I tried minikube, k3s and kind and all three make some problems.

I started the docker daemon via sudo docker-up

minikube:

$minikube start --vm-driver=none --addons=ingress
  minikube v1.17.1 on Ubuntu 20.04 (amd64)
โœจ  Using the none driver based on existing profile
๐Ÿ‘  Starting control plane node minikube in cluster minikube
๐Ÿ”„  Restarting existing none bare metal machine for "minikube" ...
โ„น๏ธ  OS release is Ubuntu 20.04.1 LTS

โŒ  Exiting due to RUNTIME_ENABLE: sudo systemctl daemon-reload: exit status 1
stdout:

stderr:
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down

Seems minikube tries to restart docker ?!? but systemctl does not work as expected.

k3s

$ curl -sfL https://get.k3s.io | sh -s - --disable traefik
[INFO]  Finding release for channel stable
[INFO]  Using v1.20.2+k3s1 as release
[INFO]  Downloading hash https://github.com/rancher/k3s/releases/download/v1.20.2+k3s1/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/rancher/k3s/releases/download/v1.20.2+k3s1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping /usr/local/bin/kubectl symlink to k3s, command exists in PATH at /usr/bin/kubectl
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service โ†’ /etc/systemd/system/k3s.service.
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down

Same here. It seems all is set up correctly but the communication with docker is a problem.

kind

$ kind create cluster --config=kind-config.yaml
Creating cluster "kind" ...
 โœ“ Ensuring node image (kindest/node:v1.17.0) ๐Ÿ–ผ 
 โœ— Preparing nodes ๐Ÿ“ฆ  
docker run error: command "docker run --hostname kind-control-plane --name kind-control-plane --label io.x-k8s.kind.role=control-plane --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run --volume /var --volume /lib/modules:/lib/modules:ro --detach --tty --label io.x-k8s.kind.cluster=kind --publish=3080:80/TCP --publish=3443:443/TCP --publish=127.0.0.1:0:6443/TCP kindest/node:v1.17.0@sha256:9512edae126da271b66b990b6fff768fbb7cd786c7d39e86bdf55906352fdf62" failed with error: exit status 125
ERROR: failed to create cluster: docker run error: command "docker run --hostname kind-control-plane --name kind-control-plane --label io.x-k8s.kind.role=control-plane --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run --volume /var --volume /lib/modules:/lib/modules:ro --detach --tty --label io.x-k8s.kind.cluster=kind --publish=3080:80/TCP --publish=3443:443/TCP --publish=127.0.0.1:0:6443/TCP kindest/node:v1.17.0@sha256:9512edae126da271b66b990b6fff768fbb7cd786c7d39e86bdf55906352fdf62" failed with error: exit status 125

with my config file being

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    extraPortMappings:
      - containerPort: 80
        hostPort: 3080
      - containerPort: 443
        hostPort: 3443

Here the cluster can be created partially but something goes wrong again with the permissions and Iโ€™ve no idea how to deal with this.


My .gitpod.Dockerfile for these tests is

FROM gitpod/workspace-full

USER gitpod

# dev tools
RUN sudo apt-get -q update
RUN bash -c ". .nvm/nvm.sh && nvm install v12 && nvm alias default v12"
RUN npm install -g yarn

# kubectl
RUN sudo apt-get -q update && sudo apt-get install -y apt-transport-https gnupg2 curl
RUN curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
RUN echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
RUN sudo apt-get -q update 
RUN sudo apt-get -q install -y kubectl

# minikube
RUN sudo apt-get update -y
RUN sudo apt-get install -y conntrack
RUN curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && chmod +x minikube
RUN sudo cp minikube /usr/local/bin && rm minikube

# kind
RUN curl -Lo kind https://github.com/kubernetes-sigs/kind/releases/download/v0.7.0/kind-$(uname)-amd64
RUN chmod +x kind
RUN sudo mv kind /usr/local/bin

USER root

I also tried minikube with docker driver but this fails like kind.

$ minikube start --driver=docker --addons=ingress
๐Ÿ˜„  minikube v1.17.1 on Ubuntu 20.04 (amd64)
โœจ  Using the docker driver based on user configuration
๐Ÿ‘  Starting control plane node minikube in cluster minikube
๐Ÿšœ  Pulling base image ...
๐Ÿ’พ  Downloading Kubernetes v1.20.2 preload ...
    > preloaded-images-k8s-v8-v1....: 491.22 MiB / 491.22 MiB  100.00% 78.80 Mi
๐Ÿ”ฅ  Creating docker container (CPUs=2, Memory=16000MB) ...
๐Ÿคฆ  StartHost failed, but will try again: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var --security-opt apparmor=unconfined --memory=16000mb --memory-swap=16000mb --cpus=2 -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e: exit status 125
stdout:
86993432de164c79285649a61c37ef10b3a40bffe0227ac29aa64ce3d54a9feb

stderr:
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"cgroup\\\" to rootfs \\\"/workspace/.docker-root/overlay2/385e96e1faea7f80b448958a052104cfbc66bbe32288345fcedfc889faf8d347/merged\\\" at \\\"/sys/fs/cgroup\\\" caused \\\"operation not permitted\\\"\"": unknown.

๐Ÿคท  docker "minikube" container is missing, will recreate.

๐Ÿ”ฅ  Creating docker container (CPUs=2, Memory=16000MB) ...
๐Ÿ˜ฟ  Failed to start docker container. Running "minikube delete" may fix it: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var --security-opt apparmor=unconfined --memory=16000mb --memory-swap=16000mb --cpus=2 -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e: exit status 125
stdout:
385c8dbf4f3248249811a0f42fea6025d70d6c7bfe655cda8c5688c4a1b1254b

stderr:
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"cgroup\\\" to rootfs \\\"/workspace/.docker-root/overlay2/a6f43ad5ff7e75b715e62e81521a448e197fb3f5de409b55a3e5bd66d69c8785/merged\\\" at \\\"/sys/fs/cgroup\\\" caused \\\"operation not permitted\\\"\"": unknown.


โŒ  Exiting due to GUEST_PROVISION: Failed to start host: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var --security-opt apparmor=unconfined --memory=16000mb --memory-swap=16000mb --cpus=2 -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 gcr.io/k8s-minikube/kicbase:v0.0.17@sha256:1cd2e039ec9d418e6380b2fa0280503a72e5b282adea674ee67882f59f4f546e: exit status 125
stdout:
385c8dbf4f3248249811a0f42fea6025d70d6c7bfe655cda8c5688c4a1b1254b

stderr:
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"cgroup\\\" to rootfs \\\"/workspace/.docker-root/overlay2/a6f43ad5ff7e75b715e62e81521a448e197fb3f5de409b55a3e5bd66d69c8785/merged\\\" at \\\"/sys/fs/cgroup\\\" caused \\\"operation not permitted\\\"\"": unknown.


๐Ÿ˜ฟ  If the above advice does not help, please let us know: 
๐Ÿ‘‰  https://github.com/kubernetes/minikube/issues/new/choose

Since there doesnโ€™t seem to be a solution for this right now (at least none I could get running) :cry:, I added my experiences to the according issue on GitHub.

I think Iโ€™ll try to setup a Kubernetes cluster somewhere (like Digital Ocean) and try to get minikube running inside a self hosted GitPod. :roll_eyes: - Hopefully this works.

Iโ€™ll keep you updated on that and share my experience for others who want to work on Kubernetes projects inside a GitPod container. :+1: