Hi,
We are trying to deploy Gitpod into our private network using the Gitpod 0.6.0 helm chart. We have customized the templates to use our interal docker registry using this guide. However the dind container in image-builder pod continues to try to call https://registry-1.docker.io/v2/: where it gets a timeout as we don’t allow external connections. How can we resolve this issue?
Regards,
~Amulya
Hi @iamulya,
Having a Gitpod installation that works without internet access is something that we don’t fully support yet (or at least it isn’t tested). However, I will try to point you to some configuration values that could maybe help you to get a running Gitpod installation in your private network.
At first, I would have a look at these lines:
https://github.com/gitpod-io/gitpod/blob/5b6325f0de76c9d78aca768e39f341e5621d0068/chart/values.yaml#L296-L322 Here are some images configured that are used to run a Gitpod workspace. When you download these images and push them to your local registry, change the values in your local values.yaml file, this could be a good first step in the right direction.
There is also a baseImageName and workspaceImageName values for the imageBuilder that you can override to use local images from your registry: https://github.com/gitpod-io/gitpod/blob/5b6325f0de76c9d78aca768e39f341e5621d0068/chart/values.yaml#L217-L218
Please let me know if you make any progress with this information!
Best regards,
Cornelius
Hi Cornelius,
Thanks for your reply. Unfortunately that won’t solve the issue as image-builder/dind isn’t even contacting our docker registry - I only see the external docker registry in the logs…By the way, we were able to fix a lot of issues we had by hacking around the helm chart/templates. Most pods now are in running state, although we are not sure if its really all working properly:
Another major issue that we are facing is that all three ws-daemon pods remain on PodInitializing state with no log entries. Upon checking events we found that there are failed mount warnings:
Any known reason why this would happen and any pointers to resolve the issue?
Thanks,
~Amulya
For the containerd mount error check out these lines: https://github.com/gitpod-io/gitpod/blob/5b6325f0de76c9d78aca768e39f341e5621d0068/chart/values.yaml#L418-L424
Yes, that’s because the default images are without a registry domain (which leads to pulls from the official Docker Hub). But when you set for example:
components:
workspace:
defaultImage:
imagePrefix: "your-registry.example.com:5000/gitpod"
It will push the workspace-full image from your registry instead of from Docker Hub. I’m not sure if you really can configure all images that are used in Gitpod that way. But starting with my pointers would be a first step in that direction.
Does it make sense?
Hi Cornelius,
Right on target! Setting moby path as container storage worked. Node daemon status now running. Only image-builder left now. Thanks!
Cheers,
~Amulya
Hi Cornelius,
Even after changing the image prefix and registry name in image-builder to our internal artifactory registry, we continue to get the error that it can’t reach the default docker registry: registry-1.docker.io
I am guessing that the error might have been fixed for workspace-full, but not for the other images? How should we move forward here?
Regards,
~Amulya
Hi @iamulya,
There is a hard dependency on the alpine image on Docker Hub. That’s what image-builder prevents from coming up in your private (air-gapped) network. We’ve created an issue to fix this. I’ll take care of this now and let you know when it’s fixed.
Cornelius
@iamulya: I’ve create a PR that has the needed changes: https://github.com/gitpod-io/gitpod/pull/3228 See also the PR description for more information.
