Which installation way works on GCP

To install self-hosted Gitpod on GKE with terraform, which way we should do ?
current branch: https://github.com/gitpod-io/gitpod/commit/19fb170e5f4aecef0551acbbedca24d71615bb0d

Getting started with Gitpod on GCP

it shows docker command as below

mkdir -p $PWD/gpinstall
docker run --rm -it \
    -v $PWD/gcloud:/root/.config/gcloud \
    -v $PWD/gpinstall:/workspace \
    gcr.io/gitpod-io/self-hosted/installer:0.7.0 \
    gcp

Installer tagged 0.7.0 not exist, so I used latest tag. terraform apply worked, but some pods are not running (proxy, resistry-facade, ws-proxy keep ContainerCreating) so I cannot open Gitpod workspace page with this way ( even if I use custom domain or use temporary xxx-xxx-xxx-xxx.ip.mygitpod.com )

Gitpod installation on GCP using Terraform

With using this terraform code ,I got some errors, same as I had found before https://github.com/gitpod-io/gitpod/issues/2985

Even if fixing there problem, ws-manager pod shows error (CrashLoopBackOff ) and I cannot open Gitpod workspace page.


Only I found were these 2 ways.
Are there any ways to install ?

Hi @egch,

the installer is outdated and needs an update. The Terraform script is the most up to date version.

Could you provide some logs for ws-manager to check which problem it is.

Best regards
Wulf

@wulfthimm Thank you for your help

ws-manager shows these log

 jsonPayload: {
  @type: "type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent"   
  error: "theiaHostPath: cannot be blank."   
  level: "fatal"   
  message: "invalid configuration"   
  serviceContext: {
   service: "ws-manager"    
   version: ""    
  }
 }

I figured out not only ws-manager, these pod also god error

ws-proxy : cannot start workspace info provider - will retry in 10 seconds

proxy : nginx: [emerg] unknown “proxy_domain” variable

content-service : ulling image “gcr.io/gitpod-io/self-hosted/content-service:0.6.0
Warning Failed 13m (x4 over 14m) kubelet Failed to pull image “gcr.io/gitpod-io/self-hosted/content-service:0.6.0”: rpc error: code = Unknown desc = failed to resolve image “gcr.io/gitpod-io/self-hosted/content-service:0.6.0”: no available registry endpoint: gcr.io/gitpod-io/self-hosted/content-service:0.6.0 not found

Hi @egch,

content-service is not available for version 0.6.0. Please update your version to 0.7.0.

In Google Container Registry, there are only 0.7.0-beta1
So should I use that version ?

in main.tf , it would be like this, isn’t it ?

  gitpod = {
    chart        = "../../../../chart"
    image_prefix = "gcr.io/gitpod-io/self-hosted/"
    version      = "0.7.0-beta1"
  }

Hi @egch,
sorry, 0.7.0-beta1 is the current one. You are right.

Ok, now I use 0.7.0-beta1, but still error on content-service, ws-manager(CrashLoopBackOff)

$ kubectl logs content-service-796c9ff998-db4mz
{"level":"warning","message":"no TLS configured - gRPC server will be unsecured","serviceContext":{"service":"content-service","version":""},"severity":"WARNING","time":"2021-02-26T08:30:50Z"}
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"invalid config:\n    github.com/gitpod-io/gitpod/content-service/pkg/storage.newPresignedGCPAccess\n        /tmp/build/components-content-service--app.c72eea5dc4fb813054091f520f845bd5018e78f8/pkg/storage/gcloud.go:693\n  - credentialsFile: stat /credentials/key.json: no such file or directory.","level":"fatal","message":"cannot create content service","serviceContext":{"service":"content-service","version":""},"severity":"CRITICAL","time":"2021-02-26T08:30:50Z"}
$ kubectl logs ws-manager-5898bd86cb-pkqw6
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"theiaHostPath: cannot be blank.","level":"fatal","message":"invalid configuration","serviceContext":{"service":"ws-manager","version":""},"severity":"CRITICAL","time":"2021-02-26T08:30:47Z"}

gitpod application itself is runnning ( I cann access gitpod.my-domain.com), but I wonder how these pod must work

Hi @egch,
the important line is

{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"invalid config:\n    github.com/gitpod-io/gitpod/content-service/pkg/storage.newPresignedGCPAccess\n        /tmp/build/components-content-service--app.c72eea5dc4fb813054091f520f845bd5018e78f8/pkg/storage/gcloud.go:693\n  - credentialsFile: stat /credentials/key.json: no such file or directory.","level":"fatal","message":"cannot create content service","serviceContext":{"service":"content-service","version":""},"severity":"CRITICAL","time":"2021-02-26T08:30:50Z"}

Gitpod is configured to use the GCP container registry. Therefore a service account has to be created with a service account key. This service account should have admin permissions for the registry.

The values.yaml should look like this:

components:
  imageBuilder:
    registryCerts: []
    registry:
      name: "eu.gcr.io/${project}"
      secretName: ${secretName}
     # not necessary when the secret is already present:
     # path: ${pathToServiceAccountKey}

  workspace:
    pullSecret:
      secretName: ${secretName}

docker-registry:
  enabled: false

gitpod_selfhosted:
  variants:
    customRegistry: true

When Terraform is used, which is also used by the installer, the service account is created and a Kubernetes secret is created with the content of the key file. Therefore only ${secretName} has to be set. You can check if the secret gitpod-registry is present in Kubernetes. The secret should have key named key.json with a value like:

{
	"auths": {
		"eu.gcr.io": {
			"auth": "${auth}"
		}
	}
}

${auth} is the base64 encoded service account key.

Please verify that the secret is present.

Okay, I found secret gitpod-registry secret in kubernetes and encoded data looks what we had expected.

{
	"auths": {
		"eu.gcr.io": {
			"auth": [KEY]
		}
	}
}

But still content-service, ws-manager is CrashLoopBackOff . I execute terraform with project-owner role account, so it should not be permission issue

Hi @egch,

that is good to know. Please verify that content-service and ws-manager are using that secret and mount it at the correct path.
With kubectl get deployment content-service -o yaml you could check if the secret is acutally mounted.

Okay, now checking each deployment.

Looking into these deployments,

contents-service looks not mount any secret

$ kubectl get deployment content-service -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    meta.helm.sh/release-name: gitpod
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2021-03-01T14:09:42Z"
  generation: 1
  labels:
    app: gitpod
    app.kubernetes.io/managed-by: Helm
    component: content-service
    gitpod.io/nodeService: content-service
    kind: deployment
    stage: production
  name: content-service
  namespace: default
  resourceVersion: "6868"
  selfLink: /apis/apps/v1/namespaces/default/deployments/content-service
  uid: dbb4a905-0fed-4c6b-82a9-b0d685a171b5
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: gitpod
      component: content-service
      gitpod.io/nodeService: content-service
      kind: pod
      stage: production
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      annotations:
        prometheus.io/path: /metrics
        prometheus.io/port: "9500"
        prometheus.io/scrape: "true"
      creationTimestamp: null
      labels:
        app: gitpod
        component: content-service
        gitpod.io/nodeService: content-service
        kind: pod
        stage: production
      name: content-service
    spec:
      containers:
      - args:
        - run
        - -v
        - --config
        - /config/config.json
        env:
        - name: KUBE_STAGE
          value: production
        - name: KUBE_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: VERSION
          value: 0.7.0-beta1
        - name: HOST_URL
          value: https://gitpod.my-domain.com
        - name: GITPOD_REGION
          value: local
        - name: GITPOD_INSTALLATION_LONGNAME
          value: production.gitpod.local.00
        - name: GITPOD_INSTALLATION_SHORTNAME
          value: local-00
        image: gcr.io/gitpod-io/self-hosted/content-service:0.7.0-beta1
        imagePullPolicy: Always
        name: content-service
        ports:
        - containerPort: 9500
          name: metrics
          protocol: TCP
        resources:
          requests:
            cpu: 100m
            memory: 32Mi
        securityContext:
          privileged: false
          runAsUser: 1000
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /config
          name: config
          readOnly: true
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: content-service
      serviceAccountName: content-service
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          name: content-service-config
        name: config
status:
  conditions:
  - lastTransitionTime: "2021-03-01T14:09:42Z"
    lastUpdateTime: "2021-03-01T14:10:03Z"
    message: ReplicaSet "content-service-796c9ff998" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2021-03-01T14:13:22Z"
    lastUpdateTime: "2021-03-01T14:13:22Z"
    message: Deployment does not have minimum availability.
    reason: MinimumReplicasUnavailable
    status: "False"
    type: Available
  observedGeneration: 1
  replicas: 1
  unavailableReplicas: 1
  updatedReplicas: 1

and ws-manager mount secret, but not sure it’s correct one

$ kubectl get deployment ws-manager -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    meta.helm.sh/release-name: gitpod
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2021-03-01T14:09:42Z"
  generation: 1
  labels:
    app: gitpod
    app.kubernetes.io/managed-by: Helm
    component: ws-manager
    kind: deployment
    stage: production
  name: ws-manager
  namespace: default
  resourceVersion: "7048"
  selfLink: /apis/apps/v1/namespaces/default/deployments/ws-manager
  uid: c959238a-ec85-4f6c-84de-32c904f2bbf8
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: gitpod
      component: ws-manager
      kind: pod
      stage: production
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      annotations:
        checksum/tlskey: 9a2a5cd3f62d89daa3a03798d1c21c90574edbd52441ef1665a1f9bbc3270ab8
        prometheus.io/path: /metrics
        prometheus.io/port: "9500"
        prometheus.io/scrape: "true"
      creationTimestamp: null
      labels:
        app: gitpod
        component: ws-manager
        kind: pod
        stage: production
      name: ws-manager
    spec:
      containers:
      - args:
        - run
        - -v
        - --config
        - /config/config.json
        env:
        - name: KUBE_STAGE
          value: production
        - name: KUBE_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: VERSION
          value: 0.7.0-beta1
        - name: HOST_URL
          value: https://gitpod.my-domain.com
        - name: GITPOD_REGION
          value: local
        - name: GITPOD_INSTALLATION_LONGNAME
          value: production.gitpod.local.00
        - name: GITPOD_INSTALLATION_SHORTNAME
          value: local-00
        - name: GRPC_GO_RETRY
          value: "on"
        image: gcr.io/gitpod-io/self-hosted/ws-manager:0.7.0-beta1
        imagePullPolicy: Always
        name: ws-manager
        ports:
        - containerPort: 9500
          name: metrics
          protocol: TCP
        - containerPort: 8080
          name: rpc
          protocol: TCP
        resources:
          requests:
            cpu: 100m
            memory: 32Mi
        securityContext:
          privileged: false
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /config
          name: config
          readOnly: true
        - mountPath: /workspace-template
          name: workspace-template
          readOnly: true
        - mountPath: /certs
          name: tls-certs
          readOnly: true
        - mountPath: /credentials
          name: gcloud-creds
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        runAsUser: 31002
      serviceAccount: ws-manager
      serviceAccountName: ws-manager
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          name: ws-manager-config
        name: config
      - name: tls-certs
        secret:
          defaultMode: 420
          secretName: ws-daemon-tls
      - configMap:
          defaultMode: 420
          name: workspace-template
        name: workspace-template
      - name: gcloud-creds
        secret:
          defaultMode: 420
          secretName: gcloud-creds
status:
  conditions:
  - lastTransitionTime: "2021-03-01T14:09:42Z"
    lastUpdateTime: "2021-03-01T14:10:23Z"
    message: ReplicaSet "ws-manager-7d8b8f565b" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2021-03-01T14:13:53Z"
    lastUpdateTime: "2021-03-01T14:13:53Z"
    message: Deployment does not have minimum availability.
    reason: MinimumReplicasUnavailable
    status: "False"
    type: Available
  observedGeneration: 1
  replicas: 1
  unavailableReplicas: 1
  updatedReplicas: 1

I wish if you could find something wrong

@wulfthimm
I wonder if you have any idea to fix this issue, or I should give more info.

Currently, Gitpod webpage itself worked but cannot open workspace form Gitlab;

accessing https://gitpod.my-domain.com/#https://gitlab.com/myaccount/test-project/-/tree/master/
gave me error,

Error 4: DEADLINE_EXCEEDED

Downloading workspace-image worked; I found image was saved in container registry.

Thank you