Error when Workspace is starting

Hi,
I’m using gitpod selfhosted 0.5.0 Helm on eks.

When I try to start a workspace, I got an " Request start workspace failed with message: 13 INTERNAL: cannot resolve workspace image: Error response from daemon: errors: denied: requested access to the resource is denied unauthorized: authentication required" error.

Could you help to resolve this issue?

We found this error in the server log:

Image-builder:

time=“2020-10-29T15:25:29.174247225Z” level=error msg=“Handler for GET /v1.38/distribution/registry/workspace-images:835c95423ec4d3254bb7fa41910686e75630a97a4b74f8cc9accb9d01a8d4841/json returned error: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required\n”
time=“2020-10-29T15:27:00.128714773Z” level=error msg=“Handler for GET /v1.38/distribution/registry/workspace-images:835c95423ec4d3254bb7fa41910686e75630a97a4b74f8cc9accb9d01a8d4841/json returned error: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required\n”

Server:

{“component”:“server”,“severity”:“INFO”,“time”:“2020-10-29T15:34:46.149Z”,“environment”:“production”,“region”:“local”,“context”:{“userId”:“d6061c3d-ac31-447d-9dc4-8e1a07514a6a”,“workspaceId”:“d76be4cc-cd86-4b3b-af32-0500402f1d12”},“message”:“startWorkspace”}
{"@type":“type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent",“serviceContext”:{“service”:“server”,“version”:“0.5.0”},“stack_trace”:"Error: 13 INTERNAL: cannot resolve workspace image: Error response from daemon: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required\n at Object.exports.createStatusError (/app/node_modules/grpc/src/common.js:91:15)\n at Object.onReceiveStatus (/app/node_modules/grpc/src/client_interceptors.js:1204:28)\n at InterceptingListener._callNext (/app/node_modules/grpc/src/client_interceptors.js:568:42)\n at InterceptingListener.onReceiveStatus (/app/node_modules/grpc/src/client_interceptors.js:618:8)\n at callback (/app/node_modules/grpc/src/client_interceptors.js:845:24)”,“component”:“server”,“severity”:“ERROR”,“time”:“2020-10-29T15:34:47.756Z”,“environment”:“production”,“region”:“local”,“message”:“Request startWorkspace failed with internal server error”,“error”:“Error: 13 INTERNAL: cannot resolve workspace image: Error response from daemon: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required\n at Object.exports.createStatusError (/app/node_modules/grpc/src/common.js:91:15)\n at Object.onReceiveStatus (/app/node_modules/grpc/src/client_interceptors.js:1204:28)\n at InterceptingListener._callNext (/app/node_modules/grpc/src/client_interceptors.js:568:42)\n at InterceptingListener.onReceiveStatus (/app/node_modules/grpc/src/client_interceptors.js:618:8)\n at callback (/app/node_modules/grpc/src/client_interceptors.js:845:24)”,“payload”:{“method”:“startWorkspace”,“args”:[“d76be4cc-cd86-4b3b-af32-0500402f1d12”,{"_isCancelled":false}]}}

and lot of from this type:

{"@type":“type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent",“serviceContext”:{“service”:“server”,“version”:“0.5.0”},“component”:“server”,“severity”:“ERROR”,“time”:“2020-10-29T15:36:27.695Z”,“environment”:“production”,“region”:“local”,“message”:"Error in fetching sampling strategy: Error: connect ECONNREFUSED 0.0.0.0:5778.”,“loggedViaConsole”:true}

Thanks

Hi,
the problem occurs when Gitpod has no proper access to the image repository. Could you please check the configuration. Here is a link to a template that can be used.

thanks the quick answer,
I would like to use the builtin registry.

Should I use this that case too?

Hi,
that setting is necessary when you use an external registry. The built-in registry is used by default and does not need any configuration.

Hi,
I am facing the same issue with self-hosted.
I found the below enty in the image-builder(service) log:

{“message”:“pre-cached Docker ref”,“ref”:“gitpod/workspace-full:latest”,“resolved-to”:“docker.io/gitpod/workspace-full:latest@sha256:854b8222bc34b621c0d2c72f41292ea2e9d8ab0c795f55c73dada06d2b445c50",“serviceContext”:{“service”:“image-builder”,“version”:""},“severity”:“debug”,“time”:"2020-10-30T09:27:12Z”}
{“a”:{“All”:false,“Explicit”:null},“message”:“registry not allowed”,“ref”:{},“reg”:“docker.io”,“serviceContext”:{“service”:“image-builder”,“version”:""},“severity”:“debug”,“time”:“2020-10-30T09:27:56Z”}

How can I set this to work? docker .io needs authentication to access the images. Any way to set it, or change it from docker .io to a public repository?

any ideas @wulfthimm ?
any help is highly appreciated!

We found the root of the problem it was my fault, I rewrite the registry name in the _helper, and the proxy’s server block cannot match with the new name :slight_smile:

What nazgrath mentioned seems a different issue, it is still in the logs.

1 Like

Hi @nazgrath,

if you log in to any registry using docker -config <PATH> login ... the credentials are written to the specified path. Including the values file for the registry (link) this config will be stored in as Kubernetes secret called image-builder-registry-secret. Please verify that it is crorrectly set up.