Unable to open workspaces that use custom Docker image

Hi this morning I started having issues opening workspaces.

I getting this error message that seems to indicate an issue pulling the Docker image

The repository can be found here: https://github.com/appdev-projects/base-rails

Whatever this issue is, it is also preventing me from opening past workspaces or snapshots.

Is this a bug? Or is there a fix available for this?

Hey @jelaniwoods! Welcome to our community. :orange_heart:

I’ve just tried to open your repo in a new workspace and have got the same issue. It looks like there is something wrong with the docker image that you specified in your .gtipod.yml file, it’s looking for an image in localhost:8080 it looks like, but obviously cannot find it. I don’t know what your image looks like (jelaniwoods/appdev2021-base-rails@sha256:91427ecf043c8cdd8040bbd3626eb9bbbf3bee99d44ed3d1281504f71681d661) but that would be the first clue/place to look at debugging!

Hi @Pauline,

Thanks for the suggestion! I verified that the Docker image exists. I can pull and run it locally with

docker pull jelaniwoods/appdev2021-base-rails@sha256:91427ecf043c8cdd8040bbd3626eb9bbbf3bee99d44ed3d1281504f71681d661

It also seemed strange to me since I was able to make a new workspace using the same repo and docker image yesterday and everything worked— did anything change recently?

I also tested building the image from scratch using the Dockerfile in the repo, which took a while but eventually opened successfully, which is further confusing me.

Further testing reveals to me that if I remove the SHA in .gitpod.yml (as I did in this commit) the workspace can create.

So the issue seems to be caused by pulling images that specify the SHA. Does that seem like it could be the case?

That is an interesting discovery… I don’t believe we changed anything from our end, but let me ask internally. :thinking: I see that you’ve created a bug report too, so thanks for that! Stay tuned. :slight_smile:

1 Like

Thank you for analysing this problem further. I believe the issue arrises from a recent component update. I’ve filed an issue and we’ll look into this.

Do I understand correctly that you’re not blocked by this right now because you can resort to the un-digested form of the workspace image?

Thanks for the update @csweichel. I can create a new workspace after changing the image reference on GitHub, but I’m still unable to open any existing workspaces that are using the digest of the image, which I have many. This means any unsaved git changes are locked in workspaces that no longer open.

We use Gitpod for class, so unless there is a way to download uncommitted changes, we aren’t able to access workspaces that were previously working.

We have found and deployed a fix for this issue. Please test if you can now open your workspaces again?

@csweichel yes! So far I’m able to create/open workspaces that include the digest again. Thank you!

@csweichel While the main issue of being unable to open workspaces was solved, it looks like the image that’s being pulled for the workspace is not respecting the digest and is instead using the latest tag.

For example, the image I referenced before (jelaniwoods/appdev2021-base-rails@sha256:91427ecf043c8cdd8040bbd3626eb9bbbf3bee99d44ed3d1281504f71681d661)

only has ruby 2.7.3 installed while the image with the latest tag only uses Ruby 2.6.6.

Because of this issue, repositories like base-rails have issues running, since the expected Ruby version and gems are not installed.

Sorry for the inconvenience. We have deployed a fix for this issue. When I open https://github.com/appdev-projects/base-rails/tree/5e42ef0d03bcc267baf7f13b47f3b7f3ad71dc32 in Gitpod now, I see ruby in the correct version.

@csweichel I’m still seeing Ruby gem issues when opening existing workspaces or creating new ones.

I just created a new workspace using

https://gitpod.io/#https://github.com/appdev-projects/base-rails/tree/5e42ef0d03bcc267baf7f13b47f3b7f3ad71dc32

and the from the Ruby version I can tell it’s still using the :latest tag and not the digest. The workspace id is tan-peafowl-ds5jhzon if that helps.

I’m seeing this issue when trying to open existing workspaces as well where the ruby version is 2.6.6 instead of 2.7.3

Hi all,

I just wanted to point out that this issue is still occurring, and we’ve got hundreds of students who are blocked by it right now (during their final project week :grimacing:). They had already created their workspaces using an image identified by a SHA prior to whatever change is causing the failures, so it would be very inconvenient for them to have to start over with a fresh workspace.

Any updates you can provide would be very helpful! Thanks again,

— Raghu

Hey @raghubetina

Thank you for sharing the context, that sounds very stressful, let me see what I can do to help.

If I understand correctly, the problem is the workspace loads with Ruby 2.6.6, but, you require Ruby 2.7.3. Without version 2.7.3, the application code does not run. Is that correct?

Assuming I am correct with the above problem statement, I think this PR may help.

May I ask you to take a look at the above PR? Try it out via the Open in Gitpod button, and let me know what you think? It should start pretty quick and open a preview for the server.

Also, if I am missing something and this does not solve your problem, let us know?

Hi @raghubetina and @jelaniwoods ,

It looks like the default branch is okay (is using Ruby 2.7.3, updated a couple hours ago by @jelaniwoods, I can start workspaces). The image was changed to an un-digested form.

Are students able to absorb the changes from the default branch? After doing so are they still having trouble? Here’s hoping that change is able to unblock everyone.

Hi @raghubetina you shared some workspaces will not start. Are you able to start workspaces from the default (master) branch?

Assuming you can start workspaces originating from the default (master) branch, I recommend you try a workaround for one student, and if it helps, share with the others who cannot start their workspaces.

The workaround is to have a user (1) download their files from their workspace and (2) add them to git and start a fresh workspace / or start a fresh workspace and copy/paste the files manually.

When creating the new workspace, I assume it would be best to have them create a new branch from master and then open a workspace, but defer to you and @jelaniwoods. For example, I’m not sure if the assignment has students branching from something that is not the default (master) branch.

This video will show you how to download files from a workspace:

Let us know if you’re unable to download files from the stopped workspaces (which are not starting), or are still having trouble?