Stuck on 'Acquiring Node ... (1/3)'

I’m stuck in ‘Acquiring Node … (1/3)’

then I got redirected to workspaces page with this error message

‘Creating - Started 4 minutes ago: cannot pull image: Back-off pulling image “reg.gitpod.io:31000/remote/34681892-f968-4ad8-bb66-92506718213e”’

Tried starting a new workspace, which displayed a different error. And the error message changed for the original workspace as well.

Sorry to let you down. We are having an incident and are fixing it right now. We deployed a new component which should prescale and warm up clusters. Turns out it warms up a bit too much.
We will keep you posted.

It got worse now :frowning:

Any ETA on a fix?

I was able to connect to the pod using a VPN (EU), and going to code with that for now, until ws-us03 gets fixed.

However, the most important pod for me (PINHA (NEW)) is still stuck in the ‘Creating’ stage, and thus cannot stop or restart the container.

Is there a way to force stop that container, so that I can re-open it on ws-eu03 instead (with the saved but uncommitted files untouched)?

Hi @sanvit!

This incident is now resolved and the service is back to fully operational. See also status.gitpod.io

Unfortunately, there is no way to force-stop a container.

However, there is a hard timeout of 1 hour in Gitpod, for these cases where an incident prevents a workspace from starting or stopping correctly (it’s set to such a large value in order to leave enough time for any running workspace to still complete a full backup if needed, even if the Kubernetes master lost track of it).

So, long story short, the workspace should be back to “stopped” after a maximum of 1 hour, at which point you’ll be able to start it again and resume your work. Sorry for the disruption there.

1 Like

Thanks for the fast response! The issue seems to be resolved, and workspaces are working properly now.

However, all local changes were lost. Could you help me out?

The workspace ID is : tan-partridge-cv9ncbb7

Hi, issue is not solved for me and my class who are based in Singapore. We able to run Gitpod, but port forwarding does not work. Anything that requires nodejs, live server, flask etc does not work.

We also have workspaces that died with “last backup failed: workspace content was never ready”. Please advise.

Thanks for confirming!

Oh, that’s very unfortunate, I’m sorry about that. I’ve investigated this workspace (which was started 3 times today, at least once during the incident) but I wasn’t able to recover a backup prior to the latest one.

Please double-check whether any other workspaces in https://gitpod.io/workspaces/ have some uncommitted changes you can retrieve (should be explicitly displayed on the workspaces), but other than that, I’m afraid the changes are gone. :disappointed:

Hi @kunxin-chor! This does indeed sound like a different issue. If you’re still unable to expose ports, could you please open a new issue under https://github.com/gitpod-io/gitpod/issues ?

That’s yet another problem, but it seems like it could be related to the now-resolved incident. Please let me know if this happens again.

Fortunately this is actually mostly copied from my old repo so it shouldn’t take more than few minutes to recover :slight_smile:
Thanks for the confirmation!

By the way, do you keep more than one backup of the workspace?

1 Like

Aha, then I’m relieved. Thanks! (But still, any data loss should absolutely be prevented by design – This morning’s occurrence might have uncovered a new/rare bug we should address.)

That’s an excellent point. We did have a rolling system to keep the last 3 backups of a workspace in cloud storage (all removed if you manually delete the workspace), but I don’t think it works well because this morning your workspace had only a single backup (without your changes).

A working rolling backup would be awesome to ensure against data loss even if the system becomes unstable.

1 Like

Come to think of it, my workspace was changed to ‘offline’ (at least theia showed offline). I initially thought it was my internet connection, but come to think of it, it might be the earlier bug just stopping my workspace unexpectedly, and the workspace didn’t had the time to back up?

Maybe it would be possible to add to the starting screen of Gitpod workspace:

Does it take more than 3 minutes to build this workspace? Try checking https://status.gitpod.io