Issue Uploading Extensions

Greetings,

I am getting an error when attempting to upload extensions. In the workspace pod error log I see:

2020-06-19T00:37:04.460389583Z  {"component":"workspace","severity":"error","time":"2020-06-19T00:37:04.460Z","message":"Upload failed: Moved Temporarily","payload":{"name":"root","params":["Error: Moved Temporarily\n    at Request._callback (/theia/node_modules/@typefox/gitpod-extension/lib/node/extensions/gitpod-plugin-deployer.js:568:44)\n    at Request.self.callback (/theia/node_modules/request/request.js:185:22)\n    at Request.emit (events.js:189:13)\n    at Request.<anonymous> (/theia/node_modules/request/request.js:1161:10)\n    at Request.emit (events.js:189:13)\n    at IncomingMessage. (/theia/node_modules/request/request.js:1083:12)\n    at Object.onceWrapper (events.js:277:13)\n    at IncomingMessage.emit (events.js:194:15)\n    at endReadableNT (_stream_readable.js:1125:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"]}}
2020-06-19T00:37:04.460432862Z  {"component":"workspace","severity":"error","time":"2020-06-19T00:37:04.460Z","message":"Request upload failed with error: Upload failed","payload":{"name":"root","params":["Error: Upload failed\n    at GitpodPluginDeployer.<anonymous> (/theia/node_modules/@typefox/gitpod-extension/lib/node/extensions/gitpod-plugin-deployer.js:604:31)\n    at step (/theia/node_modules/@typefox/gitpod-extension/lib/node/extensions/gitpod-plugin-deployer.js:42:23)\n    at Object.throw (/theia/node_modules/@typefox/gitpod-extension/lib/node/extensions/gitpod-plugin-deployer.js:23:53)\n    at rejected (/theia/node_modules/@typefox/gitpod-extension/lib/node/extensions/gitpod-plugin-deployer.js:15:65)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"]}}
2020-06-19T00:37:04.558403165Z  {"component":"workspace","severity":"error","time":"2020-06-19T00:37:04.558Z","message":"Failed to share gaearon.subliminal@1.0.0 extension from gaearon.subliminal-1.0.0.vsix","payload":{"name":"root","params":["Error: Request 'upload' failed\n    at Proxy.<anonymous> (https://bfbf83dd-1c51-4f47-a9b3-3b5ae5c7211e.ws.mydomain.com/bundle.js:85:90000)\n    at t. (https://bfbf83dd-1c51-4f47-a9b3-3b5ae5c7211e.ws.mydomain.com/bundle.js:152:483374)\n    at https://bfbf83dd-1c51-4f47-a9b3-3b5ae5c7211e.ws.mydomain.com/bundle.js:152:478526\n    at Object.next (https://bfbf83dd-1c51-4f47-a9b3-3b5ae5c7211e.ws.mydomain.com/bundle.js:152:478631)\n    at a (https://bfbf83dd-1c51-4f47-a9b3-3b5ae5c7211e.ws.mydomain.com/bundle.js:152:477337)"]}}

And the proxy pod is showing:

2020-06-19T00:45:56.710341279Z  2020/06/19 00:45:56 [error] 121#121: *157280 upstream prematurely closed connection while reading response header from upstream, client: 10.244.1.1, server: mydomain.com, request: "PUT /plugins?id=d7a485ea-549f-4359-b528-5c24f6a05627 HTTP/1.1", subrequest: "/plugins-preflight", upstream: "http://10.106.130.246:3000/plugins/preflight?type=upload&token=fF7%2BaCPvF9Pa0AEjmoZ%2ByWVh6PqBjM5VEA0wyQs3FH4%3D&id=d7a485ea-549f-4359-b528-5c24f6a05627", host: "mydomain.com"
2020-06-19T00:45:56.710400109Z  2020/06/19 00:45:56 [error] 121#121: *157280 [lua] access_by_lua(lib.gitpod-plugins.conf:47):7: Bad Request: /plugins/preflight returned with code 502 while sending to client, client: 10.244.1.1, server: mydomain.com, request: "PUT /plugins?id=d7a485ea-549f-4359-b528-5c24f6a05627 HTTP/1.1", host: "mydomain.com"
2020-06-19T00:45:56.71503236Z  {"serverPort":"443","requestScheme":"https","requestTime":"120.008","context":{},"httpConnection":"close","remotePort":"65030","serverName":"mydomain.com","connection":"157280","responseHeaders":{"content-type":"text\/html","connection":"close","content-length":"161","x-gitpod-region":"production.gitpod.local.00","strict-transport-security":"max-age=31557600","location":"https:\/\/mydomain.com\/sorry\/#Error%20"},"requestHeaders":{"host":"mydomain.com","content-type":"*\/*","connection":"close","content-length":"742063"},"httpRequest":{"latency":"120.008","requestMethod":"PUT","serverIp":"10.244.1.236","status":"302","protocol":"HTTP\/1.1","remoteIp":"10.244.1.1","requestSize":"145","responseSize":"455","requestUrl":"\/plugins"},"requestHost":"mydomain.com","severity":200}

Any ideas on where the misconfiguration/issue is? The extension/theme I am trying to install is:

https://marketplace.visualstudio.com/items?itemName=gaearon.subliminal

Thanks!

Hi @astuffedtiger,

could you please provide some information about your setup, especially the domain configuration. That would help to identify the problem.

Sure! What exactly about the domain configuration are you looking for? I do have my DNS configured as mentioned in the setup docs (mydomain.com, *.mydomain.com and *.ws.mydomain.com pointing to my IP).

And the domain is resolved properly on your machines? The certificates are self-signed?

By “resolved properly on my machines”, what do you mean? The SSL certificates are from Let’s Encrypt.

Bump bump, I’d love to figure this one out. Where else can I go to troubleshoot, any logs I can share or anything?

Hey @wulfthimm, just to say I have this issue too on GitPod Self Hosted (sounds like a similar thread to [SelfHosted][Gitlab] Extensions install error)

Seems exactly same - a 302 is observed in the workspace container:

{"component":"workspace","severity":"error","time":"2020-07-01T22:47:53.231Z","message":"Upload failed: Moved Temporarily","payload":{"name":"root","params":["Error: Moved Temporarily\n    at Request._callback (/theia/node_modules/@typefox/gitpod-extension/lib/node/extensions/gitpod-plugin-deployer.js:568:44)\n    at Request.self.callback (/theia/node_modules/request/request.js:185:22)\n    at Request.emit (events.js:189:13)\n    at Request.<anonymous> (/theia/node_modules/request/request.js:1161:10)\n    at Request.emit (events.js:189:13)\n    at IncomingMessage.<anonymous> (/theia/node_modules/request/request.js:1083:12)\n    at Object.onceWrapper (events.js:277:13)\n    at IncomingMessage.emit (events.js:194:15)\n    at endReadableNT (_stream_readable.js:1125:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"]}}

Weirdly, I can see that actually the extension did upload into the GitPod workspace in the tmp directory:

gitpod ~ $ ls -lh /tmp/upload-raw-vscode-extensions/
total 20K 
-rw-r--r-- 1 gitpod gitpod 17K Jul  1 22:50 Dart-Code.flutter-3.8.1.vsix 

However, the Proxy pod logs show a 400 on /plugins?id=<> just before serving a 302 to the error page.

2020/07/05 16:47:26 [error] 126#126: *580225 [lua] access_by_lua(lib.gitpod-plugins.conf:47):7: Bad Request: /plugins/preflight returned with code 400 while sending to client, client: 10.32.0.1, server: <<domain>>, request: "PUT /plugins?id=5f2b832f-3b9c-4af4-97a6-7fc73ee657a4 HTTP/1.1", host: "<<domain>>"
{"serverPort":"443","requestScheme":"https","requestTime":"0.020","context":{},"httpConnection":"Upgrade","remotePort":"7763","serverName":"<<domain>>","connection":"580225","responseHeaders":{"content-type":"text\/html","connection":"close","content-length":"161","x-gitpod-region":"production.gitpod.local.00","strict-transport-security":"max-age=31557600","location":"https:\/\/<<domain<<\/sorry\/#Error%20"},"requestHeaders":{"host":"<<domain>>","x-forwarded-for":"31.53.xx.xx","x-forwarded-proto":"https","connection":"Upgrade","content-length":"17403","x-real-ip":"31.53.xx.xx","content-type":"*\/*"},"httpRequest":{"latency":"0.020","requestMethod":"PUT","serverIp":"10.32.0.10","status":"302","protocol":"HTTP\/1.1","remoteIp":"10.32.0.1","requestSize":"235","responseSize":"460","requestUrl":"\/plugins"},"requestHost":"<<domain>>","severity":200}

Can we provide anymore logs to find out where the fault might lie?

Well, thought I was getting closer… but then still stuck!

For context, I’m using an external Minio instance as defined in config:

$ cat values/minio.yaml 
gitpod:
  components:
wsSync:
  remoteStorage:
    kind: minio
    minio:
      endpoint: ip-of-minio:9000
      accessKey: example-Key
      secretKey: super-Secret-Key
      tmpdir: /tmp

  minio:
    enabled: false

I took a look at the Server pod logs…
I can see my access to the workspace and see the prepare upload message:

{"component":"server","severity":"INFO","time":"2020-07-07T06:03:13.110Z","environment":"production","region":"local","context":{"userId":"192a545d-****-****-****-************"},"message":"preparePluginUpload"}

However, it then errors because I don’t have a pod named “minio” (I haven’t deployed Minio on Kubernetes - hence why it can’t find it)

{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","serviceContext":{"service":"server","version":"v0.4.0"},"stack_trace":"Error: getaddrinfo ENOTFOUND minio minio:80\n    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:56:26)","component":"server","severity":"WARNING","time":"2020-07-07T06:03:13.625Z","environment":"production","region":"local","message":"Failed to create a signed URL for plugin with DB id 8a2ecdb4-1de8-463e-b9f0-c65d4430a987!","error":"Error: getaddrinfo ENOTFOUND minio minio:80\n    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:56:26)","payload":{"bucketName":"gitpod-my--domain-plugins","path":"192a545d-****-****-****-************.plugin.8a2ecdb4-1de8-463e-b9f0-c65d4430a987.Dart-Code.flutter~3.12.1","state":"uploading"}}

With the next message being a stack trace:

{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","serviceContext":{"service":"server","version":"v0.4.0"},"stack_trace":"Error: getaddrinfo ENOTFOUND minio minio:80\n    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:56:26)","component":"server","severity":"WARNING","time":"2020-07-07T06:03:13.625Z","environment":"production","region":"local","message":"Upload failed (Step: preflight)","error":"Error: getaddrinfo ENOTFOUND minio minio:80\n    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:56:26)","payload":"{ req:\n   IncomingMessage {\n     _readableState:\n      ReadableState {\n        objectMode: false,\n        highWaterMark: 16384,\n        buffer: BufferList { head: null, tail: null, length: 0 },\n        length: 0,\n        pipes: null,\n        pipesCount: 0,\n        flowing: null,\n        ended: true,\n        endEmitted: false,\n        reading: false,\n        sync: true,\n        needReadable: false,\n        emittedReadable: true,\n        readableListening: false,\n        resumeScheduled: false,\n        paused: true,\n        emitClose: true,\n        destroyed: false,\n        defaultEncoding: 'utf8',\n        awaitDrain: 0,\n        readingMore: true,\n        decoder: null,\n        encoding: null },\n     readable: true,\n     domain: null,\n     _events:\n      [Object: null prototype] { end: [Function: resetHeadersTimeoutOnReqEnd] },\n     _eventsCount: 1,\n     _maxListeners: undefined,\n     socket:\n      Socket {\n        connecting: false,\n        _hadError: false,\n        _handle: [TCP],\n        _parent: null,\n        _host: null,\n        _readableState: [ReadableState],\n        readable: true,\n        domain: null,\n        _events: [Object],\n        _eventsCount: 8,\n        _maxListeners: undefined,\n        _writableState: [WritableState],\n        writable: true,\n        allowHalfOpen: true,\n        _sockname: null,\n        _pendingData: null,\n        _pendingEncoding: '',\n        server: [Server],\n        _server: [Server],\n        timeout: 120000,\n        parser: [HTTPParser],\n        on: [Function: socketOnWrap],\n        _paused: false,\n        _httpMessage: [ServerResponse],\n        [Symbol(asyncId)]: 44194,\n        [Symbol(lastWriteQueueSize)]: 0,\n        [Symbol(timeout)]:\n         Timeout {\n           _called: false,\n           _idleTimeout: 120000,\n           _idlePrev: [TimersList],\n           _idleNext: [TimersList],\n           _idleStart: 263990,\n           _onTimeout: [Function: bound ],\n           _timerArgs: undefined,\n           _repeat: null,\n           _destroyed: false,\n           [Symbol(unrefed)]: true,\n           [Symbol(asyncId)]: 44195,\n           [Symbol(triggerId)]: 44194 },\n        [Symbol(kBytesRead)]: 0,\n        [Symbol(kBytesWritten)]: 0 },\n     connection:\n      Socket {\n        connecting: false,\n        _hadError: false,\n        _handle: [TCP],\n        _parent: null,\n        _host: null,\n        _readableState: [ReadableState],\n        readable: true,\n        domain: null,\n        _events: [Object],\n        _eventsCount: 8,\n        _maxListeners: undefined,\n        _writableState: [WritableState],\n        writable: true,\n        allowHalfOpen: true,\n        _sockname: null,\n        _pendingData: null,\n        _pendingEncoding: '',\n        server: [Server],\n        _server: [Server],\n        timeout: 120000,\n        parser: [HTTPParser],\n        on: [Function: socketOnWrap],\n        _paused: false,\n        _httpMessage: [ServerResponse],\n        [Symbol(asyncId)]: 44194,\n        [Symbol(lastWriteQueueSize)]: 0,\n        [Symbol(timeout)]:\n         Timeout {\n           _called: false,\n           _idleTimeout: 120000,\n           _idlePrev: [TimersList],\n           _idleNext: [TimersList],\n           _idleStart: 263990,\n           _onTimeout: [Function: bound ],\n           _timerArgs: undefined,\n           _repeat: null,\n           _destroyed: false,\n           [Symbol(unrefed)]: true,\n           [Symbol(asyncId)]: 44195,\n           [Symbol(triggerId)]: 44194 },\n        [Symbol(kBytesRead)]: 0,\n        [Symbol(kBytesWritten)]: 0 },\n     httpVersionMajor: 1,\n     httpVersionMinor: 0,\n     httpVersion: '1.0',\n     complete: true,\n     headers:\n      { host: 'apiserver',\n        connection: 'close',\n        'content-length': '0',\n        'x-real-ip': '31.53.xx.xx',\n        'x-forwarded-for': '31.53.xx.xx',\n        'x-forwarded-proto': 'https',\n        'content-type': '*/*' },\n     rawHeaders:\n      [ 'Host',\n        'apiserver',\n        'Connection',\n        'close',\n        'Content-Length',\n        '0',\n        'X-Real-IP',\n        '31.53.xx.xx',\n        'X-Forwarded-For',\n        '31.53.xx.xx',\n        'X-Forwarded-Proto',\n        'https',\n        'Content-Type',\n        '*/*' ],\n     trailers: {},\n     rawTrailers: [],\n     aborted: false,\n     upgrade: false,\n     url:\n      '/preflight?type=upload&token=fF7%2Ba<<token>>&id=8a2ecdb4-1de8-463e-b9f0-c65d4430a987',\n     method: 'GET',\n     statusCode: null,\n     statusMessage: null,\n     client:\n      Socket {\n        connecting: false,\n        _hadError: false,\n        _handle: [TCP],\n        _parent: null,\n        _host: null,\n        _readableState: [ReadableState],\n        readable: true,\n        domain: null,\n        _events: [Object],\n        _eventsCount: 8,\n        _maxListeners: undefined,\n        _writableState: [WritableState],\n        writable: true,\n        allowHalfOpen: true,\n        _sockname: null,\n        _pendingData: null,\n        _pendingEncoding: '',\n        server: [Server],\n        _server: [Server],\n        timeout: 120000,\n        parser: [HTTPParser],\n        on: [Function: socketOnWrap],\n        _paused: false,\n        _httpMessage: [ServerResponse],\n        [Symbol(asyncId)]: 44194,\n        [Symbol(lastWriteQueueSize)]: 0,\n        [Symbol(timeout)]:\n         Timeout {\n           _called: false,\n           _idleTimeout: 120000,\n           _idlePrev: [TimersList],\n           _idleNext: [TimersList],\n           _idleStart: 263990,\n           _onTimeout: [Function: bound ],\n           _timerArgs: undefined,\n           _repeat: null,\n           _destroyed: false,\n           [Symbol(unrefed)]: true,\n           [Symbol(asyncId)]: 44195,\n           [Symbol(triggerId)]: 44194 },\n        [Symbol(kBytesRead)]: 0,\n        [Symbol(kBytesWritten)]: 0 },\n     _consuming: false,\n     _dumped: false,\n     next: [Function: next],\n     baseUrl: '/plugins',\n     originalUrl:\n      '/plugins/preflight?type=upload&token=fF7%2Ba<<token>>&id=8a2ecdb4-1de8-463e-b9f0-c65d4430a987',\n     _parsedUrl:\n      Url {\n        protocol: null,\n        slashes: null,\n        auth: null,\n        host: null,\n        port: null,\n        hostname: null,\n        hash: null,\n        search:\n         '?type=upload&token=fF7%2Ba<<token>>&id=8a2ecdb4-1de8-463e-b9f0-c65d4430a987',\n        query:\n         'type=upload&token=fF7%2Ba<<token>>&id=8a2ecdb4-1de8-463e-b9f0-c65d4430a987',\n        pathname: '/preflight',\n        path:\n         '/preflight?type=upload&token=fF7%2Ba<<token>>&id=8a2ecdb4-1de8-463e-b9f0-c65d4430a987',\n        href:\n         '/preflight?type=upload&token=fF7%2Ba<<token>>&id=8a2ecdb4-1de8-463e-b9f0-c65d4430a987',\n        _raw:\n         '/preflight?type=upload&token=fF7%2Ba<<token>>&id=8a2ecdb4-1de8-463e-b9f0-c65d4430a987' },\n     params: {},\n     query:\n      { type: 'upload',\n        token: 'fF7+a<<token>>',\n        id: '8a2ecdb4-1de8-463e-b9f0-c65d4430a987' },\n     res:\n      ServerResponse {\n        domain: null,\n        _events: [Object],\n        _eventsCount: 1,\n        _maxListeners: undefined,\n        output: [],\n        outputEncodings: [],\n        outputCallbacks: [],\n        outputSize: 0,\n        writable: true,\n        _last: false,\n        chunkedEncoding: false,\n        shouldKeepAlive: false,\n        useChunkedEncodingByDefault: false,\n        sendDate: true,\n        _removedConnection: false,\n        _removedContLen: false,\n        _removedTE: false,\n        _contentLength: null,\n        _hasBody: true,\n        _trailer: '',\n        finished: false,\n        _headerSent: false,\n        socket: [Socket],\n        connection: [Socket],\n        _header: null,\n        _onPendingData: [Function: bound updateOutgoingData],\n        _sent100: false,\n        _expect_continue: false,\n        end: [Function: end],\n        req: [Circular],\n        locals: [Object: null prototype] {},\n        writeHead: [Function: writeHead],\n        [Symbol(isCorked)]: false,\n        [Symbol(outHeadersKey)]: [Object] },\n     body: {},\n     _parsedOriginalUrl:\n      Url {\n        protocol: null,\n        slashes: null,\n        auth: null,\n        host: null,\n        port: null,\n        hostname: null,\n        hash: null,\n        search:\n         '?type=upload&token=fF7%2Ba<<token>>&id=8a2ecdb4-1de8-463e-b9f0-c65d4430a987',\n        query:\n         'type=upload&token=fF7%2Ba<<token>>&id=8a2ecdb4-1de8-463e-b9f0-c65d4430a987',\n        pathname: '/plugins/preflight',\n        path:\n         '/plugins/preflight?type=upload&token=fF7%2Ba<<token>>&id=8a2ecdb4-1de8-463e-b9f0-c65d4430a987',\n        href:\n         '/plugins/preflight?type=upload&token=fF7%2Ba<<token>>&id=8a2ecdb4-1de8-463e-b9f0-c65d4430a987',\n        _raw:\n         '/plugins/preflight?type=upload&token=fF7%2Ba<<token>>&id=8a2ecdb4-1de8-463e-b9f0-c65d4430a987' },\n     sessionStore:\n      MySQLStore {\n        connection: [Pool],\n        options: [Object],\n        generate: [Function],\n        _events: [Object],\n        _eventsCount: 2,\n        _expirationInterval:\n         Timeout {\n           _called: false,\n           _idleTimeout: 900000,\n           _idlePrev: [TimersList],\n           _idleNext: [TimersList],\n           _idleStart: 23555,\n           _onTimeout: [Function],\n           _timerArgs: undefined,\n           _repeat: 900000,\n           _destroyed: false,\n           [Symbol(unrefed)]: false,\n           [Symbol(asyncId)]: 428,\n           [Symbol(triggerId)]: 157 } },\n     sessionID: '73a037be-3943-4fd2-97ef-1c4f5697cb73',\n     session: Session { cookie: [Object] },\n     _passport: { instance: [Authenticator] },\n     route:\n      Route { path: '/preflight', stack: [Array], methods: [Object] } } }"}

I then did a bit more digging… executing:

$ kubectl describe deployment server

I see that, actually, the server hasn’t been provided the values from values/minio.yaml:

  GITPOD_STORAGE_CLIENT:               minio
  MINIO_END_POINT:                     minio
  MINIO_ACCESS_KEY:                    EXAMPLEvalue
  MINIO_SECRET_KEY:                    Someone.Should/ReallyChangeThisKey!!

Which seems to be because the helm chart isn’t set to override it :frowning: because, we set the Access and Secret key in .Value.components.wsSync.remoteStorage.minio.accessKey, but even if that’s set, the MINIO endpoint wouldn’t be picked up in: ./charts/gitpod/templates/server-deployment.yaml

    {{- if eq .Values.components.wsSync.remoteStorage.kind "minio" }}
    - name: GITPOD_STORAGE_CLIENT
      value: minio
    - name: MINIO_END_POINT
      value: minio
    - name: MINIO_ACCESS_KEY
      value: {{ .Values.minio.accessKey }}
    - name: MINIO_SECRET_KEY
      value: {{ .Values.minio.secretKey }}
    {{- end }}

So, locally, I tried patching it… and creating the gitpod installation again… but this time the server wouldn’t start :man_facepalming:

{"component":"server","severity":"INFO","time":"2020-07-07T07:47:13.888Z","environment":"production","region":"local","message":"Using MinIO storage client"}
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","serviceContext":{"service":"server","version":"v0.4.0"},"stack_trace":"InvalidEndpointError: Invalid endPoint : <minio ip>:9000\n    at new Client (/app/node_modules/minio/dist/main/minio.js:88:13)\n    at Binding.dynamicValue (/app/node_modules/@typefox/server/dist/container-module.js:213:62)\n    at /app/node_modules/inversify/lib/resolution/resolver.js:63:118\n    at invokeFactory (/app/node_modules/inversify/lib/resolution/resolver.js:10:16)\n    at /app/node_modules/inversify/lib/resolution/resolver.js:63:26\n    at Object.resolve (/app/node_modules/inversify/lib/resolution/resolver.js:96:12)\n    at /app/node_modules/inversify/lib/container/container.js:319:37\n    at Container._get (/app/node_modules/inversify/lib/container/container.js:310:44)\n    at Container.get (/app/node_modules/inversify/lib/container/container.js:230:21)\n    at Binding.dynamicValue (/app/node_modules/@typefox/server/dist/container-module.js:225:34)\n    at /app/node_modules/inversify/lib/resolution/resolver.js:63:118\n    at invokeFactory (/app/node_modules/inversify/lib/resolution/resolver.js:10:16)\n    at /app/node_modules/inversify/lib/resolution/resolver.js:63:26\n    at Array.map (<anonymous>)\n    at _injectProperties (/app/node_modules/inversify/lib/resolution/instantiation.js:11:57)\n    at Object.resolveInstance (/app/node_modules/inversify/lib/resolution/instantiation.js:42:18)\n    at /app/node_modules/inversify/lib/resolution/resolver.js:72:42\n    at Array.map (<anonymous>)\n    at _injectProperties (/app/node_modules/inversify/lib/resolution/instantiation.js:11:57)\n    at Object.resolveInstance (/app/node_modules/inversify/lib/resolution/instantiation.js:42:18)\n    at /app/node_modules/inversify/lib/resolution/resolver.js:72:42\n    at Array.map (<anonymous>)\n    at _injectProperties (/app/node_modules/inversify/lib/resolution/instantiation.js:11:57)\n    at Object.resolveInstance (/app/node_modules/inversify/lib/resolution/instantiation.js:42:18)\n    at /app/node_modules/inversify/lib/resolution/resolver.js:72:42\n    at Object.resolve (/app/node_modules/inversify/lib/resolution/resolver.js:96:12)\n    at /app/node_modules/inversify/lib/container/container.js:319:37\n    at Container._get (/app/node_modules/inversify/lib/container/container.js:310:44)\n    at Container.get (/app/node_modules/inversify/lib/container/container.js:230:21)\n    at /app/node_modules/@typefox/server/dist/main.js:90:36\n    at step (/app/node_modules/@typefox/server/dist/main.js:33:23)\n    at Object.next (/app/node_modules/@typefox/server/dist/main.js:14:53)\n    at /app/node_modules/@typefox/server/dist/main.js:8:71\n    at new WrappedPromise (/app/node_modules/async-listener/es6-wrapped-promise.js:13:18)\n    at __awaiter (/app/node_modules/@typefox/server/dist/main.js:4:12)\n    at init (/app/node_modules/@typefox/server/dist/main.js:57:33)\n    at Object.<anonymous> (/app/node_modules/@typefox/server/dist/main.js:122:1)\n    at Module._compile (internal/modules/cjs/loader.js:701:30)\n    at Object.Module._extensions..js (internal/modules/cjs/loader.js:712:10)\n    at Module.load (internal/modules/cjs/loader.js:600:32)\n    at tryModuleLoad (internal/modules/cjs/loader.js:539:12)\n    at Function.Module._load (internal/modules/cjs/loader.js:531:3)\n    at Function.Module.runMain (internal/modules/cjs/loader.js:754:12)\n    at startup (internal/bootstrap/node.js:283:19)\n    at bootstrapNodeJSCore (internal/bootstrap/node.js:622:3)","component":"server","severity":"ERROR","time":"2020-07-07T07:47:13.892Z","environment":"production","region":"local","message":"Error during startup or operation. Exiting.","error":"InvalidEndpointError: Invalid endPoint : <minio ip>:9000\n    at new Client (/app/node_modules/minio/dist/main/minio.js:88:13)\n    at Binding.dynamicValue (/app/node_modules/@typefox/server/dist/container-module.js:213:62)\n    at /app/node_modules/inversify/lib/resolution/resolver.js:63:118\n    at invokeFactory (/app/node_modules/inversify/lib/resolution/resolver.js:10:16)\n    at /app/node_modules/inversify/lib/resolution/resolver.js:63:26\n    at Object.resolve (/app/node_modules/inversify/lib/resolution/resolver.js:96:12)\n    at /app/node_modules/inversify/lib/container/container.js:319:37\n    at Container._get (/app/node_modules/inversify/lib/container/container.js:310:44)\n    at Container.get (/app/node_modules/inversify/lib/container/container.js:230:21)\n    at Binding.dynamicValue (/app/node_modules/@typefox/server/dist/container-module.js:225:34)\n    at /app/node_modules/inversify/lib/resolution/resolver.js:63:118\n    at invokeFactory (/app/node_modules/inversify/lib/resolution/resolver.js:10:16)\n    at /app/node_modules/inversify/lib/resolution/resolver.js:63:26\n    at Array.map (<anonymous>)\n    at _injectProperties (/app/node_modules/inversify/lib/resolution/instantiation.js:11:57)\n    at Object.resolveInstance (/app/node_modules/inversify/lib/resolution/instantiation.js:42:18)\n    at /app/node_modules/inversify/lib/resolution/resolver.js:72:42\n    at Array.map (<anonymous>)\n    at _injectProperties (/app/node_modules/inversify/lib/resolution/instantiation.js:11:57)\n    at Object.resolveInstance (/app/node_modules/inversify/lib/resolution/instantiation.js:42:18)\n    at /app/node_modules/inversify/lib/resolution/resolver.js:72:42\n    at Array.map (<anonymous>)\n    at _injectProperties (/app/node_modules/inversify/lib/resolution/instantiation.js:11:57)\n    at Object.resolveInstance (/app/node_modules/inversify/lib/resolution/instantiation.js:42:18)\n    at /app/node_modules/inversify/lib/resolution/resolver.js:72:42\n    at Object.resolve (/app/node_modules/inversify/lib/resolution/resolver.js:96:12)\n    at /app/node_modules/inversify/lib/container/container.js:319:37\n    at Container._get (/app/node_modules/inversify/lib/container/container.js:310:44)\n    at Container.get (/app/node_modules/inversify/lib/container/container.js:230:21)\n    at /app/node_modules/@typefox/server/dist/main.js:90:36\n    at step (/app/node_modules/@typefox/server/dist/main.js:33:23)\n    at Object.next (/app/node_modules/@typefox/server/dist/main.js:14:53)\n    at /app/node_modules/@typefox/server/dist/main.js:8:71\n    at new WrappedPromise (/app/node_modules/async-listener/es6-wrapped-promise.js:13:18)\n    at __awaiter (/app/node_modules/@typefox/server/dist/main.js:4:12)\n    at init (/app/node_modules/@typefox/server/dist/main.js:57:33)\n    at Object.<anonymous> (/app/node_modules/@typefox/server/dist/main.js:122:1)\n    at Module._compile (internal/modules/cjs/loader.js:701:30)\n    at Object.Module._extensions..js (internal/modules/cjs/loader.js:712:10)\n    at Module.load (internal/modules/cjs/loader.js:600:32)\n    at tryModuleLoad (internal/modules/cjs/loader.js:539:12)\n    at Function.Module._load (internal/modules/cjs/loader.js:531:3)\n    at Function.Module.runMain (internal/modules/cjs/loader.js:754:12)\n    at startup (internal/bootstrap/node.js:283:19)\n    at bootstrapNodeJSCore (internal/bootstrap/node.js:622:3)"}

So, I uninstalled gitpod and reverted my changes (by deleting the ./charts directory and updating from chart.gitpod.io)
I then removed the minio entry in configuration.txt and created gitpod again (so this time, using the Minio instance provided in the Helm chart)

However, this still didn’t work :slightly_frowning_face: it failed with creating the signed url again (you’ll see 10.97.10.203 - this is the IP of Minio that the Helm chart installed. I also know Minio works fine, because I updated the proxy pod to include a vhost for minio.gitpod.my.domain proxy pass to minio.default.svc.cluster.local:9000 - and I could access Minio in a web browser using the “EXAMPLEvalue” key):

{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","serviceContext":{"service":"server","version":"v0.4.0"},"stack_trace":"Error: connect ETIMEDOUT 10.97.10.203:80\n    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1097:14)","component":"server","severity":"WARNING","time":"2020-07-07T09:18:51.045Z","environment":"production","region":"local","message":"Failed to create a signed URL for plugin with DB id 6776365a-fa62-464d-ad87-6bab998cece4!","error":"Error: connect ETIMEDOUT 10.97.10.203:80\n    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1097:14)","payload":{"bucketName":"gitpod-my--domain-plugins","path":"192a545d-****-****-****-************.plugin.6776365a-fa62-464d-ad87-6bab998cece4.Dart-Code.flutter~3.12.1","state":"uploading"}}    
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","serviceContext":{"service":"server","version":"v0.4.0"},"stack_trace":"Error: connect ETIMEDOUT 10.97.10.203:80\n    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1097:14)","component":"server","severity":"WARNING","time":"2020-07-07T09:18:51.045Z","environment":"production","region":"local","message":"Upload failed (Step: preflight)","error":"Error: connect ETIMEDOUT 10.97.10.203:80\n    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1097:14)","payload":"{ req:\n   <<omitted as I need to get the character count down>>,\n     baseUrl: '/plugins',\n     originalUrl:\n      '/plugins/preflight?type=upload&token=fF7%2Ba<<token>>%3D&id=6776365a-fa62-464d-ad87-6bab998cece4',\n     _parsedUrl:\n      Url {\n        protocol: null,\n        slashes: null,\n        auth: null,\n        host: null,\n        port: null,\n        hostname: null,\n        hash: null,\n        search:\n         '?type=upload&token=fF7%2Ba<<token>>%3D&id=6776365a-fa62-464d-ad87-6bab998cece4',\n        query:\n         'type=upload&token=fF7%2Ba<<token>>%3D&id=6776365a-fa62-464d-ad87-6bab998cece4',\n        pathname: '/preflight',\n        path:\n         '/preflight?type=upload&token=fF7%2Ba<<token>>%3D&id=6776365a-fa62-464d-ad87-6bab998cece4',\n        href:\n         '/preflight?type=upload&token=fF7%2Ba<<token>>%3D&id=6776365a-fa62-464d-ad87-6bab998cece4',\n        _raw:\n         '/preflight?type=upload&token=fF7%2Ba<<token>>%3D&id=6776365a-fa62-464d-ad87-6bab998cece4' },\n     params: {},\n     query:\n      { type: 'upload',\n        token: 'fF7+<<token>>',\n        id: '6776365a-fa62-464d-ad87-6bab998cece4' },\n     res:\n      ServerResponse {\n        domain: null,\n        _events: [Object],\n        _eventsCount: 1,\n        _maxListeners: undefined,\n        output: [],\n        outputEncodings: [],\n        outputCallbacks: [],\n        outputSize: 0,\n        writable: true,\n        _last: false,\n        chunkedEncoding: false,\n        shouldKeepAlive: false,\n        useChunkedEncodingByDefault: false,\n        sendDate: true,\n        _removedConnection: false,\n        _removedContLen: false,\n        _removedTE: false,\n        _contentLength: null,\n        _hasBody: true,\n        _trailer: '',\n        finished: false,\n        _headerSent: false,\n        socket: [Socket],\n        connection: [Socket],\n        _header: null,\n        _onPendingData: [Function: bound updateOutgoingData],\n        _sent100: false,\n        _expect_continue: false,\n        end: [Function: end],\n        req: [Circular],\n        locals: [Object: null prototype] {},\n        writeHead: [Function: writeHead],\n        [Symbol(isCorked)]: false,\n        [Symbol(outHeadersKey)]: [Object] },\n     body: {},\n     _parsedOriginalUrl:\n      Url {\n        protocol: null,\n        slashes: null,\n        auth: null,\n        host: null,\n        port: null,\n        hostname: null,\n        hash: null,\n        search:\n         '?type=upload&token=fF7%2Ba<<token>>%3D&id=6776365a-fa62-464d-ad87-6bab998cece4',\n        query:\n         'type=upload&token=fF7%2Ba<<token>>%3D&id=6776365a-fa62-464d-ad87-6bab998cece4',\n        pathname: '/plugins/preflight',\n        path:\n         '/plugins/preflight?type=upload&token=fF7%2Ba<<token>>%3D&id=6776365a-fa62-464d-ad87-6bab998cece4',\n        href:\n         '/plugins/preflight?type=upload&token=fF7%2Ba<<token>>%3D&id=6776365a-fa62-464d-ad87-6bab998cece4',\n        _raw:\n         '/plugins/preflight?type=upload&token=fF7%2Ba<<token>>%3D&id=6776365a-fa62-464d-ad87-6bab998cece4' },\n     sessionStore:\n      MySQLStore {\n        connection: [Pool],\n        options: [Object],\n        generate: [Function],\n        _events: [Object],\n        _eventsCount: 2,\n        _expirationInterval:\n         Timeout {\n           _called: true,\n           _idleTimeout: 900000,\n           _idlePrev: [TimersList],\n           _idleNext: [TimersList],\n           _idleStart: 2701975,\n           _onTimeout: [Function],\n           _timerArgs: undefined,\n           _repeat: 900000,\n           _destroyed: false,\n           [Symbol(unrefed)]: false,\n           [Symbol(asyncId)]: 177,\n           [Symbol(triggerId)]: 125 } },\n     sessionID: '39e81793-0610-4945-a5ef-dfff1803c172',\n     session: Session { cookie: [Object] },\n     _passport: { instance: [Authenticator] },\n     route:\n      Route { path: '/preflight', stack: [Array], methods: [Object] } } }"}

So I’m not really sure where to go with this now! I guess it’s something to do with the DB creating the signed url… but the database should work fine… (it is external)

Cheers!

Bump!