Service Pod fails behind proxy. Apk can't add files off image-builder

Is there a way to configure the image-builder script within gcr.io/gitpod-io/image-builder:v0.4.0 to use a proxy when building the service image?

Because I keep getting errors when apk tries to download files. Also, I tried modifying the image-builder script within the image-builder image within the service container, but when I tried to run it, I kept getting a Segmentation fault error.

I tried setting proxy environment variables within the image-builder deployment containers, but that didn’t fix it.

Any ideas how to fix this issue?

INFO[0000] enabled verbose logging                       serviceContext="{image-builder }"
INFO[0000] starting Docker ref pre-cache                 interval=6h0m0s refs="[gitpod/workspace-full:latest]" serviceContext="{image-builder }"
INFO[0000] computed Gitpod layer hash                    gitpodLayer=/app/workspace-image-layer.tar.gz hash=db4a9cc9a727cdf8521ff2ce71e7c2eee05fd01b012aa0e8aadbaaa652f0e319 serviceContext="{image-builder }"
INFO[0000] running self-build                            gitpodLayer=/app/workspace-image-layer.tar.gz serviceContext="{image-builder }"
DEBU[0000] self-build context sent                       serviceContext="{image-builder }"
DEBU[0000] Step 1/7 : FROM alpine:3.9
DEBU[0002] 3.9: Pulling from library/alpine
DEBU[0003] Digest: sha256:414e0518bb9228d35e4cd5165567fb91d26c6a214e9c95899e1e056fcd349011
DEBU[0003] Status: Image is up to date for alpine:3.9
DEBU[0003]  ---> 78a2ce922f86
DEBU[0003] Step 2/7 : RUN addgroup -g 33333 gitpod     && adduser -D -h /home/gitpod -s /bin/sh -u 33333 -G gitpod gitpod     && echo "gitpod:gitpod" | chpasswd
DEBU[0003]  ---> Using cache
DEBU[0003]  ---> b5248ebcbc40
DEBU[0003] Step 3/7 : RUN apk add --no-cache git bash openssh-client lz4 coreutils
DEBU[0003]  ---> Running in c1f30946f18e
DEBU[0003] fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
DEBU[0017] pre-cached Docker ref                         ref="gitpod/workspace-full:latest" resolved-to="docker.io/gitpod/workspace-full:latest@sha256:29afadcbe7d1d4f87c393432616cc374de69e8c1fa3559cf7126e41b9de3e4d2" serviceContext="{image-builder }"
DEBU[0131] WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz: network error (check Internet connection and firewall)
DEBU[0131] fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz
DEBU[0258] WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz: network error (check Internet connection and firewall)
DEBU[0258] ERROR: unsatisfiable constraints:
DEBU[0258]   bash (missing):
DEBU[0258]     required by: world[bash]
DEBU[0258]   coreutils (missing):
DEBU[0258]     required by: world[coreutils]
DEBU[0258]   git (missing):
DEBU[0258]     required by: world[git]
DEBU[0258]   lz4 (missing):
DEBU[0258]     required by: world[lz4]
DEBU[0258]   openssh-client (missing):
DEBU[0258]     required by: world[openssh-client]
ERRO[0259] self-build failed                             error="The command '/bin/sh -c apk add --no-cache git bash openssh-client lz4 coreutils' returned a non-zero code: 5" serviceContext="{image-builder }"
FATA[0259] The command '/bin/sh -c apk add --no-cache git bash openssh-client lz4 coreutils' returned a non-zero code: 5  serviceContext="{image-builder }"
3 Likes

Are these the ENV variables you did set? https://docs.docker.com/network/proxy/
Did you rebuild the image using these ENVs so they are there when the dockerd starts?

Can you share a bit of background what kind of network you’re in and which domains can be reached without a proxy and which domains require you to use a proxy? This would help to understand how important it is for Gitpod to have better proxy support.

Also, do developers on your network need to configure a proxy for every tool they use?

2 Likes

I don’t think it’s an issue with the docker daemon reaching out to the web, rather apk from within the docker-in-docker container that cannot reach out to the web. The image is created in the image-builder pod so there seems to be no way to add proxies in.

1 Like

Current Issue:

I think there needs to be an option from either the image-builder.json or image-builder script to allow an option for http_proxy, https_proxy, and no_proxy to be added, so when the new image is being built (within the service container with the image-builder script), alpine can use those ENV proxy variables to pull in files for this specific Dockerfile step:

DEBU[0003] Step 3/7 : RUN apk add --no-cache git bash openssh-client lz4 coreutils ### INSERT proxy option here or image-builder adds another step before this that configures ENV http_proxy=http://internal-proxy-XXXXX:80 , ENV https_proxy=http://internal-proxy-XXXXX:80 , ENV no_proxy=http://internal-proxy-XXXXX:80

Current Logs/Configuration

Service pod shell

apk update works on the service container, and dind container within the image-builder pod.

bash-5.0# apk update
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
v3.11.6-215-g0999d9b171 [http://dl-cdn.alpinelinux.org/alpine/v3.11/main]
v3.11.6-212-g3a84324cad [http://dl-cdn.alpinelinux.org/alpine/v3.11/community]
OK: 11275 distinct packages available
bash-5.0#

I mounted the proxy values for the alpine containers (dind and service) so they could use the internal proxy.

bash-5.0# cat /etc/profile.d/env.sh
export http_proxy=http://internal-proxy-XXXXX:80
export https_proxy=http://internal-proxy-XXXXX:80 
bash-5.0#

bash-5.0# cat ~/.docker/config.json
{
        "proxies":
        {
                "default":
                {
                        "httpProxy": "http://internal-proxy-XXXXX:80",
                        "httpsProxy": "http://internal-proxy-XXXXX:80",
                        "noProxy": "localhost,127.0.0.1"
                }
        }
}
bash-5.0#

COULD A PROXY OPTION BE ADDED HERE???

bash-5.0# ./image-builder --help
Workspace image-builder service

Usage:
  image-builder [command]

Available Commands:
  bob         Bob the builder are a set of utility functions executed during an image build.
  generate    Generate Typescript/JSON schema for parts of this application
  help        Help about any command
  run         Starts the image-builder service

Flags:
      --config string   config file
  -h, --help            help for image-builder
  -v, --verbose         Enable verbose JSON logging

Use "image-builder [command] --help" for more information about a command.
bash-5.0#

NEED A WAY TO INSERT PROXY HERE AFTER THE FROM alpine:3.9, since I can’t modify this image-builder script file.

MAYBE THE IMAGE-BUILDER Script can take in an argument for http_proxy, https_proxy, and no_proxy???

bash-5.0# ls
image-builder                 workspace-image-layer.tar.gz
bash-5.0# vi image-builder

NoMatchEmptyMatchLiteralCharClassAnyCharNotNLAnyCharBeginLineEndLineBeginTextEndTextWordBoundaryNoWordBoundaryCaptureStarPlusQuestRepeatConcatA
                the client. Often this will be accompanied by a
                Www-Authenticate HTTP response header indicating how to
                authenticate.image %s could not be accessed on a registry to record
its digest. Each node will access %s independently,
possibly leading to different nodes running different
versions of the image.
A trace of execution of the current program. You can specify the duration in the seconds GET parameter. After you get the trace file, use the g
FROM alpine:3.9

\# Add gitpod user for operations (e.g. checkout because of the post-checkout hook!)
RUN addgroup -g 33333 gitpod \
    && adduser -D -h /home/gitpod -s /bin/sh -u 33333 -G gitpod gitpod \
    && echo "gitpod:gitpod" | chpasswd

RUN apk add --no-cache git bash openssh-client lz4 coreutils

COPY bob /bob
COPY gitpodLayer.tar.gz /gitpodLayer.tar.gz
RUN mkdir /gitpod-layer && cd /gitpod-layer && tar xzfv /gitpodLayer.tar.gz
<html>

ISSUE building the image within dind, because there’s no proxy option to add in to the image-build (that I know of)

bash-5.0# ./image-builder run -v --config /config/image-builder.json
    INFO[0000] enabled verbose logging                       serviceContext="{image-builder }"
    INFO[0000] starting Docker ref pre-cache                 interval=6h0m0s refs="[gitpod/workspace-full:latest]" serviceContext="{image-builder }"
    INFO[0000] computed Gitpod layer hash                    gitpodLayer=/app/workspace-image-layer.tar.gz hash=db4a9cc9a727cdf8521ff2ce71e7c2eee05fd01b012aa0e8aadbaaa652f0e319 serviceContext="{image-builder }"
    INFO[0000] running self-build                            gitpodLayer=/app/workspace-image-layer.tar.gz serviceContext="{image-builder }"
    DEBU[0000] self-build context sent                       serviceContext="{image-builder }"
    DEBU[0000] Step 1/7 : FROM alpine:3.9
    DEBU[0003] 3.9: Pulling from library/alpine
    DEBU[0003] Digest: sha256:414e0518bb9228d35e4cd5165567fb91d26c6a214e9c95899e1e056fcd349011
    DEBU[0003] Status: Image is up to date for alpine:3.9
    DEBU[0003]  ---> 78a2ce922f86
    DEBU[0003] Step 2/7 : RUN addgroup -g 33333 gitpod     && adduser -D -h /home/gitpod -s /bin/sh -u 33333 -G gitpod gitpod     && echo "gitpod:gitpod" | chpasswd
    DEBU[0003]  ---> Using cache
    DEBU[0003]  ---> b5248ebcbc40

DEBU[0003] Step 3/7 : RUN apk add --no-cache git bash openssh-client lz4 coreutils

DEBU[0004]  ---> Running in 6c002c769f5f
DEBU[0004] fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
DEBU[0004] pre-cached Docker ref                         ref="gitpod/workspace-full:latest" resolved-to="docker.io/gitpod/workspace-full:latest@sha256:29afadcbe7d1d4f87c393432616cc374de69e8c1fa3559cf7126e41b9de3e4d2" serviceContext="{image-builder }"
DEBU[0132] WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz: network error (check Internet connection and firewall)
DEBU[0132] fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz

DEBU[0259] WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz: network error (check Internet connection and firewall)`

DEBU[0259] ERROR: unsatisfiable constraints:
DEBU[0259]   bash (missing):
DEBU[0259]     required by: world[bash]
DEBU[0259]   coreutils (missing):
DEBU[0259]     required by: world[coreutils]
DEBU[0259]   git (missing):
DEBU[0259]     required by: world[git]
DEBU[0259]   lz4 (missing):
DEBU[0259]     required by: world[lz4]
DEBU[0259]   openssh-client (missing):
DEBU[0259]     required by: world[openssh-client]
ERRO[0260] self-build failed                             error="The command '/bin/sh -c apk add --no-cache git bash openssh-client lz4 coreutils' returned a non-zero code: 5" serviceContext="{image-builder }"
FATA[0260] The command '/bin/sh -c apk add --no-cache git bash openssh-client lz4 coreutils' returned a non-zero code: 5  serviceContext="{image-builder }"
bash-5.0#
3 Likes

This is the exact same issue that I am having as well!

1 Like

Good Lord! I thought I was the only one experiencing this :flushed:

1 Like

@Jwoozy Good find!

1 Like

This is indeed a missing feature in the image-builder. I’ve created an issue for this: https://github.com/gitpod-io/gitpod/issues/2467

1 Like

Same here.