[microk8s][gitlab-runner][helm] slow connection / connection failed on apt-get update in docker build

686 views Asked by At

Chart.yaml

  - name: gitlab-runner
    version: 0.21.1
    repository: https://charts.gitlab.io/

values.yaml

gitlab-runner:
  gitlabUrl: 'xxx'
  runnerRegistrationToken: 'xxx'
  runners:
    privileged: true
    env:
      DOCKER_HOST: tcp://localhost:2375
      DOCKER_DRIVER: overlay2
      DOCKER_TLS_CERTDIR: ""
      NO_PROXY: "docker,localhost,127.0.0.1,.local" <!-- tried with and without this line
  unregisterRunners: true
  metrics:
    enabled: false
  rbac:
    create: true

Microk8s:

microk8s is running
high-availability: no
  datastore master nodes: 127.0.0.1:19001
  datastore standby nodes: none
addons:
  enabled:
    dns                  # CoreDNS
    ha-cluster           # Configure high availability on the current node
    helm3                # Helm 3 - Kubernetes package manager
    rbac                 # Role-Based Access Control for authorisation
    storage              # Storage class; allocates storage from host directory

Kubernetes:

Client Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.2-34+1b3fa60b402c1c", GitCommit:"1b3fa60b402c1c4cb0df8a99b733ad41141a2eb7", GitTreeState:"clean", BuildDate:"2020-09-16T21:34:17Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.2-34+1b3fa60b402c1c", GitCommit:"1b3fa60b402c1c4cb0df8a99b733ad41141a2eb7", GitTreeState:"clean", BuildDate:"2020-09-16T21:36:35Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

First few job lines

Running with gitlab-runner 13.4.1 (e95f89a0)
  on local-gitlab-runner-859c8c448f-x99fc SKURUyFs
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: default
Using Kubernetes executor with image docker:19.03 ...
Preparing environment
Waiting for pod default/runner-skuruyfs-project-12604088-concurrent-0dz9cn to be running, status is Pending
Running on runner-skuruyfs-project-12604088-concurrent-0dz9cn via local-gitlab-runner-859c8c448f-x99fc...
Getting source from Git repository

default images gitlab-ci.yaml

image: docker:19.03
services:
  - docker:19.03-dind

Problem:

I've installed this chart on my local microk8s cluster. I'm using docker dind to build images from my code. I'm currently building two images, which are based on FROM php:7.4.11-fpm and FROM node:lts-alpine. (Using older images does not fix my issue). When I run apk add for the node image, the process is stuck here indefinitely (until a timeout occurs):

Step 3/27 : RUN apk add --no-cache git     && apk add --no-cache --virtual .build-deps-yarn curl     && curl -fSLO --compressed "https://yarnpkg.com/downloads/$YARN_VERSION/yarn-v$YARN_VERSION.tar.gz"     && tar -xzf yarn-v$YARN_VERSION.tar.gz -C /opt/     && ln -snf /opt/yarn-v$YARN_VERSION/bin/yarn /usr/local/bin/yarn     && ln -snf /opt/yarn-v$YARN_VERSION/bin/yarnpkg /usr/local/bin/yarnpkg     && rm yarn-v$YARN_VERSION.tar.gz     && apk del .build-deps-yarn
 ---> Running in 45465897f01c
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
(1/6) Installing ca-certificates (20191127-r2)
(2/6) Installing nghttp2-libs (1.40.0-r1)
(3/6) Installing libcurl (7.67.0-r1)
(4/6) Installing expat (2.2.9-r1)
(5/6) Installing pcre2 (10.34-r1)
(6/6) Installing git (2.24.3-r0)
Executing busybox-1.31.1-r9.trigger
Executing ca-certificates-20191127-r2.trigger
OK: 23 MiB in 22 packages
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz

With the php image, I get Connection failed errors. The moment these errors occur is different every time.

Err:1 http://deb.debian.org/debian buster/main amd64 python3-lib2to3 all 3.7.3-1
  Connection failed [IP: 151.101.36.204 80]

Or:

+ apt-get update
Get:1 http://apt.postgresql.org/pub/repos/apt buster-pgdg InRelease [103 kB]
Get:2 http://apt.postgresql.org/pub/repos/apt buster-pgdg/main amd64 Packages [203 kB]
Err:3 http://security.debian.org/debian-security buster/updates InRelease
  Connection failed [IP: 151.101.36.204 80]
Err:4 http://deb.debian.org/debian buster InRelease
  Connection failed [IP: 151.101.36.204 80]
Err:5 http://deb.debian.org/debian buster-updates InRelease
  Connection failed [IP: 151.101.36.204 80]
Fetched 307 kB in 2min 0s (2550 B/s)
Reading package lists...
W: Failed to fetch http://deb.debian.org/debian/dists/buster/InRelease  Connection failed [IP: 151.101.36.204 80]
W: Failed to fetch http://security.debian.org/debian-security/dists/buster/updates/InRelease  Connection failed [IP: 151.101.36.204 80]
W: Failed to fetch http://deb.debian.org/debian/dists/buster-updates/InRelease  Connection failed [IP: 151.101.36.204 80]
W: Some index files failed to download. They have been ignored, or old ones used instead.

Googling for these kind of issues, I get results which are always related to the NO_PROXY, HTTP_PROXY and HTTPS_PROXY settings. I'm not running behind a proxy, so I tried fixing this with the NO_PROXY env variable but this unfortunately didnt help. Building these images on Win10 2004 + Docker Desktop 2.4.0.0 (on the same network) is working without any issue.

Am I using the correct values for the NO_PROXY env var? Should I even be using it / is it related? The result without the NO_PROXY env var is the same. Is there maybe something else im missing?

Before I went to microk8s, I was running the Docker executor on Portainer; then this issue did not occur.

0

There are 0 answers