Skaffold dev works with minikube only. Other on-prem cluster fails

Question

I have a Spring Boot app with jib-maven configured

POM

<plugin>
<groupId>com.google.cloud.tools</groupId>
<artifactId>jib-maven-plugin</artifactId>
<version>2.1.0</version>
<configuration>
    <from>
        <image>openjdk:11-jre-slim</image>
    </from>
    <to>
        <image>registry.demo/${project.artifactId}</image>
        <tags>
            <tag>${project.version}</tag>
        </tags>
        <tags>
            <tag>latest</tag>
        </tags>
    </to>
    <container>
        <jvmFlags>
            <jvmFlag>-XX:+UseContainerSupport</jvmFlag>
            <jvmFlag>-XX:MinRAMPercentage=60.0</jvmFlag>
            <jvmFlag>-XX:MaxRAMPercentage=90.0</jvmFlag>
            <jvmFlag> -XshowSettings:vm</jvmFlag>
        </jvmFlags>
        <mainClass>com.demo.DemoApplication</mainClass>
    </container>
</configuration>

SKAFFOLD.YAML

        apiVersion: skaffold/v2beta1
        kind: Config
        metadata:
        name: springtokube
        build:
        artifacts:
        - image: registry.demo/springtokube
            jib:
            project: com.demo:springtokube
        local:
            push: true
            concurrency: 1
            useBuildkit: false
            useDockerCLI: true
        deploy:
        kubectl:
            manifests:
            - deployment.yaml

ALSO SET INSECURE REGISTRY

skaffold config set --global insecure-registries  registry.demo

But when using minikube I can run successfully

skaffold dev

When using other cluster (ON-PREM) I get

FATA[0016] exiting dev mode because first build failed: build failed: building [registry.demo/springtokube]: build artifact: getting image: GET http://registry.demo/v2/: : Not Found

What might be the problem?

I restarted today using kubectl context

skaffold debug --no-prune=false --cache-artifacts=false

And It Failed

        Listing files to watch...
        Generating tags...
        - registry.demo/springtokube -> registry.demo/springtokube:cf60c31
        Found [minikube] context, using local docker daemon.
        Building [registry.demo/springtokube]...
        .............
        ...............

        [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.294 s - in com.demo.springtokube.SpringtokubeApplicationTests
        2020-04-15 08:45:48.277  INFO 30662 --- [extShutdownHook] o.s.s.concurrent.ThreadPoolTaskExecutor  : Shutting down ExecutorService 'applicationTaskExecutor'
        [INFO] 
        [INFO] Results:
        [INFO] 
        [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
        [INFO] 
        [INFO] 
        [INFO] --- maven-jar-plugin:3.1.2:jar (default-jar) @ springtokube ---
        [INFO] Building jar: ....../springtokube/target/springtokube.jar
        [INFO] 
        [INFO] --- spring-boot-maven-plugin:2.2.6.RELEASE:repackage (repackage) @ springtokube ---
        [INFO] Replacing main artifact with repackaged archive
        [INFO] 
        [INFO] --- jib-maven-plugin:2.1.0:build (default-cli) @ springtokube ---
        [INFO] 
        [INFO] Containerizing application to registry.demo/springtokube:cf60c31, registry.demo/springtokube...
        [WARNING] Base image 'openjdk:11-jre-slim' does not use a specific image digest - build may not be reproducible
        [INFO] Getting manifest for base image openjdk:11-jre-slim...
        [INFO] Building dependencies layer...
        [INFO] Building resources layer...
        [INFO] Building classes layer...
        [INFO] Using credentials from Docker config (~/.docker/config.json) for registry.demo/springtokube:cf60c31
        [WARNING] Cannot verify server at https://registry.demo/v2/. Attempting again with no TLS verification.
        [WARNING] Cannot verify server at https://registry.demo/v2/springtokube/blobs/sha256:1fb3fb86aa52691fa3705554da5ba07dcb556f62a93ba7efab0e397ca3db092c. Attempting again with no TLS verification.
        [WARNING] Cannot verify server at https://registry.demo/v2/springtokube/blobs/sha256:88a7d9887f9fdeb5a4736d07c64818453e00e71fe916b13f413eb6e545445a68. Attempting again with no TLS verification.
        [WARNING] Cannot verify server at https://registry.demo/v2/springtokube/blobs/sha256:a6c851c4b90b9eb7af89d240dd4f438dba9feba5c78600fed7eadddf8cb7b647. Attempting again with no TLS verification.
        [INFO] The base image requires auth. Trying again for openjdk:11-jre-slim...
        [INFO] Using credentials from Docker config (~/.docker/config.json) for openjdk:11-jre-slim
        [INFO] Using base image with digest: sha256:01669f539159a1b5dd69c4782be9cc7da0ac1f4ddc5e2c2d871ef1481efd693e
        [INFO] 
        [INFO] Container entrypoint set to [java, -XX:+UseContainerSupport, -XX:MinRAMPercentage=60.0, -XX:MaxRAMPercentage=90.0, -XshowSettings:vm, -cp, /app/resources:/app/classes:/app/libs/*, com.demo.springtokube.SpringtokubeApplication]
        [INFO] 
        [INFO] Built and pushed image as registry.demo/springtokube:cf60c31, registry.demo/springtokube
        [INFO] 
        [INFO] ------------------------------------------------------------------------
        [INFO] BUILD SUCCESS
        [INFO] ------------------------------------------------------------------------
        [INFO] Total time:  20.058 s
        [INFO] Finished at: 2020-04-15T08:45:57+03:00
        [INFO] ------------------------------------------------------------------------
        Pruning images...
        FATA[0024] exiting dev mode because first build failed: build failed: building [registry.demo/springtokube]: build artifact: getting image: GET http://registry.demo/v2/: : Not Found 

I thought the minikube works. But disabling cache fails to build

if I run

 skaffold debug OR  skaffold dev

Works Fine

But if I run with cache disabled

skaffold debug --no-prune=false --cache-artifacts=false

FAILS it shows the logs above

Answer 1

After days of struggling I found a solution.

Following Brian de Alwis suggestions I was able to make Skaffold work with Self Signed Certificate.

Skaffold build or dev does not use certificate put in.

/etc/docker/certs.d/myregistrydomain.com/ca.crt

The path is used by docker client only.

The solution was to put yout registry certificate into

 /usr/local/share/ca-certificates/myregistrydomain.com.crt

Then

update-ca-certificates

Check The link for more info

If you are using self signed certificate no need for insecure registry in your scaffold yaml file

apiVersion: skaffold/v2beta1
kind: Config
metadata:
  name: springtokube
build:
#  insecureRegistries:
#    - myregistrydomain.com

Or Running skaffold with

skaffold dev --insecure-registry=myregistrydomain.com

Hope this help someone else struggling to make skaffold works with self signed certificate