spark resolve external packages behind corporate artifactory

808 views Asked by At

I want to resolve packages in spark when behind a corporate Artifactory:

spark-shell --repositories https://bin.mycorp.com/artifactory/maven/ --packages io.delta:delta-core_2.12:0.7.0

simply doesn't resolve for me. In the logs I find:

found io.delta#delta-core_2.12;0.7.0 in repo-1
found org.antlr#antlr4;4.7 in repo-1 https://bin.mycorp.com/artifactory/maven/ --packages io.delta:delta-core_2.12:0.7.0                                                                      [https://found org.antlr#antlr4-runtime;4.7 in repo-1a remote repository with the name: repo-1
Ivy Default Cache set to: ~/.ivy2/cache
The jars for the packages stored in: ~/.ivy2/jars
:: loading settings :: url = jar:file:/usr/hdp/3.1.x.x-xxx/spark2/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
io.delta#delta-core_2.12 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-ae4e5fbb-afc7-4c44-8262-b5c40ee2a1d5;1.0
        confs: [default]

no output.

The ivy settings from the JAR (/usr/hdp/3.1.4.0-315/spark2/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml) are:

<ivysettings>
        <settings defaultResolver="default"/>
        <include url="${ivy.default.settings.dir}/ivysettings-public.xml"/>
        <include url="${ivy.default.settings.dir}/ivysettings-shared.xml"/>
        <include url="${ivy.default.settings.dir}/ivysettings-local.xml"/>
        <include url="${ivy.default.settings.dir}/ivysettings-main-chain.xml"/>
        <include url="${ivy.default.settings.dir}/ivysettings-default-chain.xml"/>
</ivysettings>
0

There are 0 answers