Shapeless error while trying trying to use PureConfig and Circe for scala

Question

shapeless.DefaultSymbolicLabelling  
shapeless.DefaultSymbolicLabelling$.instance(shapeless.HList)

getting this error while using both pureconfig and circe. I'm using spark 3.1.2 with spark k8s operator.

Answer 1

This error is because of conflicting shapeless library versions. Spark 3.1.2 ships with shapeless 2.3.3 whereas both these packages need shapeless 2.3.7. To solve this I followed the steps mentioned here which involve shading ie renaming the dependency.

For SBT

If you are using the sbt-assembly plugin to create your JARs you can shade shapeless by adding to your assembly.sbt file the following setting:

assembly / assemblyShadeRules := Seq(ShadeRule.rename("shapeless.**" -> "new_shapeless.@1").inAll)

Maven

The maven-shade-plugin can shade shapeless by adding to your pom.xml file the following block:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>3.0.0</version>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <createDependencyReducedPom>false</createDependencyReducedPom>
        <relocations>
            <relocation>
                <pattern>shapeless</pattern>
                <shadedPattern>shapelesspureconfig</shadedPattern>
            </relocation>
        </relocations>
    </configuration>
</plugin>