Using reflections to access methods in Amazon Deequ

Question

I plan on creating a user config file that I will later parse in order to run some checks from Amazon Deequ. I want to be able to pass the string names from the config file to get the methods; however, in my attempts to do so, I keep hitting roadblocks.

import com.amazon.deequ.checks._

val ru = scala.reflect.runtime.universe
val rm = ru.runtimeMirror(getClass.getClassLoader)

val myClass = new Check(CheckLevel.Error, "unit testing my data")
val im = rm.reflect(myClass)

val methodSymbolNonNeg = ru.typeOf[Check].decl(ru.TermName("isNonNegative")).asMethod
val methodNonNeg = im.reflectMethod(methodSymbolNonNeg)

Name: Compile Error
Message: <console>:48: error: type mismatch;
 found   : ru.MethodSymbol
 required: ru.MethodSymbol
       val methodNonNeg = im.reflectMethod(methodSymbolNonNeg)

I have tried a couple of examples from the documentation as well what I have found on here on SO. I have also checked the method names in the class myClass.getClass.getMethods.map(_.getName).foreach{println} as well as on Github.

Additionally, I have tried using .getClass.getMethod and invoking the method as well

val m = myClass.getClass.getMethod("isNonNegative", "".getClass, 2.13.getClass, Option("").getClass)

Name: java.lang.NoSuchMethodException
Message: com.amazon.deequ.checks.Check.isNonNegative(java.lang.String, double, scala.Some)
StackTrace:   at java.lang.Class.getMethod(Class.java:1786)

Answer 1

Can't reproduce the compile error in your Scala reflection code. With Scala 2.13.3 + Deequ 1.0.5 and Scala 2.11.12 + Deequ 1.0.5 your code compiles. Write your versions of dependencies and Scala. Try clean rebuild of your project (e.g. sbt clean compile if you build your project with sbt).

Deequ 1.0.5 depends on versions of Scala libraries (Shapeless, Twitter Chill, Spark, json4s, scala-parser-combinators, scala-xml, Breeze, Spire, Machinist) for Scala 2.11.x so you should use Scala 2.11.x.

With Scala 2.13 I have java.lang.NoClassDefFoundError: scala/Serializable.

Also notice that at https://github.com/awslabs/deequ it's written

Deequ depends on Java 8 and is known to work with Apache Spark versions 2.2.x to 2.4.x.

Your compile error found: ru.MethodSymbol, required: ru.MethodSymbol looks like some issue with versions of dependencies.

Regarding your Java reflection code you just specified incorrectly classes of method parameters. The signature of Check#isNonNegative is

def isNonNegative(
  column: String,
  assertion: Double => Boolean,
  hint: Option[String])
: CheckWithLastConstraintFilterable

so you should do

myClass.getClass.getMethod("isNonNegative", classOf[String], classOf[Double => Boolean], classOf[Option[_]])

By the way, mixing Scala reflection and Java reflection is weird.