How does the library "Symja" need to be imported with Gradle in order to work with the Elasticsearch server module?

Question

For a project I wanted to extend Elasticsearch and therefore need to use the package Symja. In the Github for Symja, there is a manual for the usage with Maven provided.

Since the Elasticsearch repository is build with Gradle, I also need to use Gradle instead of Maven. Testing the suggested example Symja project, the following build.gradle (which I basically generated by using gradle init and adjusted a little) imports the library flawlessly:

apply plugin: 'java'
apply plugin: 'maven-publish'

repositories {
    mavenLocal()
    maven {
        url = uri('https://oss.sonatype.org/content/repositories/snapshots')
    }
    maven {
        url = uri('https://repo.maven.apache.org/maven2/')
    }
}

dependencies {
    implementation 'org.matheclipse:matheclipse-core:1.0.0-SNAPSHOT'
}

publishing {
    publications {
        maven(MavenPublication) {
            from(components.java)
        }
    }
}

Now, I wanted to add Symja to my Elasticsearch fork into the server module.

When I try to build this, though, the imports

import org.matheclipse.parser.client.SyntaxError;
import org.matheclipse.parser.client.math.MathException;

of the Symja example aren't detected: java: package org.matheclipse.parser.client does not exist. What may have gone wrong?

My updated build.gradle of the server module looks like this:

...
apply plugin: 'java'
apply plugin: 'maven-publish'

publishing {
    publications {
        ...
        maven(MavenPublication) {
            from(components.java)
        }
    }
}

repositories {
    mavenLocal()
    maven {
        url = uri('https://oss.sonatype.org/content/repositories/snapshots')
    }
    maven {
        url = uri('https://repo.maven.apache.org/maven2/')
    }
}

...

dependencies {
    ...
    // Symja
    implementation "org.matheclipse:matheclipse-core:1.0.0-SNAPSHOT"
    ...
}

...

Edit: After following @axelclk's suggestion to add implementation "org.matheclipse:matheclipse-frontend:1.0.0-SNAPSHOT", the packages are now detected by my IDE. Although, if I build the build.gradle (while containing the Symja example whithin a method in org.elasticsearch.index.query.BoolQueryBuilder.java), following error appears:

C:\{path here...}\elasticsearch\server\src\main\java\org\elasticsearch\index\query\BoolQueryBuilder.java:454: error: cannot access GcdRingElem
            System.out.println("Out[2]: " + result.toString());
                                                  ^
  class file for edu.jas.structure.GcdRingElem not found
C:\{path here...}\elasticsearch\server\src\main\java\org\elasticsearch\index\query\BoolQueryBuilder.java:459: error: cannot access FieldElement
            System.out.println("Out[3]: " + result.toString());
                                                  ^
  class file for org.hipparchus.FieldElement not found

It seems like somehow, the toString method doesn't work properly. Also, if I run my Elasticsearch fork with gradlew run -Dtests.es.xpack.security.enabled=false -Dtests.heap.size=1G, the build fails with following output:

> Configure project :modules:reindex
Disabling reindex-from-old tests because we can't get the pid file on windows

> Configure project :plugins:repository-hdfs
hdfsFixture unsupported, please set HADOOP_HOME and put HADOOP_HOME\bin in PATH

> Configure project :x-pack:plugin:searchable-snapshots:qa:hdfs
hdfsFixture unsupported, please set HADOOP_HOME and put HADOOP_HOME\bin in PATH
=======================================
Elasticsearch Build Hamster says Hello!
  Gradle Version        : 6.8
  OS Info               : Windows 10 10.0 (amd64)
  JDK Version           : 15 (Oracle)
  JAVA_HOME             : C:\Program Files\Java\jdk-15.0.1
  Random Testing Seed   : C5A79F8CABA0D6FA
  In FIPS 140 mode      : false
=======================================

> Task :server:compileJava
C:\{path here ...}\elasticsearch\server\src\main\java\org\elasticsearch\index\query\BoolQueryBuilder.java:454: error: cannot access GcdRingElem
            System.out.println("Out[2]: " + result.toString());
                                                  ^
  class file for edu.jas.structure.GcdRingElem not found
C:\{path here ...}\elasticsearch\server\src\main\java\org\elasticsearch\index\query\BoolQueryBuilder.java:459: error: cannot access FieldElement
            System.out.println("Out[3]: " + result.toString());
                                                  ^
  class file for org.hipparchus.FieldElement not found
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
2 errors

> Task :server:compileJava FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':server:compileJava'.
> Compilation failed; see the compiler error output for details.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 58s
228 actionable tasks: 8 executed, 220 up-to-date

Edit 2: I added implementation "org.matheclipse:matheclipse-external:1.0.0-SNAPSHOT" and replaced mavenLocal() with mavenCentral() as suggested by @axelclk which doesn't lead to any different results.

Edit 3: I added -- again, based on suggestions from @axelclk -- the implementations described by the pom.xml files of matheclipse-external and matheclipse-core:

...
  // Symja
  implementation "org.matheclipse:matheclipse-external:1.0.0-SNAPSHOT"
  //implementation "org.apache.logging.log4j:log4j-core:2.13.2"
  //implementation "org.apache.logging.log4j:log4j-api:2.13.2"
  testImplementation "org.slf4j:slf4j-simple:1.7.25"
  implementation "org.slf4j:log4j-over-slf4j:1.7.25"
  //implementation "com.google.guava:failureaccess:1.0.1"
  //implementation "com.google.guava:guava:30.1.1-jre"
  implementation "com.fasterxml.jackson.core:jackson-databind:2.11.0"
  implementation "com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.11.0"
  implementation "io.pebbletemplates:pebble:3.1.5"
  implementation "org.jheaps:jheaps:0.13"
  implementation "org.antlr:antlr4-runtime:4.8-1"
  implementation "org.apache.commons:commons-text:1.8"
  //testImplementation "junit:junit:4.13.1"

  implementation "org.matheclipse:matheclipse-frontend:1.0.0-SNAPSHOT"

  implementation "org.matheclipse:matheclipse-core:1.0.0-SNAPSHOT"
  implementation "org.commonmark:commonmark:0.17.1"
  implementation "org.commonmark:commonmark-ext-gfm-tables:0.17.1"
  implementation "org.apfloat:apfloat:1.10.0-SNAPSHOT"
  //implementation "org.apache.logging.log4j:log4j-core:2.13.2"
  //implementation "org.apache.logging.log4j:log4j-api:2.13.2"
  //testImplementation "org.slf4j:slf4j-simple:1.7.25"
  //implementation "org.slf4j:log4j-over-slf4j:1.7.25"
  implementation "org.hipparchus:hipparchus-core:1.8"
  implementation "org.hipparchus:hipparchus-clustering:1.8"
  implementation "org.hipparchus:hipparchus-fft:1.8"
  implementation "org.hipparchus:hipparchus-fitting:1.8"
  implementation "org.hipparchus:hipparchus-ode:1.8"
  implementation "org.hipparchus:hipparchus-optim:1.8"
  implementation "org.hipparchus:hipparchus-stat:1.8"
  //implementation "commons-codec:commons-codec:1.14"
  implementation "org.apache.commons:commons-lang3:3.11"
  implementation "com.google.code.findbugs:jsr305:3.0.0"
  implementation "org.logicng:logicng:2.0.2"
  implementation "org.gavaghan:geodesy:1.1.3"
  implementation "it.unimi.dsi:fastutil:8.5.2"
  implementation "org.codehaus.janino:janino:3.1.3"
  implementation "com.ibm.icu:icu4j:68.2"
  implementation "com.univocity:univocity-parsers:2.8.4"
  testImplementation "org.junit.jupiter:junit-jupiter-api:5.4.2"
  testImplementation "org.junit.jupiter:junit-jupiter-engine:5.4.2"
  testImplementation "org.junit.vintage:junit-vintage-engine:5.4.2"

  //implementation "org.matheclipse:matheclipse-api:1.0.0-SNAPSHOT"
  //implementation "org.matheclipse:matheclipse-io:1.0.0-SNAPSHOT"
  //implementation "org.matheclipse:matheclipse-beakerx:1.0.0-SNAPSHOT"
...

Now, the server build seems to work. But, if I now run by using gradlew run -Dtests.es.xpack.security.enabled=false -Dtests.heap.size=1G, there are different tasks that fail:

> Configure project :modules:reindex
Disabling reindex-from-old tests because we can't get the pid file on windows

> Configure project :plugins:repository-hdfs
hdfsFixture unsupported, please set HADOOP_HOME and put HADOOP_HOME\bin in PATH

> Configure project :x-pack:plugin:searchable-snapshots:qa:hdfs
hdfsFixture unsupported, please set HADOOP_HOME and put HADOOP_HOME\bin in PATH
=======================================
Elasticsearch Build Hamster says Hello!
  Gradle Version        : 6.8
  OS Info               : Windows 10 10.0 (amd64)
  JDK Version           : 15 (Oracle)
  JAVA_HOME             : C:\Program Files\Java\jdk-15.0.1
  Random Testing Seed   : 266D7D1003A7C996
  In FIPS 140 mode      : false
=======================================

> Task :modules:kibana:bundlePlugin FAILED
> Task :modules:ingest-user-agent:bundlePlugin FAILED
> Task :modules:aggs-matrix-stats:bundlePlugin FAILED
> Task :modules:ingest-common:bundlePlugin FAILED
> Task :modules:lang-mustache:bundlePlugin FAILED
> Task :modules:ingest-geoip:bundlePlugin FAILED
> Task :modules:analysis-common:bundlePlugin FAILED
> Task :modules:geo:bundlePlugin FAILED

FAILURE: Build completed with 8 failures.

1: Task failed with an exception.
-----------
* What went wrong:
...

* Get more help at https://help.gradle.org

BUILD FAILED in 13s
204 actionable tasks: 9 executed, 195 up-to-date

I find this to be curious, since I didn't change anything on these parts of my Elasticsearch fork. What happened here?

Edit 4: After some problems with the "jar hell" functionality of Elasticsearch, I managed to build my fork with the help of @IanGabes by adding the repositories to the root's build.gradle. The problem is, that the Symja classes doesn't seem to work yet. I have a method normalize which currently does the following:

private void normalize(String queryString) {
    System.out.println("I'm here!");
    String eString = "";
    ExprEvaluator util = new ExprEvaluator(false, (short) 100);
    IExpr e;
}

The method is called in Elasticsearch's BoolQueryBuilder.doToQuery() right before the addBooleanClauses calls. Now I'm POSTing a BooleanQuery. There is the output "I'm here!", just as expected. After that, the following happens:

...
╗ [2021-04-19T22:54:29,779][INFO ][stdout ] [runTask-0] I'm here!
╗ [2021-04-19T20:54:30.497923300Z] [BUILD] Stopping node

=== Standard error of node `node{::runTask-0}` ===
╗ ? last 40 non error or warning messages from C:\{path...}\elasticsearch\build\testclusters\runTask-0\logs\es.stderr.log ?
╗ fatal error in thread [elasticsearch[runTask-0][search][T#1]], exiting
╗ java.lang.NoClassDefFoundError: com/google/common/util/concurrent/UncheckedTimeoutException
╗ at org.elasticsearch.index.query.BoolQueryBuilder.normalize(BoolQueryBuilder.java:440)
╗ at org.elasticsearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:332)
╗ at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:103)
╗ at org.elasticsearch.index.query.SearchExecutionContext.lambda$toQuery$3(SearchExecutionContext.java:464)
╗ at org.elasticsearch.index.query.SearchExecutionContext.toQuery(SearchExecutionContext.java:476)
╗ at org.elasticsearch.index.query.SearchExecutionContext.toQuery(SearchExecutionContext.java:463)
╗ at org.elasticsearch.search.SearchService.parseSource(SearchService.java:912)
╗ at org.elasticsearch.search.SearchService.createContext(SearchService.java:750)
╗ at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:438)
╗ at org.elasticsearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:406)
╗ at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:58)
╗ at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73)
╗ at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
╗ at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)
╗ at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:739)
╗ at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
╗ at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
╗ at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
╗ at java.base/java.lang.Thread.run(Thread.java:832)
╗ Caused by: java.lang.ClassNotFoundException: com.google.common.util.concurrent.UncheckedTimeoutException
╗ at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:606)
╗ at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:168)
╗ at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
╗ ... 19 more

> Task :run FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':run'.
> Elasticsearch cluster died

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at help.gradle.org

BUILD FAILED in 3m 34s
429 actionable tasks: 13 executed, 416 up-to-date

I don't see how the few commands for initializing of Symja classes can do any harm here. But when I comment them out, everything works fine, so the problem seems to be at this place. Following dependencies are included in the gradle.build at the moment (or commented out respecitvely, if they e.g. collide with other dependencies of different modules):

...
  implementation "org.matheclipse:matheclipse-external:1.0.0-SNAPSHOT"
  //implementation "org.apache.logging.log4j:log4j-core:2.13.2"
  //implementation "org.apache.logging.log4j:log4j-api:2.13.2"
  testImplementation "org.slf4j:slf4j-simple:1.7.25"
  //implementation "org.slf4j:log4j-over-slf4j:1.7.25"
  //implementation "com.google.guava:failureaccess:1.0.1"
  //implementation "com.google.guava:guava:30.1.1-jre"
  //implementation "com.fasterxml.jackson.core:jackson-databind:2.11.0"
  implementation "com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.11.0"
  implementation "io.pebbletemplates:pebble:3.1.5"
  implementation "org.jheaps:jheaps:0.13"
  //implementation "org.antlr:antlr4-runtime:4.8-1"
  implementation "org.apache.commons:commons-text:1.8"
  //testImplementation "junit:junit:4.13.1"

  implementation "org.matheclipse:matheclipse-frontend:1.0.0-SNAPSHOT"

  implementation "org.matheclipse:matheclipse-core:1.0.0-SNAPSHOT"
  implementation "org.commonmark:commonmark:0.17.1"
  implementation "org.commonmark:commonmark-ext-gfm-tables:0.17.1"
  implementation "org.apfloat:apfloat:1.10.0-SNAPSHOT"
  //implementation "org.apache.logging.log4j:log4j-core:2.13.2"
  //implementation "org.apache.logging.log4j:log4j-api:2.13.2"
  //testImplementation "org.slf4j:slf4j-simple:1.7.25"
  //implementation "org.slf4j:log4j-over-slf4j:1.7.25"
  implementation "org.hipparchus:hipparchus-core:1.8"
  implementation "org.hipparchus:hipparchus-clustering:1.8"
  implementation "org.hipparchus:hipparchus-fft:1.8"
  implementation "org.hipparchus:hipparchus-fitting:1.8"
  implementation "org.hipparchus:hipparchus-ode:1.8"
  implementation "org.hipparchus:hipparchus-optim:1.8"
  implementation "org.hipparchus:hipparchus-stat:1.8"
  //implementation "commons-codec:commons-codec:1.14"
  implementation "org.apache.commons:commons-lang3:3.11"
  implementation "com.google.code.findbugs:jsr305:3.0.0"
  implementation "org.logicng:logicng:2.0.2"
  implementation "org.gavaghan:geodesy:1.1.3"
  implementation "it.unimi.dsi:fastutil:8.5.2"
  implementation "org.codehaus.janino:janino:3.1.3"
  //implementation "com.ibm.icu:icu4j:68.2"
  implementation "com.univocity:univocity-parsers:2.8.4"
  testImplementation "org.junit.jupiter:junit-jupiter-api:5.4.2"
  testImplementation "org.junit.jupiter:junit-jupiter-engine:5.4.2"
  testImplementation "org.junit.vintage:junit-vintage-engine:5.4.2"
...

Answer 1

For the sake of completeness, I want to subsume at least the part of the solutions given by @axelclk and @IanGabes that worked. First of all, it seemed to be necessary to manually add all implicit dependencies plus the repositories they originate from to server's build.gradle, corresponding to the pom.xml files of matheclipse-core and of matheclipse-external:

...
apply plugin: 'java'
apply plugin: 'maven-publish'

publishing {
    publications {
        ...
        maven(MavenPublication) {
            from(components.java)
        }
    }
}

repositories {
  mavenCentral()
  maven {
    url = uri('https://oss.sonatype.org/content/repositories/snapshots')
  }
  maven {
    url = uri('https://repo.maven.apache.org/maven2/')
  }
}

...

  // Symja
  implementation "org.matheclipse:matheclipse-external:1.0.0-SNAPSHOT"
  //implementation "org.apache.logging.log4j:log4j-core:2.13.2"
  //implementation "org.apache.logging.log4j:log4j-api:2.13.2"
  testImplementation "org.slf4j:slf4j-simple:1.7.25"
  //implementation "org.slf4j:log4j-over-slf4j:1.7.25"
  implementation "com.google.guava:failureaccess:1.0.1"
  //implementation "com.google.guava:guava:30.1.1-jre"
  //implementation "com.fasterxml.jackson.core:jackson-databind:2.11.0"
  implementation "com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.11.0"
  implementation "io.pebbletemplates:pebble:3.1.5"
  implementation "org.jheaps:jheaps:0.13"
  //implementation "org.antlr:antlr4-runtime:4.8-1"
  implementation "org.apache.commons:commons-text:1.8"
  //testImplementation "junit:junit:4.13.1"

  implementation "org.matheclipse:matheclipse-frontend:1.0.0-SNAPSHOT"

  implementation "org.matheclipse:matheclipse-core:1.0.0-SNAPSHOT"
  implementation "org.commonmark:commonmark:0.17.1"
  implementation "org.commonmark:commonmark-ext-gfm-tables:0.17.1"
  implementation "org.apfloat:apfloat:1.10.0-SNAPSHOT"
  //implementation "org.apache.logging.log4j:log4j-core:2.13.2"
  //implementation "org.apache.logging.log4j:log4j-api:2.13.2"
  //testImplementation "org.slf4j:slf4j-simple:1.7.25"
  //implementation "org.slf4j:log4j-over-slf4j:1.7.25"
  implementation "org.hipparchus:hipparchus-core:1.8"
  implementation "org.hipparchus:hipparchus-clustering:1.8"
  implementation "org.hipparchus:hipparchus-fft:1.8"
  implementation "org.hipparchus:hipparchus-fitting:1.8"
  implementation "org.hipparchus:hipparchus-ode:1.8"
  implementation "org.hipparchus:hipparchus-optim:1.8"
  implementation "org.hipparchus:hipparchus-stat:1.8"
  //implementation "commons-codec:commons-codec:1.14"
  implementation "org.apache.commons:commons-lang3:3.11"
  implementation "com.google.code.findbugs:jsr305:3.0.0"
  implementation "org.logicng:logicng:2.0.2"
  implementation "org.gavaghan:geodesy:1.1.3"
  implementation "it.unimi.dsi:fastutil:8.5.2"
  implementation "org.codehaus.janino:janino:3.1.3"
  //implementation "com.ibm.icu:icu4j:68.2"
  implementation "com.univocity:univocity-parsers:2.8.4"
  testImplementation "org.junit.jupiter:junit-jupiter-api:5.4.2"
  testImplementation "org.junit.jupiter:junit-jupiter-engine:5.4.2"
  testImplementation "org.junit.vintage:junit-vintage-engine:5.4.2"
...

As depicted in the code above, some dependencies were commented out, though. This is because Elasticsearch has a functionality that throws build errors because of jar hell, when another version of a particular dependency is already used by other parts of the build.

Furthermore, since the server package depends on its parent package, it is necessary to add the repositories to the subprojects environment of elasticsearch's build.gradle:

...
subprojects {
  ...
  repositories {
    mavenCentral()
    maven {
      url = uri('https://oss.sonatype.org/content/repositories/snapshots')
    }
    maven {
      url = uri('https://repo.maven.apache.org/maven2/')
    }
  }
}
...

With this done, the packages of the Symja example are detected in my IDE (for me: IntelliJ IDEA), when the example is added to an existing class of Elasticsearch within the server module. (In my case, I called a method in org.elasticsearch.index.query.BoolQueryBuilder which contains the Symja example).

If I try and build/run everything using e.g. gradlew run -Dtests.es.xpack.security.enabled=false -Dtests.heap.size=1G from my Elasticsearch root directory, everything builds perfectly in the first place. It is also possible to create a document using PUT.... It is more of a problem to try and run a, in my case, Boolean query (which then calls the added method using Symja). In this case, Elasticsearch crashes immediately with the following output:

...
╗ [2021-04-22T08:34:21.178673600Z] [BUILD] Stopping node

=== Standard error of node `node{::runTask-0}` ===
╗   ? last 40 non error or warning messages from C:\...\elasticsearch\build\testclusters\runTask-0\logs\es.stderr.log ?
╗ fatal error in thread [elasticsearch[runTask-0][search][T#1]], exiting
╗  java.lang.NoClassDefFoundError: com/google/common/util/concurrent/UncheckedTimeoutException
...

> Task :run FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':run'.
> Elasticsearch cluster died

...

BUILD FAILED in 7m 37s
429 actionable tasks: 19 executed, 410 up-to-date

This is because Symja relies on the existence of the Guava package during runtime, which needed to be commented out from the build.gradle. On one hand, it is not possible to add Guava to any package with the implementation option, since it's defined as a "forbidden dependency" by Elasticsearch in the file gradle\forbidden-dependencies.gradle:

// we do not want any of these dependencies on the compilation classpath
// because they could then be used within Elasticsearch
List<String> FORBIDDEN_DEPENDENCIES = [
  'guava'
]
...

It's also not possible, on the other hand, to add it via runtimeOnly "com.google.guava:guava:30.1.1-jre" to the server module. This, again, leads to a build failure due to jar hell, because some testing parts of Elasticsearch also use Guava in this way. It's not effectife to try and adjust the version to the existing ones, either.

All in all, adding Symja content to Elasticsearch's server module seems to be not possible. Within the Elastic Stack Community, the following was stated:

[...] The problems you’re facing with dependencies etc are a consequence of trying to add stuff to server, and you won’t have that problem if you build a plugin.

This seems to be the best practice. I will tread this path, since it is my goal to create a plugin anyways.

Update: Since I managed installing the plugin I'm writing, using the workflow described above is perfectly working, indeed (with Guava included via runtimeOnly and using the build.gradle of my plugin).