For a project I wanted to extend Elasticsearch and therefore need to use the package Symja. In the Github for Symja, there is a manual for the usage with Maven provided.
Since the Elasticsearch repository is build with Gradle, I also need to use Gradle instead of Maven. Testing the suggested example Symja project, the following build.gradle
(which I basically generated by using gradle init
and adjusted a little) imports the library flawlessly:
apply plugin: 'java'
apply plugin: 'maven-publish'
repositories {
mavenLocal()
maven {
url = uri('https://oss.sonatype.org/content/repositories/snapshots')
}
maven {
url = uri('https://repo.maven.apache.org/maven2/')
}
}
dependencies {
implementation 'org.matheclipse:matheclipse-core:1.0.0-SNAPSHOT'
}
publishing {
publications {
maven(MavenPublication) {
from(components.java)
}
}
}
Now, I wanted to add Symja to my Elasticsearch fork into the server
module.
When I try to build this, though, the imports
import org.matheclipse.parser.client.SyntaxError;
import org.matheclipse.parser.client.math.MathException;
of the Symja example aren't detected: java: package org.matheclipse.parser.client does not exist
. What may have gone wrong?
My updated build.gradle
of the server
module looks like this:
...
apply plugin: 'java'
apply plugin: 'maven-publish'
publishing {
publications {
...
maven(MavenPublication) {
from(components.java)
}
}
}
repositories {
mavenLocal()
maven {
url = uri('https://oss.sonatype.org/content/repositories/snapshots')
}
maven {
url = uri('https://repo.maven.apache.org/maven2/')
}
}
...
dependencies {
...
// Symja
implementation "org.matheclipse:matheclipse-core:1.0.0-SNAPSHOT"
...
}
...
Edit:
After following @axelclk's suggestion to add implementation "org.matheclipse:matheclipse-frontend:1.0.0-SNAPSHOT"
, the packages are now detected by my IDE. Although, if I build the build.gradle
(while containing the Symja example whithin a method in org.elasticsearch.index.query.BoolQueryBuilder.java
), following error appears:
C:\{path here...}\elasticsearch\server\src\main\java\org\elasticsearch\index\query\BoolQueryBuilder.java:454: error: cannot access GcdRingElem
System.out.println("Out[2]: " + result.toString());
^
class file for edu.jas.structure.GcdRingElem not found
C:\{path here...}\elasticsearch\server\src\main\java\org\elasticsearch\index\query\BoolQueryBuilder.java:459: error: cannot access FieldElement
System.out.println("Out[3]: " + result.toString());
^
class file for org.hipparchus.FieldElement not found
It seems like somehow, the toString
method doesn't work properly. Also, if I run my Elasticsearch fork with gradlew run -Dtests.es.xpack.security.enabled=false -Dtests.heap.size=1G
, the build fails with following output:
> Configure project :modules:reindex
Disabling reindex-from-old tests because we can't get the pid file on windows
> Configure project :plugins:repository-hdfs
hdfsFixture unsupported, please set HADOOP_HOME and put HADOOP_HOME\bin in PATH
> Configure project :x-pack:plugin:searchable-snapshots:qa:hdfs
hdfsFixture unsupported, please set HADOOP_HOME and put HADOOP_HOME\bin in PATH
=======================================
Elasticsearch Build Hamster says Hello!
Gradle Version : 6.8
OS Info : Windows 10 10.0 (amd64)
JDK Version : 15 (Oracle)
JAVA_HOME : C:\Program Files\Java\jdk-15.0.1
Random Testing Seed : C5A79F8CABA0D6FA
In FIPS 140 mode : false
=======================================
> Task :server:compileJava
C:\{path here ...}\elasticsearch\server\src\main\java\org\elasticsearch\index\query\BoolQueryBuilder.java:454: error: cannot access GcdRingElem
System.out.println("Out[2]: " + result.toString());
^
class file for edu.jas.structure.GcdRingElem not found
C:\{path here ...}\elasticsearch\server\src\main\java\org\elasticsearch\index\query\BoolQueryBuilder.java:459: error: cannot access FieldElement
System.out.println("Out[3]: " + result.toString());
^
class file for org.hipparchus.FieldElement not found
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
2 errors
> Task :server:compileJava FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':server:compileJava'.
> Compilation failed; see the compiler error output for details.
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 58s
228 actionable tasks: 8 executed, 220 up-to-date
Edit 2: I added implementation "org.matheclipse:matheclipse-external:1.0.0-SNAPSHOT"
and replaced mavenLocal()
with mavenCentral()
as suggested by @axelclk which doesn't lead to any different results.
Edit 3: I added -- again, based on suggestions from @axelclk -- the implementations described by the pom.xml
files of matheclipse-external
and matheclipse-core
:
...
// Symja
implementation "org.matheclipse:matheclipse-external:1.0.0-SNAPSHOT"
//implementation "org.apache.logging.log4j:log4j-core:2.13.2"
//implementation "org.apache.logging.log4j:log4j-api:2.13.2"
testImplementation "org.slf4j:slf4j-simple:1.7.25"
implementation "org.slf4j:log4j-over-slf4j:1.7.25"
//implementation "com.google.guava:failureaccess:1.0.1"
//implementation "com.google.guava:guava:30.1.1-jre"
implementation "com.fasterxml.jackson.core:jackson-databind:2.11.0"
implementation "com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.11.0"
implementation "io.pebbletemplates:pebble:3.1.5"
implementation "org.jheaps:jheaps:0.13"
implementation "org.antlr:antlr4-runtime:4.8-1"
implementation "org.apache.commons:commons-text:1.8"
//testImplementation "junit:junit:4.13.1"
implementation "org.matheclipse:matheclipse-frontend:1.0.0-SNAPSHOT"
implementation "org.matheclipse:matheclipse-core:1.0.0-SNAPSHOT"
implementation "org.commonmark:commonmark:0.17.1"
implementation "org.commonmark:commonmark-ext-gfm-tables:0.17.1"
implementation "org.apfloat:apfloat:1.10.0-SNAPSHOT"
//implementation "org.apache.logging.log4j:log4j-core:2.13.2"
//implementation "org.apache.logging.log4j:log4j-api:2.13.2"
//testImplementation "org.slf4j:slf4j-simple:1.7.25"
//implementation "org.slf4j:log4j-over-slf4j:1.7.25"
implementation "org.hipparchus:hipparchus-core:1.8"
implementation "org.hipparchus:hipparchus-clustering:1.8"
implementation "org.hipparchus:hipparchus-fft:1.8"
implementation "org.hipparchus:hipparchus-fitting:1.8"
implementation "org.hipparchus:hipparchus-ode:1.8"
implementation "org.hipparchus:hipparchus-optim:1.8"
implementation "org.hipparchus:hipparchus-stat:1.8"
//implementation "commons-codec:commons-codec:1.14"
implementation "org.apache.commons:commons-lang3:3.11"
implementation "com.google.code.findbugs:jsr305:3.0.0"
implementation "org.logicng:logicng:2.0.2"
implementation "org.gavaghan:geodesy:1.1.3"
implementation "it.unimi.dsi:fastutil:8.5.2"
implementation "org.codehaus.janino:janino:3.1.3"
implementation "com.ibm.icu:icu4j:68.2"
implementation "com.univocity:univocity-parsers:2.8.4"
testImplementation "org.junit.jupiter:junit-jupiter-api:5.4.2"
testImplementation "org.junit.jupiter:junit-jupiter-engine:5.4.2"
testImplementation "org.junit.vintage:junit-vintage-engine:5.4.2"
//implementation "org.matheclipse:matheclipse-api:1.0.0-SNAPSHOT"
//implementation "org.matheclipse:matheclipse-io:1.0.0-SNAPSHOT"
//implementation "org.matheclipse:matheclipse-beakerx:1.0.0-SNAPSHOT"
...
Now, the server
build seems to work. But, if I now run by using gradlew run -Dtests.es.xpack.security.enabled=false -Dtests.heap.size=1G
, there are different tasks that fail:
> Configure project :modules:reindex
Disabling reindex-from-old tests because we can't get the pid file on windows
> Configure project :plugins:repository-hdfs
hdfsFixture unsupported, please set HADOOP_HOME and put HADOOP_HOME\bin in PATH
> Configure project :x-pack:plugin:searchable-snapshots:qa:hdfs
hdfsFixture unsupported, please set HADOOP_HOME and put HADOOP_HOME\bin in PATH
=======================================
Elasticsearch Build Hamster says Hello!
Gradle Version : 6.8
OS Info : Windows 10 10.0 (amd64)
JDK Version : 15 (Oracle)
JAVA_HOME : C:\Program Files\Java\jdk-15.0.1
Random Testing Seed : 266D7D1003A7C996
In FIPS 140 mode : false
=======================================
> Task :modules:kibana:bundlePlugin FAILED
> Task :modules:ingest-user-agent:bundlePlugin FAILED
> Task :modules:aggs-matrix-stats:bundlePlugin FAILED
> Task :modules:ingest-common:bundlePlugin FAILED
> Task :modules:lang-mustache:bundlePlugin FAILED
> Task :modules:ingest-geoip:bundlePlugin FAILED
> Task :modules:analysis-common:bundlePlugin FAILED
> Task :modules:geo:bundlePlugin FAILED
FAILURE: Build completed with 8 failures.
1: Task failed with an exception.
-----------
* What went wrong:
...
* Get more help at https://help.gradle.org
BUILD FAILED in 13s
204 actionable tasks: 9 executed, 195 up-to-date
I find this to be curious, since I didn't change anything on these parts of my Elasticsearch fork. What happened here?
Edit 4: After some problems with the "jar hell" functionality of Elasticsearch, I managed to build my fork with the help of @IanGabes by adding the repositories
to the root's build.gradle
.
The problem is, that the Symja classes doesn't seem to work yet. I have a method normalize
which currently does the following:
private void normalize(String queryString) {
System.out.println("I'm here!");
String eString = "";
ExprEvaluator util = new ExprEvaluator(false, (short) 100);
IExpr e;
}
The method is called in Elasticsearch's BoolQueryBuilder.doToQuery()
right before the addBooleanClauses
calls. Now I'm POST
ing a BooleanQuery
. There is the output "I'm here!"
, just as expected. After that, the following happens:
...
╗ [2021-04-19T22:54:29,779][INFO ][stdout ] [runTask-0] I'm here!
╗ [2021-04-19T20:54:30.497923300Z] [BUILD] Stopping node
=== Standard error of node `node{::runTask-0}` ===
╗ ? last 40 non error or warning messages from C:\{path...}\elasticsearch\build\testclusters\runTask-0\logs\es.stderr.log ?
╗ fatal error in thread [elasticsearch[runTask-0][search][T#1]], exiting
╗ java.lang.NoClassDefFoundError: com/google/common/util/concurrent/UncheckedTimeoutException
╗ at org.elasticsearch.index.query.BoolQueryBuilder.normalize(BoolQueryBuilder.java:440)
╗ at org.elasticsearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:332)
╗ at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:103)
╗ at org.elasticsearch.index.query.SearchExecutionContext.lambda$toQuery$3(SearchExecutionContext.java:464)
╗ at org.elasticsearch.index.query.SearchExecutionContext.toQuery(SearchExecutionContext.java:476)
╗ at org.elasticsearch.index.query.SearchExecutionContext.toQuery(SearchExecutionContext.java:463)
╗ at org.elasticsearch.search.SearchService.parseSource(SearchService.java:912)
╗ at org.elasticsearch.search.SearchService.createContext(SearchService.java:750)
╗ at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:438)
╗ at org.elasticsearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:406)
╗ at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:58)
╗ at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73)
╗ at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
╗ at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)
╗ at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:739)
╗ at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
╗ at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
╗ at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
╗ at java.base/java.lang.Thread.run(Thread.java:832)
╗ Caused by: java.lang.ClassNotFoundException: com.google.common.util.concurrent.UncheckedTimeoutException
╗ at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:606)
╗ at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:168)
╗ at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
╗ ... 19 more
> Task :run FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':run'.
> Elasticsearch cluster died
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at help.gradle.org
BUILD FAILED in 3m 34s
429 actionable tasks: 13 executed, 416 up-to-date
I don't see how the few commands for initializing of Symja classes can do any harm here. But when I comment them out, everything works fine, so the problem seems to be at this place.
Following dependencies are included in the gradle.build
at the moment (or commented out respecitvely, if they e.g. collide with other dependencies of different modules):
...
implementation "org.matheclipse:matheclipse-external:1.0.0-SNAPSHOT"
//implementation "org.apache.logging.log4j:log4j-core:2.13.2"
//implementation "org.apache.logging.log4j:log4j-api:2.13.2"
testImplementation "org.slf4j:slf4j-simple:1.7.25"
//implementation "org.slf4j:log4j-over-slf4j:1.7.25"
//implementation "com.google.guava:failureaccess:1.0.1"
//implementation "com.google.guava:guava:30.1.1-jre"
//implementation "com.fasterxml.jackson.core:jackson-databind:2.11.0"
implementation "com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.11.0"
implementation "io.pebbletemplates:pebble:3.1.5"
implementation "org.jheaps:jheaps:0.13"
//implementation "org.antlr:antlr4-runtime:4.8-1"
implementation "org.apache.commons:commons-text:1.8"
//testImplementation "junit:junit:4.13.1"
implementation "org.matheclipse:matheclipse-frontend:1.0.0-SNAPSHOT"
implementation "org.matheclipse:matheclipse-core:1.0.0-SNAPSHOT"
implementation "org.commonmark:commonmark:0.17.1"
implementation "org.commonmark:commonmark-ext-gfm-tables:0.17.1"
implementation "org.apfloat:apfloat:1.10.0-SNAPSHOT"
//implementation "org.apache.logging.log4j:log4j-core:2.13.2"
//implementation "org.apache.logging.log4j:log4j-api:2.13.2"
//testImplementation "org.slf4j:slf4j-simple:1.7.25"
//implementation "org.slf4j:log4j-over-slf4j:1.7.25"
implementation "org.hipparchus:hipparchus-core:1.8"
implementation "org.hipparchus:hipparchus-clustering:1.8"
implementation "org.hipparchus:hipparchus-fft:1.8"
implementation "org.hipparchus:hipparchus-fitting:1.8"
implementation "org.hipparchus:hipparchus-ode:1.8"
implementation "org.hipparchus:hipparchus-optim:1.8"
implementation "org.hipparchus:hipparchus-stat:1.8"
//implementation "commons-codec:commons-codec:1.14"
implementation "org.apache.commons:commons-lang3:3.11"
implementation "com.google.code.findbugs:jsr305:3.0.0"
implementation "org.logicng:logicng:2.0.2"
implementation "org.gavaghan:geodesy:1.1.3"
implementation "it.unimi.dsi:fastutil:8.5.2"
implementation "org.codehaus.janino:janino:3.1.3"
//implementation "com.ibm.icu:icu4j:68.2"
implementation "com.univocity:univocity-parsers:2.8.4"
testImplementation "org.junit.jupiter:junit-jupiter-api:5.4.2"
testImplementation "org.junit.jupiter:junit-jupiter-engine:5.4.2"
testImplementation "org.junit.vintage:junit-vintage-engine:5.4.2"
...
For the sake of completeness, I want to subsume at least the part of the solutions given by @axelclk and @IanGabes that worked. First of all, it seemed to be necessary to manually add all implicit dependencies plus the repositories they originate from to
server
'sbuild.gradle
, corresponding to thepom.xml
files ofmatheclipse-core
and ofmatheclipse-external
:As depicted in the code above, some dependencies were commented out, though. This is because Elasticsearch has a functionality that throws build errors because of
jar hell
, when another version of a particular dependency is already used by other parts of the build.Furthermore, since the
server
package depends on its parent package, it is necessary to add the repositories to thesubprojects
environment ofelasticsearch
'sbuild.gradle
:With this done, the packages of the Symja example are detected in my IDE (for me: IntelliJ IDEA), when the example is added to an existing class of Elasticsearch within the
server
module. (In my case, I called a method inorg.elasticsearch.index.query.BoolQueryBuilder
which contains the Symja example).If I try and build/run everything using e.g.
gradlew run -Dtests.es.xpack.security.enabled=false -Dtests.heap.size=1G
from my Elasticsearch root directory, everything builds perfectly in the first place. It is also possible to create a document usingPUT...
. It is more of a problem to try and run a, in my case, Boolean query (which then calls the added method using Symja). In this case, Elasticsearch crashes immediately with the following output:This is because Symja relies on the existence of the Guava package during runtime, which needed to be commented out from the
build.gradle
. On one hand, it is not possible to add Guava to any package with theimplementation
option, since it's defined as a "forbidden dependency" by Elasticsearch in the filegradle\forbidden-dependencies.gradle
:It's also not possible, on the other hand, to add it via
runtimeOnly "com.google.guava:guava:30.1.1-jre"
to theserver
module. This, again, leads to a build failure due tojar hell
, because some testing parts of Elasticsearch also use Guava in this way. It's not effectife to try and adjust the version to the existing ones, either.All in all, adding Symja content to Elasticsearch's
server
module seems to be not possible. Within the Elastic Stack Community, the following was stated:This seems to be the best practice. I will tread this path, since it is my goal to create a plugin anyways.
Update: Since I managed installing the plugin I'm writing, using the workflow described above is perfectly working, indeed (with Guava included via
runtimeOnly
and using thebuild.gradle
of my plugin).