Spoon Inria. Not resolving CtExecutableReference type and declared type depending on environment (Docker and local Spring Boot)

Question

Good day.

I encountered very strange issue using Spoon analysis library in my Java project. I get different parsing results when running same code, that uses Spoon Launcher in different environments

• 1 Environment - Spring Boot Project that is run locally from Intellij IDEA
• 2 Environment - Same Spring Boot project that is run in a Docker Container

In locally run Spring Boot project - it's all OK, but when I run the same code in Docker - I get null values in CtExecutableReference.getType() and CtExecutableReference.getDeclaredType()

I opened issue on GitHub - https://github.com/INRIA/spoon/issues/3926

Here are the details

My Spoon version is 8.2.0. (from maven repo)

I am trying to parse (build AST) of code from this GitHub repository And I have troubles parsing this class There are following lines here

...
@Service
public class ValueServices {
    
    private ValuesRepository valuesRepository;
    
    private Queue<Values> queue;

    @Autowired
    public ValueServices(ValuesRepository valuesRepository) {
        super();
        this.valuesRepository = valuesRepository;
        this.queue = new LinkedList<Values>();
    }
    
    public List<Values> getAllValues() {
        List<Values> values = new ArrayList<>();
        this.valuesRepository.findAll().forEach(values::add);
        return values;
    }

...
}

When I run analysis and try to parse CtExecutableReference for findAll() method of this.valuesRepository.findAll().forEach(values::add) statement I get null values for getType() and getDeclaredType() when running my project in Docker. When running locally both getType() and getDeclaredType() have non-null values

The same issue happens when parsing other similar code blocks in other projects. For example here

@Service
public class BetService {

    public static final String DATE_FORMAT_NOW = "yyyy-MM-dd HH";
    public static final String DATE_FORMAT_NOW_WITH_HOUR_MIN = "yyyy-MM-dd HH:mm:ss";

    private BetRepository betRepository;

    @Autowired
    public BetService(BetRepository betRepository) {
        super();
        this.betRepository = betRepository;
    }

    public List<Bet> getAllBets() {
        List<Bet> bets = new ArrayList<Bet>();
        this.betRepository.findAll().forEach(bets::add);

        return bets;
    }
}

Statement with this.betRepository.findAll() has null in both getType and getDeclaredType when running in Docker, but ok in local environment.

At the same time following code is parsed well in both environments

public class BetRepositoryTest {
    
    @Autowired
    private TestEntityManager entityManager;
 
    @Autowired
    private BetRepository betRepository;

    @Test
    public void test() {
        Bet bet = new Bet("2018-07-06 12:56", "WIN", 103333, 1082, 500.5);
        entityManager.persist(bet);
        entityManager.flush();
       
        List<Bet> bets = betRepository.findByCustomerId(bet.getCustomerId());       
        assertThat(bet.getCustomerId() == bets.get(0).getCustomerId());
    }
}

and statement betRepository.findByCustomerId() is parsed ok and has necessary type info both in Docker and n local Spring Boot run.

I double checked local tests - and all is ok - when running code in tests from IDE or starting Spring Boot project from IDE and initialising analysis by calling service from Web UI - it OK and works as expected.

But when I build Docker image - I get null in both type and declaredType.

I'm running Spoon analysis with the following code

private SourceCodeMetamodel buildMetamodelForFiles(Collection<File> javaFiles) {
        Launcher spoonAPI = new Launcher();
        log.debug("Spoon environment - {}",ToStringBuilder.reflectionToString(spoonAPI.getEnvironment()));
        log.debug("Spoon model builder - {}",ToStringBuilder.reflectionToString(spoonAPI.getModelBuilder()));
        Set<String> inputResources = new HashSet<>();
        for (File javaFile: javaFiles) {
            String javaDir = JavaFileUtils.getJavaFileStorageRootPath(javaFile);
            if (StringUtils.isNotBlank(javaDir) && !inputResources.contains(javaDir)) {
                spoonAPI.addInputResource(javaDir);
                inputResources.add(javaDir);
            }
            else if (StringUtils.isBlank(javaDir)) {
                spoonAPI.addInputResource(javaFile.getAbsolutePath());
            }
        }
        spoonAPI.buildModel();
        CtModel ctModel = spoonAPI.getModel();
        Collection<CtType<?>> modelTypes = ctModel.getAllTypes();
        return new SpoonSourceCodeMetamodel(modelTypes,false);
    }

Before running Launcher I tried to print it's settings. Here is what i got

2021-05-14 13:26:12.329 DEBUG 1 --- [         task-1] c.s.s.r.i.s.SpoonJavaSourceCodeAnalyzer  : Spoon environment - spoon.support.StandardEnvironment@4626a7ce[errorCount=0,processingStopped=false,prettyPrintingMode=FULLYQUALIFIED,warningCount=0,sourceClasspath=<null>,preserveLineNumbers=false,copyResources=true,enableComments=true,level=ERROR,shouldCompile=false,skipSelfChecks=false,complianceLevel=8,previewFeaturesEnabled=false,outputType=classes,noclasspath=true,compressionType=GZIP,sniperMode=false,ignoreDuplicateDeclarations=false,prettyPrinterCreator=<null>,useTabulations=false,tabulationSize=4,binaryOutputDirectory=/spooned-classes]

2021-05-14 13:26:12.333 DEBUG 1 --- [         task-1] c.s.s.r.i.s.SpoonJavaSourceCodeAnalyzer  : Spoon model builder - spoon.support.compiler.jdt.JDTBasedSpoonCompiler@2df98092[environment=<null>,probs=[],requestor=spoon.support.compiler.jdt.TreeBuilderRequestor@57050733,factory=spoon.reflect.factory.FactoryImpl@237d7ffa,javaCompliance=7,sources=<virtual folder>: spoon.support.compiler.VirtualFolder@42618617,templates=<virtual folder>: spoon.support.compiler.VirtualFolder@7fb32ea2,templateClasspath={},compilationUnitFilters=[],sortList=true]

I'm running project on Java8 - in both environments (details following). To build docker I use following commands

FROM java:8
COPY maven /maven/
ENTRYPOINT java -Xverify:none -XX:TieredStopAtLevel=1 -XX:+TieredCompilation -XX:+UseSerialGC -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Djava.security.egd=file:/dev/./urandom -Dspring.profiles.active=${PROFILE:-docker-dev} -jar /maven/skillcounters-sca-service-1.0-SNAPSHOT.jar

I tried to switch to different Docker base images (openjdk \ alpine etc.), but nothing helped. I tried to exclude all java run options listed above (i.e. like -XXblabla) - didn't helped either.

To get ideas about what may go wrong i print all environment (including java) data on application start.

Here what is printed for local environment

Apple_PubSub_Socket_Render : /private/tmp/com.apple.launchd.xCrhs0tTMM/Render
COMMAND_MODE : unix2003
HOME : /Users/sk
JAVA_MAIN_CLASS_66239 : org.codehaus.classworlds.Launcher
JAVA_MAIN_CLASS_66249 : com.skillcounters.sca.SCAServiceApplication
LANG : ru_RU.UTF-8
LC_CTYPE : ru_RU.UTF-8
LOGNAME : sk
PATH : /Users/sk/Develop/d20/db/liquibase:/Users/sk/anaconda/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
PID : 66249
PWD : /Users/sk/Develop/d20/d20-git-repo/skillcounters-sca-service/skillcounters-sca-service-impl
SECURITYSESSIONID : 186a9
SHELL : /bin/bash
SSH_AUTH_SOCK : /private/tmp/com.apple.launchd.ItaWSltKcA/Listeners
TMPDIR : /var/folders/lz/gjd4j2t12_39qs0hpdjqd3sh0000gn/T/
USER : sk
XPC_FLAGS : 0x0
XPC_SERVICE_NAME : com.apple.xpc.launchd.oneshot.0x10000002.idea
__CF_USER_TEXT_ENCODING : 0x1F5:0x0:0x0
awt.toolkit : sun.lwawt.macosx.LWCToolkit
file.encoding : UTF-8
file.encoding.pkg : sun.io
file.separator : /
ftp.nonProxyHosts : local|*.local|169.254/16|*.169.254/16
gopherProxySet : false
http.nonProxyHosts : local|*.local|169.254/16|*.169.254/16
java.awt.graphicsenv : sun.awt.CGraphicsEnvironment
java.awt.headless : true
java.awt.printerjob : sun.lwawt.macosx.CPrinterJob
java.class.path : {all dependent jars go here - excluded them not to pollute issue...}
java.class.version : 52.0
java.endorsed.dirs : /Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home/jre/lib/endorsed
java.ext.dirs : /Users/sk/Library/Java/Extensions:/Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home/jre/lib/ext:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
java.home : /Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home/jre
java.io.tmpdir : /var/folders/lz/gjd4j2t12_39qs0hpdjqd3sh0000gn/T/
java.library.path : /Users/sk/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
java.runtime.name : Java(TM) SE Runtime Environment
java.runtime.version : 1.8.0_131-b11
java.specification.name : Java Platform API Specification
java.specification.vendor : Oracle Corporation
java.specification.version : 1.8
java.vendor : Oracle Corporation
java.vendor.url : http://java.oracle.com/
java.vendor.url.bug : http://bugreport.sun.com/bugreport/
java.version : 1.8.0_131
java.vm.info : mixed mode
java.vm.name : Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.name : Java Virtual Machine Specification
java.vm.specification.vendor : Oracle Corporation
java.vm.specification.version : 1.8
java.vm.vendor : Oracle Corporation
java.vm.version : 25.131-b11

And here is what printed in Docker environment

CA_CERTIFICATES_JAVA_VERSION : 20140324
HOME : /root
HOSTNAME : e297584466e8
JAVA_DEBIAN_VERSION : 8u111-b14-2~bpo8+1
JAVA_HOME : /usr/lib/jvm/java-8-openjdk-amd64
JAVA_VERSION : 8u111
LANG : C.UTF-8
PATH : /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PID : 8
PROFILE : prod
PWD : /
awt.toolkit : sun.awt.X11.XToolkit
file.encoding : UTF-8
file.encoding.pkg : sun.io
file.separator : /
java.awt.graphicsenv : sun.awt.X11GraphicsEnvironment
java.awt.headless : true
java.awt.printerjob : sun.print.PSPrinterJob
java.class.path : /maven/skillcounters-skill-service-1.0-SNAPSHOT.jar
java.class.version : 52.0
java.endorsed.dirs : /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/endorsed
java.ext.dirs : /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext:/usr/java/packages/lib/ext
java.home : /usr/lib/jvm/java-8-openjdk-amd64/jre
java.io.tmpdir : /tmp
java.library.path : /usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
java.protocol.handler.pkgs : org.springframework.boot.loader
java.runtime.name : OpenJDK Runtime Environment
java.runtime.version : 1.8.0_111-8u111-b14-2~bpo8+1-b14
java.security.egd : file:/dev/./urandom
java.specification.name : Java Platform API Specification
java.specification.vendor : Oracle Corporation
java.specification.version : 1.8
java.vendor : Oracle Corporation
java.vendor.url : http://java.oracle.com/
java.vendor.url.bug : http://bugreport.sun.com/bugreport/
java.version : 1.8.0_111
java.vm.info : mixed mode
java.vm.name : OpenJDK 64-Bit Server VM
java.vm.specification.name : Java Virtual Machine Specification
java.vm.specification.vendor : Oracle Corporation
java.vm.specification.version : 1.8
java.vm.vendor : Oracle Corporation
java.vm.version : 25.111-b14

Any help would be appreciated

Answer 1

I found some clue when I enabled Spoon Launcher logging by setting it to 'ALL'

Here is what I got in Docker environment

...
The method findAll() is undefined for the type ValuesRepository at /opt/skillcounters/filestorage/github/vipinjo/assignment-consumer/work-tree/d4e050cf3566981ba386e336c5889f8d107abfbb/src/main/java/com/vipinjoseph/assignmentconsumer/service/ValueServices.java:32

And in local environment these warning are not shown...

There is some classpath problem similar to https://www.programmersought.com/article/47106442813/

After almost day of struggling with it I found a solution

0. Use Spoon 9.0.0 or later with this fix
1. Create classloader with a parent classloader from Spoon Launcher
2. Directly include each *.jar file into classloader URL classpath (not a directory containing jars but each jar in a single URL passed to classloader)
3. Set this classloader as an InputClassLoader

                URLClassLoader urlClassLoader = URLClassLoader.newInstance(
                                                                jarUrls.toArray(new URL[0])
                                                                ,spoonAPI.getClass().getClassLoader()
                                                );
                spoonAPI.getEnvironment().setInputClassLoader(urlClassLoader);