JTB generating jj file that compiles but throws NullPointerExceptions on grammatically valid input

Question

I am very new to JavaCC and JTB. I am trying to get a a moderately complicated parser/validator going using a JTB file to generate the AST for me. I'm having a lot of problems making that work, so I decided to do the simplest example possible.

I am using Eclipse 3.8.1. My JavaCC and JTB plugins are:

plugins/sf.eclipse.javacc_1.5.30/jars/javacc-5.0.jar

plugins/sf.eclipse.javacc_1.5.30/jars/jtb-1.4.9.jar

I believe these to be fairly recent versions of what's available and so they should work OK.

I created a project and inside that project, I created a new JTB file. The plugin generated a bunch of code.

/**
 * JTB template file created by SF JavaCC plugin 1.5.28+ wizard for JTB 1.4.0.2+ and JavaCC 1.5.0+
 */
options
{
  static = true;
  JTB_P = "";
}

PARSER_BEGIN(SimpleGrammar)

// this import is not needed as it is generated by JTB
// import syntaxtree.*;
// this import is needed as it is not generated by JTB
import visitor.*;
public class SimpleGrammar
{
  public static void main(String args [])
  {
    System.out.println("Reading from standard input...");
    System.out.print("Enter an expression like \"1+(2+3)*var;\" :");
    new SimpleGrammar(System.in);
    try
    {
      Start start = SimpleGrammar.Start();
      DepthFirstVoidVisitor v = new MyVisitor();
      start.accept(v);
    }
    catch (Exception e)
    {
      System.out.println("Oops.");
      System.out.println(e);
      System.out.println(e.getMessage());
    }
  }
}

class MyVisitor extends DepthFirstVoidVisitor
{
  public void visit(NodeToken n)
  {
    System.out.println("visit " + n.tokenImage);
  }
}

PARSER_END(SimpleGrammar)

SKIP :
{
  " "
| "\t"
| "\n"
| "\r"
| < "//" (~[ "\n", "\r" ])*
    (
      "\n"
    | "\r"
    | "\r\n"
    ) >
| < "/*" (~[ "*" ])* "*"
    (
      ~[ "/" ] (~[ "*" ])* "*"
    )*
    "/" >
}

TOKEN : /* LITERALS */
{
  < INTEGER_LITERAL :
    < DECIMAL_LITERAL > ([ "l", "L" ])?
  | < HEX_LITERAL > ([ "l", "L" ])?
  | < OCTAL_LITERAL > ([ "l", "L" ])? 
    >
| < #DECIMAL_LITERAL : [ "1"-"9" ] ([ "0"-"9" ])* >
| < #HEX_LITERAL : "0" [ "x", "X" ] ([ "0"-"9", "a"-"f", "A"-"F" ])+ >
| < #OCTAL_LITERAL : "0" ([ "0"-"7" ])* >
}

TOKEN : /* IDENTIFIERS */
{
  < IDENTIFIER :
    < LETTER >
    (
      < LETTER >
    | < DIGIT >
    )* >
| < #LETTER : [ "_", "a"-"z", "A"-"Z" ] >
| < #DIGIT : [ "0"-"9" ] >
}

void Start() :
{}
{
  Expression() ";"
}

void Expression() :
{}
{
  AdditiveExpression()
}

void AdditiveExpression() :
{}
{
  MultiplicativeExpression()
  (
    (
      "+"
    | "-"
    )
    MultiplicativeExpression()
  )*
}

void MultiplicativeExpression() :
{}
{
  UnaryExpression()
  (
    (
      "*"
    | "/"
    | "%"
    )
    UnaryExpression()
  )*
}

void UnaryExpression() :
{}
{
  "(" Expression() ")"
| Identifier()
| MyInteger()
}

void Identifier() :
{}
{
  < IDENTIFIER >
}

void MyInteger() :
{}
{
  < INTEGER_LITERAL >
}

It actually had a bunch of ?parser_name? tags wherever you see SimpleGrammar which I noticed and changed.

So I right-click on the SimpleGrammar.jtb file and hit "Compile with JavaCC | JJTree | JTB" and it generates all the files just like it's supposed to. So I then press F11 to run the program. Here is the output of that console session:

Reading from standard input...
Enter an expression like "1+(2+3)*var;" :1+2
Oops.
java.lang.NullPointerException
null

That's funny, it probably shouldn't be doing that. So I change the exception catching to ParseException so that it won't catch the NullPointerException and I'll get a debug console. After I do that and hunt around a bit, I find this snippet of code:

  static final public MyInteger MyInteger() throws ParseException {
  // --- JTB generated node declarations ---
  NodeToken n0 = null;
  Token n1 = null;
    jj_consume_token(INTEGER_LITERAL);
    n0 = JTBToolkit.makeNodeToken(n1);
    {if (true) return new MyInteger(n0);}
    throw new Error("Missing return statement in function");
  }

So what's happening here is that n1 is getting referenced before assigned to. I don't know why this is happening.

I've also looked in the generated jtb.out.jj file to see what's there. Here's the snippet corresponding to the code that I'm seeing:

MyInteger MyInteger() :
{
  // --- JTB generated node declarations ---
  NodeToken n0 = null;
  Token n1 = null;
}
{
  < INTEGER_LITERAL >
  { n0 = JTBToolkit.makeNodeToken(n1); }
  { return new MyInteger(n0); }
}

Is that wrong? I'm honestly not sure. I was under the impression that you did everything in the JTB file and all the files that were generated after that were not to be touched.

Any insight as to what's going wrong here would be greatly appreciated.

EDIT

So I've done a bit of poking around and I've found that if I change

<INTEGER_LITERAL>

to

n1 = <INTEGER_LITERAL>

in the above-mentioned snipped then there won't be a null pointer exception anymore I can successfully enter "1;" at the console and it'll parse.

I think what I'm trying to figure out is why JTB is generating a bogus jtb.out.jj rather than what can I do to stop the null pointer exceptions. I've got a decent sized, non-toy grammar I want to work with and manually editing the jtb.out.jj file every time is not a scalable solution.

Answer 1

I had the same problem, it's a bug in the generation. If you use version JTB 1.4.7, it will probably work. You can find the available versions here: https://java.net/projects/jtb/sources/svn/show/trunk/lib?rev=75 (the download section doesn't really work for me)