Nearley Moo - Grammar does not work with Moo lexer used

Question

I'm using the nearley.js grammar (and parser) with the moo.js tokeniser. My grammar.ne file is the following:

@{%
    const moo = require('moo')
    let lexer = moo.compile({
        number: /[0-9]+/
    });
%}

@lexer lexer

trig -> "sin" [0-9]:+

When parsing the string "sin8" to a parser, via nearley-test grammar.js -i "sin8", I get the following error:

throw new Error(this.formatError(token, "invalid syntax"))
      ^

Error: invalid syntax at line 1 col 1:

  sin8
  ^
    at Lexer.next (C:\Users\Florian\WebstormProjects\MineScript\node_modules\moo\moo.js:397:13)
    at Parser.feed (C:\Users\Florian\AppData\Roaming\npm\node_modules\nearley\lib\nearley.js:270:30)
    at Object.<anonymous> (C:\Users\Florian\AppData\Roaming\npm\node_modules\nearley\bin\nearley-test.js:83:12)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)
    at Function.Module.runMain (module.js:693:10)
    at startup (bootstrap_node.js:191:16)

However, commenting @lexer lexer makes it work, and matches the "sin8" string. This example is taken directly from the documentation and does not work, so I'm wondering where I am wrong.

Answer 1

If moo is passed as a lexer, Nearley doesn't really understand simple string inputs. You will need to specify it in your tokenizer.

@{%
    const moo = require('moo')
    let lexer = moo.compile({
        number: /[0-9]+/,
        sin: /\bsin\b/,
    });
%}

@lexer lexer

trig -> "sin" [0-9]:+

Make sure to have \b to make sure that it is a separate word, and not a part of it.

Answer 2

Having played with Nearley a bit, it seems to me that once you starting using Moo . you have to define all possible tokens in the Moo lexer and you can only use the Nearley rules and post-processors. So something like the below would probably work (I didn't test it):

@{%
    const moo = require('moo')
    let lexer = moo.compile({
        number: /[0-9]+/,
        func: ['sin','cos'] //define sin, cos etc, here
    });
%}

@lexer lexer

trig -> %func %number