Irony: How to disallow a space between 2 tokens?

Question

I'm trying to define PHP-style variables in Irony like so:

variable.Rule = "$" + identifier;

Works great, except that you're allowed to put spaces between the $ and the identifier. I want to prevent that. How?

Do I have to create a new customized terminal? If so, will I still be able to take advantage of the IdentifierTerminal magic?

---

Digging around in IdentifierTerminal I see there's actually a flag for "NameIncludesPrefix", but it's only used in one place. Looks like the prefix is stored in this CompoundTokenDetails object... which I'm not sure how to use. Edit: Nevermind, this was a dead-end. Those flags are for adding modifiers to how the variable behaves.

---

This kinda works...

class VariableTerminal : Terminal
{
    public VariableTerminal(string name) : base(name)
    {
    }

    public override IList<string> GetFirsts()
    {
        return new[] { "$" };
    }

    public override Token TryMatch(ParsingContext context, ISourceStream source)
    {
        if (source.PreviewChar != '$') return null;
        do
        {
            source.PreviewPosition++;
        } while (!source.EOF() && char.IsLetter(source.PreviewChar));

        var token = source.CreateToken(OutputTerminal);
        return token;
    }
}

I'm not really sure what OuputTerminal is though.. I guess it's some kind of dynamic property based on the current preview position? The way parsing is done in Irony is a little strange I think...

Anyway, the problem with this is what when I use this VariableTerminal, instead of how I was doing it before with "$" + IdentifierTerminal", when there's a syntax error, such as in this code:

p cat

The identifier terminal used to say

Syntax error, expected: { real string $ true false ...

But the variable gives me this error instead:

Invalid character: 'c'

The former error was more useful I think. I don't really understand why it's spitting out a different error...how can I get it to say that instead?

Answer 1

Not sure if this one might help:

http://irony.codeplex.com/discussions/70460

So, sharing it for the 2 lines:

  var identifier = new IdentifierTerminal("Identifier", IdFlags.NameIncludesPrefix);
  identifier.AddPrefix(Strings.AllLatinLetters, IdFlags.None);   //[a-zA-Z]([a-zA-Z0-9])

I think you won't be using them in the same way exactly, but maybe something similar.

Answer 2

I don't know wich version of Irony you use, but with the current version I was able to get that working using AllFirstChars:

        var localVariable = new IdentifierTerminal(NodeType.LocalVariable);
        localVariable.AllFirstChars = "$";

Hope this helps

Answer 3

var identifier = new IdentifierTerminal("identifier", IdFlags.NameIncludesPrefix);
identifier.AddPrefix("$", IdFlags.None);

should do the trick.

Answer 4

I agree with Jan that this should be handled in the scanner, not in the parser.

Does including '$' in extraFirstChars do what you want?

public IdentifierTerminal(string name, string extraChars, string extraFirstChars)

Answer 5

for me it looks clear that what you want is currently not supported (checked in the sources). See the discussion on the pascal character (the very botoom) as well which is identified as '#number' not allowing space between.

To go with non-terminal is not a way I believe. Grammars work by nature that you can have whitespaces between tokens. So what you really need is to follow advice given on the project wiki - section Custom Terminals on the bottom of the page and extend the Terminal class to fit your needs.

Or the easiest option would be to introduce flag which can make the prefix mandatory. Extending the IdentifierTerminal class and overriding TryMatch method.

If you look on this method in CompoundTerminalBase class what the TryMatch method does is basically:

1. ReadPrefix (but more less ignore if the prefix was found or not)
2. ReadBody (fails if the body wasn't read)
3. ReadSuffix

The ReadPrefix method sets a details.Prefix flag if a prefix is found. So after calling ReadPrefix you may want to check your newly introduced flag for mandatory prefix and if it is set you can check if the details.Prefix flag is set as well, otherwise you emit an error.

Good luck :)