Intellij language plugin: syntax highlighting when editing using JFlex lexer

Question

I am developing an Intellij Vala plugin. I am stuck on making keyword highlighting work while editing the file.

When I open the file all keywords (so far just class and ref) are highlighted correctly. However, when I am typing class it does not get highlighted (only after I reopen the file). What is interesting, for ref everything works.

I managed to investigate that when typing each character, my lexer is started on a segment of text of only 4 characters long. As a result ref which is shorter than that is highlighted, however class is not. It is the same for strings: they are highlighted if they are shorter than 4 chars (for example "a"). I looked into the source code of Intellij SDK and the length of text that in analyzed every time document is changed is calculated using some kind of segments. I am not experienced in Intellij architecture enough though to understand it. Could someone help?

Sources for my plugin (nothing extraordinary so far) are here. I am using Intellij Community build 139.225.3

Answer 1

OK, I found the solution. When I added an Identifier token to my lexer, the highlighting started to work.

I am guessing that now with the Identifier token the lexer when sees alphanumeric characters sees now it is appending them to some kind of buffer to construct an Identifier and then when suddenly they form a keyword they get highlighted. Before that I think the lexer when encountered a bad characted it just forgot it and proceeded. Again, these are just my assumptins. Still don't know why exactly 4 characters where analyzed previously.