I'm using ocamllex to write a lexer for a scripting language but I am facing a conflict with my rule for comments.
I want to allow my command arguments to be unquoted as long as they only contain alphanumeric characters and slashes "/". For example:
echo "quoted argument !@#%" /this/second/argument/is/unquoted
Additionally, one of my prerequesites is C++ style comments with "//"
//this is a comment
echo hello world
The problem this brings is things like
echo foo//comment
I would like my lexer to produce a "foo" token, while also leaving the "//" untouched, to be consumed in the next time I ask the lexer for token. Is that possible? The reason for this is that its possible that the input buffer might not have reached the end of the comment yet and I'd rather immediately return the "foo" token than needlessly block trying to eagerly consume the comment.
The following is a small lexer which only matches
echo
, quoted and unquoted strings, comments, and prints out the resulting tokens:It's a trick that I used in a project of mine, to overcome that same limitation in ocamllex (compared to the original C lex program which let one match patterns in "look ahead mode"). Basically, it splits the ambiguous rules in their distinct radicals, and switch the lexer to different parser accordingly. It also keeps track of the currently used parser ad the next entry point.
In your situation, the only states it needs to keep track of are the default one (
CODE
), and comment mode (LINE_COMMENT
). This could be expanded to support other states if needed.