Extends my language at runtime

Question

I am developing a new language and I am following a traditional approach: so there is a lexer, a parser, an ast and a virtual machine that executes bytecode.

When the parser encounters an operator like + then it produces an ADD virtual machine instruction and everything works as expected so far.

I would be interested in give a way to end user to extends the language, so for example the user could add an operator++ function in order to create its own post increment operator. I am not sure about the right way to proceed because the parser cannot know in advance all the operators the user could add so I am asking about what is the right approach to follow.

Answer 1

You could do it the Swift way: have operator declarations in your grammar

In Swift it looks like:

infix operator SomeToken { associativity left precedence 150 }

This tells the Swift parser that SomeToken will be a left-associative, infix operator, and that in any precedence-based decision, this operator's weight is 150. other operators will have higher, lower, or equal precedence metrics, and that will guide the parser in creating the AST

Answer 2

For example in C# the list of operators that can be overloaded is known upfront and the overloading is always indicated by using a special keyword operator so it is known already to the parser that user is "adding" an operator.
See http://msdn.microsoft.com/en-us/library/aa691324(v=vs.71).aspx

For example in Prolog new operators and their evaluation priorities can be added to the language on-the-fly using the op(Precedence,Type,Name) directive. The Prolog parser reads and evaluates the input line by line. Each "line" is terminated with .
See http://www.learnprolognow.org/lpnpage.php?pagetype=html&pageid=lpn-htmlse40

Answer 3

One way to make your language infinitely extensible is to allow a full, reflexive metaprogramming, like in Common Lisp or Forth. This way user will be able to build any number of additional transforms on top of the existing core language, reusing everything already available in the language, including all the previous extensions, to implement such transforms.

If you want to be able to extend syntax, not just semantics, you may want to consider getting rid of your lexer and use a powerful lexerless parsing approach instead, namely, PEGs. Some languages are already doing this (Katahdin, pfront, CLike).