Custom lexer in Python prompt_toolkit

Question

While the prompt_toolkit documentation indicates that "it is also possible to create a custom lexer by implementing the Lexer abstract base class" (1), it doens't really explain how. I couldn't find any tutorial or really complete code example to illustrate how it works. Does anyone have any suggestion or example? I'm looking to creating a lexer from an application-specific derivative of SQL.

Answer 1

You can use pygments to create your own lexer, for creating your own lexer you need to know regex https://docs.python.org/3/howto/regex.html

You can read this documentation link from pygments on creating your own lexer. https://pygments.org/docs/lexerdevelopment/

heres a little example

from pygments.lexer import RegexLexer
from pygments.token import Name

__all__ = ['CustomLexer']

class CustomLexer(RegexLexer):
     name = "sql type"
     tokens = {
        'root': [
            (r"\bSELECT\b", Name.Tag),
        ]}

and add this line into the file you want to import it in

lexer = pygments.lexers.load_lexer_from_file("name.py", lexername="CustomLexer")

Now in prompt_toolkit you also have the feature to edit the coloring of these docs link- https://python-prompt-toolkit.readthedocs.io/en/master/pages/printing_text.html#pygments-token-text-tuples

style = Style.from_dict({
    'pygments.name.tag': 'bg:#ffffff #000000',
})