EBNF / parboiled: how to translate regexp into PEG?

Question

This is a question both specific to the parboiled parser framework, and to BNF/PEG in general.

Let's say I have the fairly simple regular expression

^\\s*([A-Za-z_][A-Za-z_0-9]*)\\s*=\\s*(\\S+)\\s*$

which represents the pseudo-EBNF of

<line>               ::= <ws>? <identifier> <ws>? '=' <nonwhitespace> <ws>?
<ws>                 ::= (' ' | '\t' | {other whitespace characters})+
<identifier>         ::= <identifier-head> <identifier-tail>
<identifier-head>    ::= <letter> | '_'    
<identifier-tail>    ::= (<letter> | <digit> | '_')*
<letter>             ::= ('A'..'Z') | ('a'..'z')
<digit>              ::= '0'..'9'
<nonwhitespace>      ::= ___________

How would you define nonwhitespace (one or more characters that aren't whitespace) in EBNF?

For those of you familiar with the Java parboiled library, how could you implement a rule that defines nonwhitespace?

Answer 1

You are stuck with the conventions of your lexical generator for specifying character ranges and operations on character ranges.

Many lexer generators accept hex values (something like 0x) to represent characters, so you might write:

 '0'..'9'
 0x30..\0x39

for digits.

For nonwhitespace, you need to know which character set you are using. For 7 bit ASCII, nonwhitespace is conceptually all the printing characters:

0x21..\0x7E

For ISO8859-1:

( 0x21..\0x7E | 0x80-0xFF )

You can decide for yourself if the character codes above 0x80 are spaces or not (is non-breaking space a space?). You also get to decide about the status of the control characters 0x0..0x1F. Is tab (0x9) a whitespace character? How about CR 0xD and LF 0xA? How about the ETB control character?

Unicode is harder, because its a huge set, and your list gets big and messy. C'est la vie. Our DMS Software Reengineering Toolkit is used to build parsers for a wide variety of languages, and has to support lexers for ASCII, ISO8859-z for lots of z's, and Unicode. Rather than write complicated "additive" regular expression ranges, DMS allows subtractive regular expressions, and so we can write:

 <UniCodeLegalCharacters>-<UniCodeWhiteSpace>

which is much easier to understand and gets it right on the first try.

Answer 2

In EBNF I would simply define nonwhitespace as any character that isn't whitespace:

nonwhitespace ::= anycharacter - whitespace

This requires that you have a 'anycharacter' literal that defines the entire range of possible symbols, and a clear definition of which characters are whitespace.

In Parboiled you can do this using the TestNot and ANY Rules, so for example nonwhitespace would be defined as any character which doesn't match the WhiteSpace() Rule:

Sequence( TestNot(WhiteSpace()) , ANY )