{:error, {:undefined, :standard_syntax_parser, ['syntax error before: ', []]}}

Question

So I am trying to use yecc.

Terminals string string_delimeter.

Nonterminals value string_content.

Rootsymbol value.

value -> string : extract_value('$1').

value -> string_delimeter string_content string_delimeter : '$2'.
string_content -> value string_content : ['$1' | '$2'].
string_content -> value : '$1'.

Erlang code.

extract_value({_, Value}) -> Value.

For the input:

[string_delimeter: '\'', string: 'test', string_delimeter: '\'']

I get

{:error, {:undefined, :standard_syntax_parser, ['syntax error before: ', []]}}

If I delete any (left or right) string_delimeter surrounding string_content:

value -> string_delimeter string_content string_delimeter : '$2'.

value -> string_delimeter string_content : '$2'.

for

[string_delimeter: '\'', string: 'test']

it returns

{:ok, 'test'}

I don't really understand this behaviour, where is the issue?

Answer 1

I did not figure why this is a problem in yecc: token_1 token_2 token_1 I really wish I would know, but I have a way around this:

Terminals string string_delimeter whitespace.


Nonterminals value string_content.


Rootsymbol value.

value -> string : extract_value('$1').

value -> string_delimeter string_content : '$2'.
string_content -> string_content string_delimeter : '$1'.
string_content -> value whitespace string_content : ['$1' | '$3'].
string_content -> value : ['$1'].

Erlang code.

extract_value({_, Value}) -> Value.

Answer 2

For the input:

[string_delimeter: '\'', string: 'test', string_delimeter: '\'']

I think you are confused (or maybe it's me!). yecc expects a list of tokens, where a token is a 2 or 3 element tuple. From the yecc docs:

The user should implement a scanner that segments the input text, and turns it into one or more lists of tokens. Each token should be a tuple containing information about syntactic category, position in the text (e.g. line number), and the actual terminal symbol found in the text: {Category, LineNumber, Symbol}.

If a terminal symbol is the only member of a category, and the symbol name is identical to the category name, the token format may be {Symbol, LineNumber}...

Here's an example of what yecc expects:

[
     {'[',1},
     {atom,1,foo},
     {',',1},
     {'[',1},
     {int,1,1},
     {']',1},
     {',',1},
     {'[',1},
     {atom,1,bar},
     {',',1},
     {'[',1},
     {int,1,2},
     {',',1},
     {int,1,3}, 
     {']',1},
     {']',1},
     {']',1}
]

You can implement such a scanner with leex, then feed the output to your yecc parser.

A word of advice: you should never post a question describing how you ran your code--it's just a big waste of time. Instead copy and paste the exact commands you ran as well as the output those commands produced. All you have to say is:

1. This is what I tried:

[Your code here]

2. Here is the output:

[all commands you ran and the output here]

3. This is the output I expect/want, or Wtf??!! is going on.

[expected/desired output here]

Answer 3

Ah, okay. You are using elixir even though the tags on your question say that you are using erlang. I can get a simpler version of your parser to work:

string_parser.yrl:

Nonterminals the_string content.
Terminals '\'' string.
Rootsymbol the_string.

the_string -> '\'' content '\'' : '$2'.

%I guess the atom :string has to be the first element of the tuple
%returned by '$2' in previous line:
content -> string : extract_value('$1')

Erlang code.

extract_value({_, _, Value}) -> Value.

In iex:

iex(1)> :yecc.file('string_parser.yrl')  
{:ok, 'string_parser.erl'}

iex(2)> c("string_parser.erl")
[:string_parser]

iex(3)> :string_parser.parse([{:"'", 1}, {:string, 1, "hello"}, {:"'", 1}])
{:ok, "hello"}

But, I cannot get a recursive definition to work.

Okay, I'm getting closer:

string_parser.yrl:

Nonterminals string interior_strings interior_string.
Terminals left_delim right_delim result.
Rootsymbol string.

string -> left_delim interior_strings right_delim : '$2'.
string -> left_delim right_delim : "".

interior_strings -> interior_string : ['$1'].
interior_strings -> interior_string interior_strings : ['$1' | '$2'].

interior_string -> result : extract_value('$1').
interior_string -> string : '$1'.

Erlang code.

extract_value({_, _, Value}) -> Value.

In iex:

iex(49)> :yecc.file('string_parser.yrl')
{:ok, 'string_parser.erl'}

iex(50)> c("string_parser.erl")
[:string_parser]    

iex(51)> :string_parser.parse([{:left_delim, 1}, {:result, 1, "hello"}, {:left_delim, 1}, {:result, 1, "goodbye"}, {:right_delim, 1}, {:right_delim, 1}])
{:ok, ["hello", ["goodbye"]]}

iex(53)> 

At this point, I don't know why the blasted brackets are around "goodbye".

Success!

string_parser.yrl:

Nonterminals string interior_strings interior_string.
Terminals left_delim right_delim result.
Rootsymbol string.

string -> left_delim interior_strings right_delim : '$2'.
string -> left_delim right_delim : "".

interior_strings -> left_delim interior_string right_delim: ['$2'].
interior_strings -> interior_string interior_strings : ['$1' | '$2'].

interior_string -> result : extract_value('$1').
interior_string -> string : '$1'.

Erlang code.

extract_value({_, _, Value}) -> Value.

In iex:

iex(53)> :yecc.file('string_parser.yrl') 
{:ok, 'string_parser.erl'}  

iex(54)> c("string_parser.erl")
[:string_parser] 

iex(55)> :string_parser.parse([{:left_delim, 1}, {:result, 1, "hello"}, {:left_delim, 1}, {:result, 1, "goodbye"}, {:right_delim, 1}, {:right_delim, 1}])
{:ok, ["hello", "goodbye"]}

The one thing I still can't get to work is if I explicitly specify '\'' for the delimiter:

Nonterminals string interior_strings interior_string.
Terminals '\'' result.
Rootsymbol string.
Endsymbol '$end'.

string -> '\'' interior_strings '\'' : '$2'.
string -> '\'' '\'' : "".

interior_strings -> '\'' interior_string '\'': ['$2'].
interior_strings -> interior_string interior_strings : ['$1' | '$2'].

interior_string -> result : extract_value('$1').
interior_string -> string : '$1'.

Erlang code.

extract_value({_, _, Value}) -> Value.

In iex:

iex(3)> :string_parser.parse([{:"'", 1}, {:result, 1, "hello"}, 
{:"'", 1}, {:result, 1, "goodbye"}, {:"'", 1}, {:"'", 1}, {:"$end", 1}])  

{:error, {1, :string_parser, ['syntax error before: ', []]}}

It's too frustrating to work with such horrible error messages. There's a syntax error before an empty list []??! Where is the empty list in the list of tokens?

I thought that adding a last tuple: {:"$end", 1} might work -- but no luck. Same error.

I can get the string_parser to work if I use literal brackets as the delimiters:

string_parser.yrl:

Nonterminals string interior_strings interior_string.
Terminals '[' ']' content.
Rootsymbol string.

string -> '[' interior_strings ']' : '$2'.
string -> '[' ']' : "".

interior_strings -> '[' interior_string ']' : ['$2'].
interior_strings -> interior_string interior_strings : ['$1' | '$2'].

interior_string -> content : extract_content('$1').
interior_string -> string : '$1'.

Erlang code.

extract_content({_, _, Content}) -> Content.

In iex:

iex(11)> :yecc.file('string_parser.yrl')
{:ok, 'string_parser.erl'}  

iex(12)> c("string_parser.erl") 
[:string_parser]    

iex(13)> :string_parser.parse([{:"[", 1}, {:content, 1, "hello"}, {:"[", 1}, {:content, 1, "goodbye"}, {:"]", 1}, {:"]", 1}])

{:ok, ["hello", "goodbye"]}
iex(14)> 

I tried writing a string_parser in erlang, and I got the same error. The .yrl files are not erlang syntax, so whatever is parsing the .yrl files does not seem to be able to parse a single quote atom: '\''.