[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: help with lexer

Archie Cobbs wrote:
> However, one thing has always bothered me though about all parsers:
> why is EOF considered a special token?!? Why can't it just be
> another token like any other? Why does a parser always assume
> that the very last token will be EOF?

It has to do with the way a parser makes decisions, be it LL or LR.

If you had a grammar like this:

start = right_recursive eof; 
right_recursive = id right_recursive | id;

If you remove the 'eof', how does the parser know if it should parse the
following into two ASTs or one AST? (or, in the general case, if it
should return an error?)

input: id id

There is the possibility of saying: parse the longest string.

I will think about it.