[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: New Lexer
Etienne Gagnon wrote:
> Philippe wrote:
> > Another question: Why do you use a pushback reader?
> >
> > I want to be sure that I do not need a pushback reader (because this will
> > be difficult to install on a network link).
>
> The pushback reader is used by the lexer to "unread" characters.
>
> Example:
>
> Tokens
> end = 'end';
> endif = 'endif';
>
> Now, feed this lexer the following string: "endimage"
>
> It is easy to see that the lexer won't know that it must return a "TEnd"
> token before reading the "m". When it does realize this, it must
> pushback the "m" and the "i". The next time, the lexer will be able to
> scan "image" (and not "age").
>
> I assume that your native lexer will take care of all these details. So,
> all you need to do is override the getToken() method. You shouldn't
> worry about anything else.
>
> e.g.
>
> public class SomeLexer extends Lexer
> {
> final static int BLAH = 0;
> final static int OTHER_BLAH = 1;
> ...
>
> // Constructor
> SomeLexer(...)
> {...}
>
> protected Token getToken()
> {
> native.scanNextToken(); // scan the next token
>
> switch(native.getNextTokenType())
> {
> case BLAH:
> return new TBlah(native.getLine(), native.getColumn(),
> native.getText());
> case OTHER_BLAH:
> return new TBlah(native.getLine(), native.getColumn(),
> native.getText());
> ...
> }
> throw new RuntimeException("Shouldn't reach this point;-)");
> }
> }
>
> You should be able to work the details out:-)
>
> Etienne
Thanks for this explanation.
Best regards
Philippe.