Re: New Lexer

[Philippe <plegay@atos-group.com>]

Etienne Gagnon wrote:

> Philippe wrote:
> > Another question: Why do you use a pushback reader?
> >
> > I want to be sure that I do not need a pushback reader (because this will
> > be difficult to install on a network link).
>
> The pushback reader is used by the lexer to "unread" characters.
>
> Example:
>
> Tokens
>   end = 'end';
>   endif = 'endif';
>
> Now, feed this lexer the following string: "endimage"
>
> It is easy to see that the lexer won't know that it must return a "TEnd"
> token before reading the "m". When it does realize this, it must
> pushback the "m" and the "i". The next time, the lexer will be able to
> scan "image" (and not "age").
>
> I assume that your native lexer will take care of all these details. So,
> all you need to do is override the getToken() method. You shouldn't
> worry about anything else.
>
> e.g.
>
> public class SomeLexer extends Lexer
> {
>   final static int BLAH = 0;
>   final static int OTHER_BLAH = 1;
>   ...
>
>   // Constructor
>   SomeLexer(...)
>   {...}
>
>   protected Token getToken()
>   {
>     native.scanNextToken(); // scan the next token
>
>     switch(native.getNextTokenType())
>     {
>       case BLAH:
>         return new TBlah(native.getLine(), native.getColumn(),
> native.getText());
>       case OTHER_BLAH:
>         return new TBlah(native.getLine(), native.getColumn(),
> native.getText());
>       ...
>     }
>     throw new RuntimeException("Shouldn't reach this point;-)");
>   }
> }
>
> You should be able to work the details out:-)
>
> Etienne

Thanks for this explanation.

Best regards
Philippe.