Re: New Lexer

[Etienne Gagnon <gagnon@sable.mcgill.ca>]

Philippe wrote:
> Another question: Why do you use a pushback reader?
> 
> I want to be sure that I do not need a pushback reader (because this will
> be difficult to install on a network link).

The pushback reader is used by the lexer to "unread" characters.

Example:

Tokens
  end = 'end';
  endif = 'endif';

Now, feed this lexer the following string: "endimage"

It is easy to see that the lexer won't know that it must return a "TEnd"
token before reading the "m". When it does realize this, it must
pushback the "m" and the "i". The next time, the lexer will be able to
scan "image" (and not "age").

I assume that your native lexer will take care of all these details. So,
all you need to do is override the getToken() method. You shouldn't
worry about anything else.

e.g.

public class SomeLexer extends Lexer
{
  final static int BLAH = 0;
  final static int OTHER_BLAH = 1;
  ...

  // Constructor
  SomeLexer(...)
  {...}

  protected Token getToken()
  {
    native.scanNextToken(); // scan the next token

    switch(native.getNextTokenType())
    {
      case BLAH:
        return new TBlah(native.getLine(), native.getColumn(),
native.getText());
      case OTHER_BLAH:
        return new TBlah(native.getLine(), native.getColumn(),
native.getText());
      ...
    }
    throw new RuntimeException("Shouldn't reach this point;-)");
  }
}

You should be able to work the details out:-)

Etienne