overriding the filter method

[Steve Kelem <kelem@adaptivesilicon.com>]

I understand a little about how to override the filter method.  Combined
with Etienne's suggestion below, what should the filter method do when it
discovers that an id is really a keyword? I'm not changing the lexer's
state, and I assume that the token value remains unchanged. Do I set
keyword1 = token; id = null; ?

Thanks for your help,
Steve Kelem

"Etienne M. Gagnon" wrote:

> See below.
>
> Steve Kelem wrote:
> >
> > 1. I thought of that solution. (Creating helpers for each possible
> > letter capitalization, and defining each case-insensitive token as a
> > series of these helpers.) Doesn't that result in a (very) large state
> > machine for the lexer? I tried that in a grammar for VHDL, which has
> > around 100 keywords, and the resulting state machine was enormous.
> >
> > What I think would be "simpler" would be something like the following,
> > which requires a very simple state machine for recognizing the tokens,
> > but requires a method such as the following that belongs to a base
> > class common to the "id" token and all the keyword tokens.
> >
> > public class Token {
> >     String lc_id_tok = token.toString().toLowerCase();    // Convert
> > to lower case.
> >     if (kewords.containsKey(lc_id_tok)) {
> >         // If the token is one of the keywords, return the token type
> > of the keyword.
> >         return keywords.get_Token_Type();
> >     }
> >     else {    // It's an identifier. Return the token code for the
> > identifier.
> >         return (TId)token;
> >     }
> > }
>
> If you really want this, just use a customized lexer:
> - override the filter() and do your lookup
> - you also need to add empty tokens in the grammar for keywords:
>
> Tokens
>  ...
>  id = ... /* your "grab all ids and keywords" reg. exp. */ ... ;
>  keyword1 = ;
>  keyword2 = ;
>  etc.
>
> Have fun!
>
> Etienne
> --
> ----------------------------------------------------------------------
> Etienne M. Gagnon, M.Sc.                     e-mail: egagnon@j-meg.com
> Author of SableCC:                 http://www.sable.mcgill.ca/sablecc/
> ----------------------------------------------------------------------
begin:vcard 
n:Kelem;Steve
tel;fax:408-399-8905
tel;work:408-335-2718
x-mozilla-html:FALSE
url:http://www.adaptivesilicon.com
org:Adaptive Silicon, Inc.
adr:;;985 University Ave., Suite 31;Los Gatos;CA;95032-7639;U.S.
version:2.1
email;internet:kelem@adaptivesilicon.com
title:Chief Scientist
fn:Steve Kelem
end:vcard