[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
overriding the filter method
I understand a little about how to override the filter method. Combined
with Etienne's suggestion below, what should the filter method do when it
discovers that an id is really a keyword? I'm not changing the lexer's
state, and I assume that the token value remains unchanged. Do I set
keyword1 = token; id = null; ?
Thanks for your help,
Steve Kelem
"Etienne M. Gagnon" wrote:
> See below.
>
> Steve Kelem wrote:
> >
> > 1. I thought of that solution. (Creating helpers for each possible
> > letter capitalization, and defining each case-insensitive token as a
> > series of these helpers.) Doesn't that result in a (very) large state
> > machine for the lexer? I tried that in a grammar for VHDL, which has
> > around 100 keywords, and the resulting state machine was enormous.
> >
> > What I think would be "simpler" would be something like the following,
> > which requires a very simple state machine for recognizing the tokens,
> > but requires a method such as the following that belongs to a base
> > class common to the "id" token and all the keyword tokens.
> >
> > public class Token {
> > String lc_id_tok = token.toString().toLowerCase(); // Convert
> > to lower case.
> > if (kewords.containsKey(lc_id_tok)) {
> > // If the token is one of the keywords, return the token type
> > of the keyword.
> > return keywords.get_Token_Type();
> > }
> > else { // It's an identifier. Return the token code for the
> > identifier.
> > return (TId)token;
> > }
> > }
>
> If you really want this, just use a customized lexer:
> - override the filter() and do your lookup
> - you also need to add empty tokens in the grammar for keywords:
>
> Tokens
> ...
> id = ... /* your "grab all ids and keywords" reg. exp. */ ... ;
> keyword1 = ;
> keyword2 = ;
> etc.
>
> Have fun!
>
> Etienne
> --
> ----------------------------------------------------------------------
> Etienne M. Gagnon, M.Sc. e-mail: egagnon@j-meg.com
> Author of SableCC: http://www.sable.mcgill.ca/sablecc/
> ----------------------------------------------------------------------
begin:vcard
n:Kelem;Steve
tel;fax:408-399-8905
tel;work:408-335-2718
x-mozilla-html:FALSE
url:http://www.adaptivesilicon.com
org:Adaptive Silicon, Inc.
adr:;;985 University Ave., Suite 31;Los Gatos;CA;95032-7639;U.S.
version:2.1
email;internet:kelem@adaptivesilicon.com
title:Chief Scientist
fn:Steve Kelem
end:vcard