[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Token.setText()
Etienne GAGNON writes:
> > All my token classes are being generated with this method:
> >
> > public void setText(String text)
> > {
> > throw new RuntimeException("Cannot change TArrow text.");
> > }
> >
>
> Variable text tokens will allow you to "setText".
>
> Fixed text tokens have a hardcoded text string. Obviously, the error
> message should be customized on a token basis.
>
> You have just uncovered a SableCC bug. The message should be: "Cannot
> change Txxx text." where xxx is the name of a fixed text token.
Then I still have a problem.. here's what I'm doing to parse doubly
quoted strings:
[ grammar snippet ]
Helpers
valid_quoted_char = [ 0x20..0x7e ];
States
start, // beginning of line
inline, // after initial command/keyword
dquote; // parsing double quote (uses customization)
Tokens
{ inline->dquote } string_start = '"';
{ dquote } string_nl = '\n';
{ dquote } string_tab = '\t';
{ dquote } string_cr = '\r';
{ dquote } string_dquote = '\"';
{ dquote } string_backslash = '\\';
{ dquote } string_hex_char = '\x' hex_digit hex_digit;
{ dquote->inline } string_end = '"';
{ dquote } string_char = valid_quoted_char;
...
[ filter() snippet ]
public class FullLexer extends Lexer {
// The doubly-quoted string we're building
private StringBuffer text;
// The currently building token
private Token newToken;
public FullLexer(java.io.PushbackReader input) {
super(input);
}
// Flag
boolean done;
protected void filter() {
// Flag to indicate a complete token
done = false;
// Doubly quoted string decoding
if (state.equals(State.DQUOTE)) {
// Handle next thinglet in the string
token.apply(new AnalysisAdapter() {
public void caseTStringStart(TStringStart node) {
Assert.that(newToken == null);
text = new StringBuffer();
newToken = node;
}
public void caseTStringEnd(TStringEnd node) {
newToken.setText(text.toString());
done = true;
}
public void caseTStringHexChar(TStringHexChar node) {
int val = 0;
try {
val = Integer.parseInt(token.getText().substring(2), 16);
} catch (NumberFormatException e) {
Assert.that(false);
}
text.append((char) val);
}
public void caseTStringNl(TStringNl node) {
text.append('\n');
}
public void caseTStringTab(TStringTab node) {
text.append('\t');
}
public void caseTStringCr(TStringCr node) {
text.append('\r');
}
public void caseTStringDquote(TStringDquote node) {
text.append('"');
}
public void caseTStringBackslash(TStringBackslash node) {
text.append('\\');
}
public void caseTStringChar(TStringChar node) {
text.append(token.getText());
}
public void caseDefault(Node node) {
Assert.that(false);
}
});
} else {
// XXX this assertion is failing..why?
//Assert.that(newToken == null);
// Pass token along
newToken = token;
text = new StringBuffer(token.getText());
done = true;
}
// Are we done and ready to return the full string token?
if (done) {
text = null; // for good measure
token = newToken; // now let token pass
newToken = null;
} else {
token = null; // keep scanning
}
}
}
The problem is that string_start is a "fixed text" token in the
grammar, but in reality is used to represent the entire string.
I was following the example on page 39.
I agree that token.setText() should not be allowed, perhaps not
in any case. But there's no other way to attach information to
a token from a filter() method... this is the crux of the problem.
Is there a way to do this so the "newToken.setText(text.toString())"
will work, perhaps by creating a new, dummy token?
-Archie
___________________________________________________________________________
Archie Cobbs * Whistle Communications, Inc. * http://www.whistle.com