Re: Token.setText()

[Archie Cobbs <archie@whistle.com>]

Etienne GAGNON writes:
> > All my token classes are being generated with this method:
> > 
> >     public void setText(String text)
> >     {
> >         throw new RuntimeException("Cannot change TArrow text.");
> >     }
> > 
> 
> Variable text tokens will allow you to "setText". 
> 
> Fixed text tokens have a hardcoded text string. Obviously, the error
> message should be customized on a token basis.
> 
> You have just uncovered a SableCC bug. The message should be: "Cannot
> change Txxx text." where xxx is the name of a fixed text token.

Then I still have a problem.. here's what I'm doing to parse doubly
quoted strings:

[ grammar snippet ]

  Helpers

    valid_quoted_char = [ 0x20..0x7e ];

  States

    start,        // beginning of line
    inline,       // after initial command/keyword
    dquote;       // parsing double quote (uses customization)

  Tokens

    { inline->dquote } string_start = '"';
    { dquote } string_nl = '\n';
    { dquote } string_tab = '\t';
    { dquote } string_cr = '\r';
    { dquote } string_dquote = '\"';
    { dquote } string_backslash = '\\';
    { dquote } string_hex_char = '\x' hex_digit hex_digit;
    { dquote->inline } string_end = '"';
    { dquote } string_char = valid_quoted_char;

    ...

[ filter() snippet ]

  public class FullLexer extends Lexer {

    // The doubly-quoted string we're building
    private StringBuffer text;

    // The currently building token
    private Token newToken;

    public FullLexer(java.io.PushbackReader input) {
      super(input);
    }

    // Flag
    boolean done;

    protected void filter() {

      // Flag to indicate a complete token
      done = false;

      // Doubly quoted string decoding
      if (state.equals(State.DQUOTE)) {

	// Handle next thinglet in the string
	token.apply(new AnalysisAdapter() {

	  public void caseTStringStart(TStringStart node) {
	    Assert.that(newToken == null);
	    text = new StringBuffer();
	    newToken = node;
	  }
	  public void caseTStringEnd(TStringEnd node) {
	    newToken.setText(text.toString());
	    done = true;
	  }
	  public void caseTStringHexChar(TStringHexChar node) {
	    int val = 0;
	    try {
	      val = Integer.parseInt(token.getText().substring(2), 16);
	    } catch (NumberFormatException e) {
	      Assert.that(false);
	    }
	    text.append((char) val);
	  }
	  public void caseTStringNl(TStringNl node) {
	    text.append('\n');
	  }
	  public void caseTStringTab(TStringTab node) {
	    text.append('\t');
	  }
	  public void caseTStringCr(TStringCr node) {
	    text.append('\r');
	  }
	  public void caseTStringDquote(TStringDquote node) {
	    text.append('"');
	  }
	  public void caseTStringBackslash(TStringBackslash node) {
	    text.append('\\');
	  }
	  public void caseTStringChar(TStringChar node) {
	    text.append(token.getText());
	  }
	  public void caseDefault(Node node) {
	    Assert.that(false);
	  }
	});

      } else {
	// XXX this assertion is failing..why?
	//Assert.that(newToken == null);

	// Pass token along
	newToken = token;
	text = new StringBuffer(token.getText());
	done = true;
      }

      // Are we done and ready to return the full string token?
      if (done) {
	text = null;				// for good measure
	token = newToken;			// now let token pass
	newToken = null;
      } else {
	token = null;				// keep scanning
      }
    }
  }

The problem is that string_start is a "fixed text" token in the
grammar, but in reality is used to represent the entire string.
I was following the example on page 39.

I agree that token.setText() should not be allowed, perhaps not
in any case. But there's no other way to attach information to
a token from a filter() method... this is the crux of the problem.

Is there a way to do this so the "newToken.setText(text.toString())"
will work, perhaps by creating a new, dummy token?

-Archie

___________________________________________________________________________
Archie Cobbs   *   Whistle Communications, Inc.  *   http://www.whistle.com