Re: Handling an INCLUDE directive.

[willh@msoft.com (Will Hartung)]

> Will Hartung wrote:
> > I sort of hacked in the support by extending the generated Lexer class.
I
> > override both getToken and filter (though in reality, I probably don't
need
> > to override filter).
>
> You might have to override the filter method to catch the end-of-file of
> an included file.

Nah, if you looked at the code, you'll note that this is done it getToken.

> > This has a couple of problems as is, though. First, it doesn't capture
the
> > file information (though it does capture the line information). To
capture
> > this information I need to change SableCC directly, as the
LexerException
> > and ParseException don't have slots for something like filename. I was
> > debating on the best path for this.
>
> Right, we must also modify the exception classes.
>
> > Another thought is that the system could pass the actual token to the
> > exception constructor...
>
> This looks fine.  The token would be made available to the user.
>
> > Add to this an "extra info" slot in Node that is left null unless
> > the user sticks something into it.
>
> This one: I am not convinced.  It's too tempting to use this field to
> put all sort of things, instead of using hashtables to associate data
> with tokens.  I have argued against this approach both in my thesis and
> on the mailing-list.  My primary goal is good software engineering.

Ok, I don't necessarily want you to go into depth here, but I don't clearly
see the distinction between hanging an Object extra slot off of token and
having the user maintain a lookup in HashTable. Either way, having the Token
in the Exception would solve the problem. Either way, the filter method
would be used to maintain and collect the extra information.

Another thought is what if there was a way in the SableCC generator to
specify a Token (or perhaps Node) class that everything descends from. This
would let the user sub-class Token for their own purposes, and the generator
creates the rest of the heirarchy.

But I suppose if you're against extra information in the Token object,
you'll probably be opposed to this as well.

> Now, arguably, the "file" information could be included in all SableCC
> tokens, as it would be of interest to most SableCC generated compilers.
> So, I would agree to have an additional "file" field in Token.java.  I
> would also provide hooks in the Lexer class to retrieve and set the
> lexer internal "file,line,column" information.

Yeah, I guess we should extend the Lexer contrstuctor to pass it the file
name.

> > Another problem with my current system is that the lexer must "parse"
the
> > token stream looking for a properly constructed INCLUDE statement.
Hardly
> > overwhelming, but it seems kind of nutty to have "a parser in the
lexer".
>
> Did you really need to get down to the parser to detect the INCLUDE
> construct?  It's OK to cheat a little by recognizing the "INCLUDE" token
> in the lexer, then do some manual processing to get the file name that
> follows.  This way, the parser (and grammar) does not need to be aware
> of the inclusion.  In C, for example, you can detect "#include" then
> look for something of the form "..." or <>.

No, no...it's truly simple. While running through the token stream it looks
for the TInclude token, and once in that state, it starts looking for a
TStringLiteral, and finally a TSemicolon. If anything odd shows up, it just
dumps them all back to token stream as if nothing happened.

One key thing about the way this Include is done is that it's NOT a
pre-processor. The Includes are handled during the building of the parse
tree, rather than before. It doesn't scan the file twice (or more) searching
for Includes. Also, Includes can be nested to machine limits.

> > I'll look over at SourceForge and try and get the latest version to work
> > with.
>
> If you register as a SourceForge user, I can add you as a SableCC
> developer, so that you can help implementing support for file
> inclusion.  (Don't forget to send me your username and ID).

Sourceforge seems to operating intermittently right now...

Regards,

Will Hartung
willh@msoft.com