[abc] AOSD Paper Reviews (fwd)

---------- Forwarded message ----------
Date: Thu, 2 Dec 2004 22:43:03 -0500
From: Tarr <tarr@watson.ibm.com>
To: Oege.de.Moor@comlab.ox.ac.uk
Cc: tarr@watson.ibm.com
Subject: AOSD Paper Reviews

Thank you again for your submission to AOSD 2005. The reviews for your
submission entitled:

"abc: An Extensible AspectJ Compiler"

which was accepted for presentation at AOSD 2005, are included below.
The review process was extremely thorough, and the program committee as
a whole worked hard to provide you with this feedback. Please take this
feedback seriously and be sure to address the issues raised by the reviewers
when you revise your paper.

The final version of your paper is due on Monday, January 10, 2005. This
deadline is FIRM. Your paper must conform to the ACM SIG proceedings
publication format, available at
http://www.acm.org/sigs/pubs/proceed/template.html, and may be no longer than
twelve pages, including all text, references, appendices, and figures.
Overly long papers will not be included in the proceedings. Papers must be
printable on US 8.5 inch x 11 inch paper. Please submit both PDF and the
original text source for your final papers into the AOSD 2005 online submission
site, using the user name and password you were assigned when you originally
submitted your paper. Please don't hesitate to contact me if you have any
questions or problems.

Again, congratulations! I look forward to seeing the final version of your
paper, and to meeting you in Chicago.

Regards,

Peri Tarr
AOSD 2005 Program Chair
============================================================================
Review 1:

>>> Summary of the submission <<<

Abc is an AspectJ compiler that is intended to be an extensible framework for
experimenting with aspect-oriented language features. The base compiler
architecture is described, with emphasis on the Polyglot and Soot supporting
technologies. In order to demonstrate how abc can be built on four language
extensions are described.

>>> Evaluation <<<

The authors state that there is a need for an extensible AspectJ compiler,
substantiate this claim with several sources of researchers that would have
benefited, and demonstrate that their compiler is extensible. This work
addresses a genuine need of the AOP community, provides an implementation,
describes the architecture and design principles in a clear way (e.g.
proportionality), and effectively illustrates how the compiler can be extended.
Overall this is a high quality contribution.

The following problems should be addressed:
 - There is not enough validation to definitively claim that this is an
extensible compiler, and I think that this claim should be made more carefully.
It won’t be validated as extensible until more than one external group builds
something novel on top of it. Also, the language extension examples provided
are very limited in scope, to the point where they would not be that hard to
extend in ajc. However, this paper is convincing that abc architecture *is* a
much better platform for extensibility, and getting other groups to try it will
require the publication of this paper.

 - The paper is unclear about the tradeoffs of abc vs. ajc. This is problematic
because very few people are familiar enough with ajc to understand these
tradeoffs. There should be a precise description or table outlining differences
such as: architectural tradeoffs made by each, architectural limitations of each
(e.g. abc architecture is not suitable for IDE integration, limits of ajc
architecture imposed by requirement of separate compilation), compilation
performance tradeoffs (if these aren’t mention explicitly readers might get very
suspicious about the cost of all the intermediate formats), and target users
(AOSD researchers, early adopters of novel extensions, AOSD practitioners).

 - It’s surprising that there is no discussion of using abc to implement the key
next language extensions for AOP languages: generics and metadata support.

============================================================================
Review 2:

>>> Summary of the submission <<<

This paper presents an overview of abc, a research compiler that can
easily implement extended AOP language features. It discusses how abc
can be built by combining two existing libraries, namely Polyglot and
Soot, and how extended AOP language features can be implemented by
extending abc. The implementation costs are measured and compared by
counting the number of lines.

>>> Evaluation <<<

This is a well-organized, good paper. I enjoyed reading.

It would be fair if the paper wrote the drawbacks of the abc system.
As far as I tested, the compilation speed of abc is significantly
slower than ajc. Since the system looks to be designed for the ease
of extending, rather than the compilation speed, its compilation speed
would be somewhat inherent. Measurement and discussion on performance
would be useful to the system designers that need to investigate about
the trade-offs between extensibility and performance.

Minor comments:

- Section 2.2. "This representation greatly simplifies weaving of
  advice ... If it were weaving into bytecode directly, the weaver
  would need to consider the effect of the woven code on the implicit
  execution stack, ..."

  I agree with that the Soot's intermediate representation makes
  weaving simpler, but not sure if it makes that _greatly_ simpler.
  In most cases, the bytecode fragments inserted by aspects do not
  affect the operand stack.

- Section 5.1. "while the normal Java state is terminated by a `}'".
  It is not clear why the end of normal Java state is relevant to
  switching when parsing a Java expression inside if pointcut.

>>> Points in favour or against <<<

+ the system useful for research purposes.

+ the system is implemented and widely available.

+ the examples of extensions cover typical extension proposals.

- the design of the system is not surprising.

============================================================================
Review 3:

>>> Summary of the submission <<<

The paper introduces abc, the AspectBench compiler, a
publicly-available compiler development framework providing an
implementation of the AspectJ language and facilities for the
implementation of extensions of AspectJ's AOP model. The paper
describes the overall architecture of abc, which is built using the
Polyglot and Soot compiler frameworks, describes general issues
related to the extension of AO languages within such a compiler
development environment and presents several concrete extensions in
some detail.

>>> Evaluation <<<

I quite like this paper:

1. I subscribe to your point of view that AOP as a research domain can
   benefit much from a compiler-development environment which allows
   quite different AO languages to be implemented in form of a "real"
   compiler (as opposed, for instance, to a generic interpreter
   suitable only for rapid prototyping purposes).

2. You make a convincing case that abc constitutes such an environment:
   - The extension examples discussed in the paper are quite diverse.
   - Your arguments that Polyglot and Soot are appropriate building
     blocks for such an endeavor seem valid.

3. The evidence you give that the effort to implement an extension is
   proportional to its difference to standard AspectJ is quite
   compelling.

However, the paper can, in my opinion be improved by addressing the
following issues:

- Most importantly, you should give more evidence for your claim that
  ajc as a "monolithic" compiler makes extensions (a lot) more
  difficult. The discussion of the throw pointcut implementation
  (Fig. 5) seems rather biased to me: while your argument that abc
  better separates the extension from the core compiler is valid, you
  should comment on the fact that the overall effort in AspectJ
  (e.g. considering LOC) is smaller. More generally, a qualitative
  comparison (i.e., in terms of compiler architecture, abstractions
  provided and how the latter can be modified) would be most
  convincing. While a complete treatment of this issue could be rather
  lengthy to do, you should give at least some indications to this
  respect.

- Similarly, the discussion of the other related work
  could be made more pertinent. For example, you argue against
  Javassist by putting forward a lack of analysis framework. First,
  you do not (substantially) apply Soot's framework in the present
  paper. Second, I do not see why Javassist as a load-time tool could
  not, in principal, incorporate some (useful) analysis framework.

- As to presentation, Sec. 3 and 4 are redundant enough that it would
  be better to either
  - use one extension of 4 as a detailed running example in 3 and shorten the
    presentation in 4 or
  - present the general principals at the corresponding places in 4,
    thus suppressing 3.

- Re the extensibility requirements on page 2: you should mention that
  the analysis capability is not further elaborated in the paper apart
  from being supported by using Soot.

- The manipulation of grammar rules by the code in Sec. 5.2 should be
  explained.

>>> Points in favour or against <<<

+ Interesting compiler framework for the support of diverse AO
  languages;
+ Interesting use of Polyglot and Soot frameworks for AOP
+ Publicly-available tool
+ Useful discussion of proportionality w.r.t. aspect
  extensions.
+ Non-trivial aspect extensions presented in some detail.
- Case for improvement over existing approaches could be strengthened.
- Presentation partially overly redundant
Received on Fri Dec 3 08:12:40 2004