abc: The AspectBench Compiler for AspectJ



Static ahead-of-time evaluation of tracematches, FSE 2008

You may click some of the items on the following overview to get additional information about the respective analysis step. Compilation and analysis phases

Acknowledgements

We owe thanks to Manu Sridharan for help with his demand-driven points-to analysis. Stephen Fink provided valuable information about instance keys and SSA form. Brian Demsky and Nomair Naeem provided useful comments on an earlier version of this paper. We also thank the entire abc group for their continuing support. We would like to thank the anonymous referees for their detailed comments.

Download of all benchmark files

Download the complete benchmark package here (96MB). It contains:

View tracematches

You can obtain the tracematches we used as benchmarks from SVN, using:
svn co https://svn.sable.mcgill.ca/sable/bench/tracematches/generic-patterns/tags/version-1.5
or you can view them directly here.

Raw benchmark results

In this document we provide the raw benchmark data (Excel 2007, Excel 95-2003).

Potential points of failure

This directory contains *.PFG files for every benchmark that had potential points of failure remaining after analysis. The program points are ranked, as described in our paper. We marked actual points of failure with ACTUAL. For instance bloat-HasNext shows an actual match at position 1. Our filter algorithm would filter out all potential points of failure that have one of the following features attached: CALL, ANALYSIS_ABORTED, DELEGATE, NO_CONTEXT.

Visualization of potential and actual points of failure

We provide visualuzations of potential points of failure for the following benchmark/tracematch combinations. We do not provide a visualization of the "eclipse" benchmark because its source code was is packaged in a very unconventional way. We also do not provide a visualization of the "jython" benchmark because its authors compiled essential parts into bytecode without debug information. This provides us to extract line numbers, which is crucial to our current visualization approach. An IDE-based visualization could just jump to the method containing the respective shadow instead.

Please note that our visualization tool is meant to be a prototype. In the long term, we plan proper integration with the Soot Eclipse plugin, including clickable gutter annotations like in the AspectJ Development Tools and a sorted and filterable list view of all potential points of failure. Programmers will then be able to inspect these program points directly within the Eclipse IDE and mark uninteresting potential points of failure so that they will be ignored on future analysis runs until a related source file changes. Furthermore, the manual marking could be fed back to our machine learning algorithm to improve the ranking on the fly, as Kremenek et al. suggested at FSE 2004.

Actual points of failure

In this PDF document we listed some of the program points matched by our static analysis and explain why they may be of interest to the programmer. In particular we show the actual defects that wwe found with our tool.

Weka inputs and output

This file shows the input file that we gave to Weka. It was directly distilled from the *.PFG files. Weka generated the following output (full output here):

CALL = 0
|   ANALYSIS_ABORTED = 0
|   |   DELEGATE = 0
|   |   |   NO_CONTEXT = 0: ACTUAL (11.0/1.0)	% classification as true positive correct for 11 cases, wrong for 1 case
|   |   |   NO_CONTEXT = 1: FP (4.0/1.0)	% classification as false positive correct for 4 cases, wrong for 1 case
|   |   DELEGATE = 1: FP (10.0)			% classification as false positive correct for (all) 10 cases
|   ANALYSIS_ABORTED = 1: FP (30.0)		% classification as false positive correct for (all) 30 cases
CALL = 1: FP (406.0/1.0)			% classification as false positive correct for 406 cases, wrong for 1 case
You may also want to read Jörg Tiedemann's manual on interpreting Weka output.

Valid XHTML 1.1!