Re: [abc] oopsla outline

> * aim to have preliminary numbers for all
> entries of tables outlined below asap

Preliminary numbers, where I currently have them, below.

> 3. Experiments
>
> 3.1 General challenges
> Attempt to identify the main difficulties.
> Only one benchmark: safeenum on jhotdraw.
>
> Aspectj:
> naive (hash table)
> tmbenches/jhotdraw/naiveaspects

151453 milliseconds (naiveaspects)

> proper (weak refs)
> tmbenches/jhotdraw/aspects

100453 milliseconds (seminaive aspects)

> smart aspect (the gold standard for efficiency)_
> tmbenches/jhotdraw/handoptim

91441 milliseconds (aspects)
10340 milliseconds (itds)
8052 milliseconds (handoptim)

> J-Lo
> tmbenches/jhotdraw/jlo

I don't have numbers for this, since if I remember correctly it was too
slow to terminate in a reasonable time when I tried it.

>
> tracematch:
> without leakbusting [need switch for that!]
> tmbenches/jhotdraw/tracematches

You mean a switch that makes all [collectable] weakrefs into strongrefs?
That shouldn't be too hard to do, and would be very interesting to see!

> with leakbusting but without indexing
> tmbenches/jhotdraw/tracematches

272138 milliseconds

> without leakbusting but with indexing
> with leakbusting and with indexing
> tmbenches/jhotdraw/tracematches

Both of these are not ready to be benchmarked yet.

> pql:
> plain
> ??? NOT CHECKED IN!! ???
> with Julian's variation
> ??? NOT CHECKED IN!! ???

I'm not sure which one we consider the 'plain' version and which one the
'variation'. So far, I've only been benchmarking a query that
corresponds to what Julian checked in as safe_iterators2.query. For that,

2464342 milliseconds.

Note that this is an order of magnitude more than TMs. I'm not quite
sure why this set of experiments turned out so badly for PQL, possibly I
didn't supply it with a sufficient queue (or cardinal may have been busy
with other stuff at the time). I will, of course, re-run the benchmark.

Incidentally, pure java runtime is 7088 milliseconds.

>
> Conclusion: further experiments with AspectJ,
> tracematches (4 versions)
> and PQL

For the numbers for AspectJ variations that I have, see above.

>
> 3.2 Further experiments
>
> for each (pattern, base program) combination runtime numbers for 7
> systems:
> no observer
> aspectj,
> tms (with indexing and leakbusting on and off),
> pql where available
> also #lines for each of these
> (or some better metric of complexity?)
> KSLOC for base program
>
>
> pattern base program pql? where
> ------- ------------ --- -----
> nulltracker certrevsim no tmbenches/nulltracker

Pure Java: 0.37s runtime.
tracematches: 25s runtime.
tracematches-nonegbindings: 20.5s runtime.

> hashcode aprove no tmbenches/aprove

Haven't set this up yet.

> observer ajhotdraw yes
> tmbenches/ajhotdraw/ajhotdraw-tm/ajhotdraw-0.3-src/src/aspects/observdrawchange

ajhotdraw-fresh: 83078ms
ajhotdraw-tm-naive: 250764ms
ajhotdraw-tm-opt: 160761ms
PQL: 186041 ms

>
> dbpooling artificial ?? tmbenches/dbpooling

AspectJ: 4.7s
TM: 5.2s
Base: 70s
PQL: Might work, but I haven't done it yet.

> locallocks jigsaw yes
> tmbenches/Jigsaw/{LockAspect.aj,ReducedThisJPLockTraceMatch.aj}
> lor jigsaw ?? ??

A whole host of results here:
aspect: 21875ms
redaspect: 20715ms

tracematch: 609023ms
redtracematch: 18741 (!! faster than redaspect)

thisjptracematch: 584464ms
redthisjptracematch: 22576ms

thistracematch: 1197856ms
redthistracematch: 22557ms

Note that these numbers aren't from the latest version of abc.

PQL: I've got PQL to process Jigsaw, and I've contacted Michael Martin
about a PQL bug I tried to fix. I wanted to get this email out before
trying to sort it out, however.

It seems there is no 'pure java' runtime. I will add a corresponding
benchmark to my scripts.

Note that some of the benchmarks also produce memory numbers (in
particular, [A]JHotDraw, Jigsaw and, to some extent, nulltracker), so
there's scope for further analysis and comparison -- which I'm loath to
do before we have the final benchmark run results, since the analysis
would have to be repeated for them when we get them.

All numbers apart from the large PQL number for safe_iterators are the
average over three runs, where deviation was so small that discarding
top and bottom value seemed pointless.

- P
Received on Thu Mar 02 09:59:40 2006