Re: [abc] oopsla paper

Those numbers look fine.

Laurie

+-------------------------------------------------------------+
| Laurie Hendren, Professor, School of Computer Science |
| McGill University |
| 318 McConnell Engineering Building tel: (514) 398-7391 |
| 3480 University Street fax: (514) 398-3883 |
| Montreal, Quebec H3A 2A7 hendren@cs.mcgill.ca |
| CANADA http://www.sable.mcgill.ca/~hendren |
| http://wwww.sable.mcgill.ca http://aspectbench.org |
+-------------------------------------------------------------+

On Thu, 11 Aug 2005, Pavel Avgustinov wrote:

> Here the numbers that will go into the final paper:
>
> - Pure java: 28.8s, 1,092K
> - Naive aspect: 49.3s, 78,682K (I'm not sure why it suddenly takes more
> memory... I can only blame it on the JVM -- this is Sun's Java5 rather
> than Hotspot)
> - Weakref aspect: 36.8s, 1,413K
> - Tracematches: 275.2s, 1,103K
>
> I'm about to run the tracematch version, dumping memory usage every 5
> seconds (for figure 6).
>
> - P
>
> Pavel Avgustinov wrote:
>
> >After prolonged investigation and discussion with Ganesh on IRC (check
> >logs for details), we decided the following:
> >
> >- The jumps between 4s and 28s are caused by java and the X server
> >running at the same priority, so that it is essentially random which
> >gets more CPU time -- and those two results correspond to the two cases.
> >- We do not want to measure the times for a 'minimised animation', since
> >Oege was rather clear he wanted "real world measurements" -- we were of
> >the opinion that a pure CPU time measurement would be unfair.
> >- We checked the priority the X server was running at on Oege's machine
> >where (we assume) the original measurements were made. Setting my own X
> >server to that priority eliminates the jumps (the result is
> >deterministically 28s).
> >
> >In view of all this, I am now re-running all benchmarks with my own X
> >server set to Oege's priority. When I have these numbers, I will post
> >them here, and then run the tracematch version while collecting the
> >memory data needed for figure 6. Then I'll rewrite any text that needs
> >rewriting, and hopefully will have a final version of the paper in CVS
> >by later tonight.
> >
> >- P
> >
> >Pavel Avgustinov wrote:
> >
> >
> >
> >>While I'm still rather at a loss for what to do, here the running times
> >>for the benchmark on my laptop:
> >>
> >>JHotDraw with minimised animation window:
> >>- Pure java: 5s
> >>- Naive aspects: 22s
> >>- Seminaive aspects: 11.2s
> >>- Non-naive aspects: 11.1s
> >>- Tracematches: between 220 and 256 s.
> >>
> >>Clearly it would be better to quote these numbers rather than the ones I
> >>obtained on my desktop, since we "only" have a slowdown of about 50...
> >>but the current numbers in the paper show a slowdown of less than 30.
> >>
> >>What should we do? Laurie? Oege?
> >>
> >>- P
> >>
> >>Pavel Avgustinov wrote:
> >>
> >>
> >>
> >>
> >>
> >>>Pavel Avgustinov wrote:
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>>I plan to call it a day at some point this afternoon and replace the
> >>>>numbers by newly generated ones on my own machine. I'll check the
> >>>>section for any re-wording that needs to be done. There should be some
> >>>>improvement, though nothing dramatic -- we may be 15 times slower
> >>>>instead of 20, that order of magnitude.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>Slight problem with that...
> >>>
> >>>When trying to measure how long the pure-java version of the benchmark
> >>>took, I was surprised to find it fluctuating, seemingly randomly, betwee
> >>>4.4 seconds and 28.2 seconds -- always these two values, with a margin
> >>>of error of 300ms or so. Investigation revealed that the running time
> >>>depended on whether the X server or the JVM was prioritised by the OS --
> >>>if the JVM got more CPU time, it finished in 4.4 seconds, if the X
> >>>server got more it was 28.2.
> >>>
> >>>I tried the following experiment: I minimised the MDI child window
> >>>containing the animation before selecting Animation->Start from the
> >>>menu. The results are somewhat catastrophic... The pure java version
> >>>finishes in 512-520 ms, reproducible. Naive aspects runs in 13.5 ms and
> >>>accumulates a huge heap. Seminaive and non-naive aspects run in 7.2-7.4
> >>>seconds, with the expected slight upwards trend in memory usage for
> >>>seminaive and no such trend for non-naive aspects.
> >>>
> >>>Tracematches, on a minimised animation window, took me 194 seconds.
> >>>
> >>>We thus have a slowdown of almost a factor 400, when ignoring the X
> >>>server overheads for the animation.
> >>>
> >>>I'm a bit at a loss for what to do. I can't seem to get reliable numbers
> >>>for the pure java version with a non-minimised animation window, but the
> >>>numbers with minimised windows seem to destroy our 'happy story'. We
> >>>could leave the numbers as they are and investigate performance further
> >>>until OOPSLA... or try to obtain some sort of numbers with non-minimised
> >>>animation windows, these would be reasonably close to the ones in the
> >>>paper..
> >>>
> >>>Here a summary of the timings I've taken:
> >>>
> >>>JHotDraw, minimising MDI animation window before starting animation:
> >>>- pure Java: 0.5 s
> >>>- naive aspects: 13.5 s
> >>>- seminaive aspects: 7.3 s
> >>>- non-naive aspects: 7.3 s
> >>>- tracematches: 194 s
> >>>
> >>>JHotDraw, without minimising MDI animation window before starting
> >>>animation, keeping it visible at all times:
> >>>- pure Java: Fluctuates between 4.4 s and 28.2 s (discreetly, not
> >>>continuously)
> >>>- tracematches: 300-315 s.
> >>>
> >>>Any suggestions on a course of action are very welcome. If you can run
> >>>the benchmark yourself, it might be helpful to compare results.
> >>>
> >>>- P
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> >>
> >>
> >>
> >>
> >>
> >
> >
> >
> >
> >
> >
>
>
>
>
Received on Thu Aug 11 22:30:48 2005