Re: [abc] oopsla paper

gc() is in fact called before *every* measurement, including the first
one -- so I don't know why it's so much higher than the rest, no obvious
explanation comes to mind. The first measurement is done before the
actual iteration has even started.

I'll remove the title, ok. Units on the Y axis are bytes, I may look
into making the numbers a bit smaller by turning them into Kbytes. X
axis ticks correspond to 5 second intervals, I thought that was clear
from the text.. I can certainly add the labels.

- P

Prof. Laurie HENDREN wrote:

>I think the graph where gc() is called before each measurement is
>good. However, gc() should be called before the first measurment
>as well (i.e. why is the first measurement larger?).
>
>For the graph, I would remove the title, it should be in the latex
>figure. The y and x axes need to be labeled with both the numbers
>and units.
>
>Cheers, Laurie
>
>
>+-------------------------------------------------------------+
>| Laurie Hendren, Professor, School of Computer Science |
>| McGill University |
>| 318 McConnell Engineering Building tel: (514) 398-7391 |
>| 3480 University Street fax: (514) 398-3883 |
>| Montreal, Quebec H3A 2A7 hendren@cs.mcgill.ca |
>| CANADA http://www.sable.mcgill.ca/~hendren |
>| http://wwww.sable.mcgill.ca http://aspectbench.org |
>+-------------------------------------------------------------+
>
>On Thu, 11 Aug 2005, Pavel Avgustinov wrote:
>
>
>
>>I have now checked in the numbers into CVS, as well as (most of) my
>>changes to the abc code -- the optimisations could do with a large
>>amount of cleaning up, most of my time was spent trying different things
>>rather than beautifying code. ;-)
>>
>>The last thing that remains to do is change figure 6, which doesn't
>>reflect the current reality. Please take a look at
>>http://musketeer.comlab.ox.ac.uk/~pavel/memusage.pdf -- there are two
>>graphs of memory usage there, taken at 5 second intervals. The
>>difference is that for the top graph, I called System.gc() before
>>reporting memory usage every five seconds, while for the bottom graph no
>>such call was made. Since System.gc() tends to collect all junk objects,
>>the top curve is rather smoother. The bottom one looks more like what we
>>have currently in the paper. Which one should I include? The text seems
>>to suggest that we call the GC before each measurement, which would
>>indicate the top curve. The much more regular memory behaviour could be
>>explained with the optimisations I've introduced -- I have eliminated
>>99.95% of disjunct object allocations that were present when the current
>>figure 6 was generated.
>>
>>Also comments on the style of the graphs are welcome -- are jagged lines
>>better than smooth ones, say, or should I use a different colour.
>>
>>Please comment on these issues tonight or tomorrow morning, as I'd like
>>to put the final version of the figure into the paper around lunch-time
>>and submit in the early afternoon.
>>
>>- P
>>
>>Prof. Laurie HENDREN wrote:
>>
>>
>>
>>>Those numbers look fine.
>>>
>>>Laurie
>>>
>>>
>>>+-------------------------------------------------------------+
>>>| Laurie Hendren, Professor, School of Computer Science |
>>>| McGill University |
>>>| 318 McConnell Engineering Building tel: (514) 398-7391 |
>>>| 3480 University Street fax: (514) 398-3883 |
>>>| Montreal, Quebec H3A 2A7 hendren@cs.mcgill.ca |
>>>| CANADA http://www.sable.mcgill.ca/~hendren |
>>>| http://wwww.sable.mcgill.ca http://aspectbench.org |
>>>+-------------------------------------------------------------+
>>>
>>>On Thu, 11 Aug 2005, Pavel Avgustinov wrote:
>>>
>>>
>>>
>>>
>>>
>>>>Here the numbers that will go into the final paper:
>>>>
>>>>- Pure java: 28.8s, 1,092K
>>>>- Naive aspect: 49.3s, 78,682K (I'm not sure why it suddenly takes more
>>>>memory... I can only blame it on the JVM -- this is Sun's Java5 rather
>>>>than Hotspot)
>>>>- Weakref aspect: 36.8s, 1,413K
>>>>- Tracematches: 275.2s, 1,103K
>>>>
>>>>I'm about to run the tracematch version, dumping memory usage every 5
>>>>seconds (for figure 6).
>>>>
>>>>- P
>>>>
>>>>Pavel Avgustinov wrote:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>After prolonged investigation and discussion with Ganesh on IRC (check
>>>>>logs for details), we decided the following:
>>>>>
>>>>>- The jumps between 4s and 28s are caused by java and the X server
>>>>>running at the same priority, so that it is essentially random which
>>>>>gets more CPU time -- and those two results correspond to the two cases.
>>>>>- We do not want to measure the times for a 'minimised animation', since
>>>>>Oege was rather clear he wanted "real world measurements" -- we were of
>>>>>the opinion that a pure CPU time measurement would be unfair.
>>>>>- We checked the priority the X server was running at on Oege's machine
>>>>>where (we assume) the original measurements were made. Setting my own X
>>>>>server to that priority eliminates the jumps (the result is
>>>>>deterministically 28s).
>>>>>
>>>>>In view of all this, I am now re-running all benchmarks with my own X
>>>>>server set to Oege's priority. When I have these numbers, I will post
>>>>>them here, and then run the tracematch version while collecting the
>>>>>memory data needed for figure 6. Then I'll rewrite any text that needs
>>>>>rewriting, and hopefully will have a final version of the paper in CVS
>>>>>by later tonight.
>>>>>
>>>>>- P
>>>>>
>>>>>Pavel Avgustinov wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>While I'm still rather at a loss for what to do, here the running times
>>>>>>for the benchmark on my laptop:
>>>>>>
>>>>>>JHotDraw with minimised animation window:
>>>>>>- Pure java: 5s
>>>>>>- Naive aspects: 22s
>>>>>>- Seminaive aspects: 11.2s
>>>>>>- Non-naive aspects: 11.1s
>>>>>>- Tracematches: between 220 and 256 s.
>>>>>>
>>>>>>Clearly it would be better to quote these numbers rather than the ones I
>>>>>>obtained on my desktop, since we "only" have a slowdown of about 50...
>>>>>>but the current numbers in the paper show a slowdown of less than 30.
>>>>>>
>>>>>>What should we do? Laurie? Oege?
>>>>>>
>>>>>>- P
>>>>>>
>>>>>>Pavel Avgustinov wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>Pavel Avgustinov wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>I plan to call it a day at some point this afternoon and replace the
>>>>>>>>numbers by newly generated ones on my own machine. I'll check the
>>>>>>>>section for any re-wording that needs to be done. There should be some
>>>>>>>>improvement, though nothing dramatic -- we may be 15 times slower
>>>>>>>>instead of 20, that order of magnitude.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>Slight problem with that...
>>>>>>>
>>>>>>>When trying to measure how long the pure-java version of the benchmark
>>>>>>>took, I was surprised to find it fluctuating, seemingly randomly, betwee
>>>>>>>4.4 seconds and 28.2 seconds -- always these two values, with a margin
>>>>>>>of error of 300ms or so. Investigation revealed that the running time
>>>>>>>depended on whether the X server or the JVM was prioritised by the OS --
>>>>>>>if the JVM got more CPU time, it finished in 4.4 seconds, if the X
>>>>>>>server got more it was 28.2.
>>>>>>>
>>>>>>>I tried the following experiment: I minimised the MDI child window
>>>>>>>containing the animation before selecting Animation->Start from the
>>>>>>>menu. The results are somewhat catastrophic... The pure java version
>>>>>>>finishes in 512-520 ms, reproducible. Naive aspects runs in 13.5 ms and
>>>>>>>accumulates a huge heap. Seminaive and non-naive aspects run in 7.2-7.4
>>>>>>>seconds, with the expected slight upwards trend in memory usage for
>>>>>>>seminaive and no such trend for non-naive aspects.
>>>>>>>
>>>>>>>Tracematches, on a minimised animation window, took me 194 seconds.
>>>>>>>
>>>>>>>We thus have a slowdown of almost a factor 400, when ignoring the X
>>>>>>>server overheads for the animation.
>>>>>>>
>>>>>>>I'm a bit at a loss for what to do. I can't seem to get reliable numbers
>>>>>>>for the pure java version with a non-minimised animation window, but the
>>>>>>>numbers with minimised windows seem to destroy our 'happy story'. We
>>>>>>>could leave the numbers as they are and investigate performance further
>>>>>>>until OOPSLA... or try to obtain some sort of numbers with non-minimised
>>>>>>>animation windows, these would be reasonably close to the ones in the
>>>>>>>paper..
>>>>>>>
>>>>>>>Here a summary of the timings I've taken:
>>>>>>>
>>>>>>>JHotDraw, minimising MDI animation window before starting animation:
>>>>>>>- pure Java: 0.5 s
>>>>>>>- naive aspects: 13.5 s
>>>>>>>- seminaive aspects: 7.3 s
>>>>>>>- non-naive aspects: 7.3 s
>>>>>>>- tracematches: 194 s
>>>>>>>
>>>>>>>JHotDraw, without minimising MDI animation window before starting
>>>>>>>animation, keeping it visible at all times:
>>>>>>>- pure Java: Fluctuates between 4.4 s and 28.2 s (discreetly, not
>>>>>>>continuously)
>>>>>>>- tracematches: 300-315 s.
>>>>>>>
>>>>>>>Any suggestions on a course of action are very welcome. If you can run
>>>>>>>the benchmark yourself, it might be helpful to compare results.
>>>>>>>
>>>>>>>- P
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>
>
>
>
>
>
Received on Fri Aug 12 00:12:35 2005