Ashes2 Home

Introduction

Getting Started

Exploring Repositories

Customizing Ashes2

Performing Benchmarking Operations

Using additional utilities

Sable Home
McGill

Ashes2 Tutorial

Last updated: September 9, 2003 		Sable

This tutorial aims at providing a quick overview of the Ashes2 benchmarking system, its most often used commands, and its basic configuration.



Getting Started

Note

This tutorial assumes that you have installed the full distribution of Ashes2, along with the example benchmarks. If you have installed a minimal distribution of Ashes2, you may want to install the set of micro benchmarks that can be downloaded separately if you want to execute the commands on your own system.

Note

This tutorial assumes that your prompt string is $. When entering commands, do not enter the $ sign that precedes them.

To test your installation, enter

Listing 1

$ ashes2 version

at your prompt. If your setup is correct, Ashes2 should respond with a short message similar to the following:

Listing 2

Ashes2 Benchmarking System version 0.1

If you did not get such a message, refer to the installation instructions to correctly install and configure Ashes2.

To get started with Ashes2, type:

Listing 3

$ ashes2 help

at the prompt. Ashes2 will respond with its help message, like so:

Listing 4

Ashes2 Benchmarking System

Usage: ashes2 [options] <command> [args]

Available commands:
   Admin commands:
      list
      version
   Benchmark execution commands:
      compile
      run
   Benchmark management commands:
      export
      refresh
   Help commands:
      help
   Misc commands:
      prompt
   Search commands:
      describe
      locate
      search
   Shell commands:
      cd
      exec
      java
      javac
      ls
      pwd
   Utility commands:
      repeat
      time

To get help for specific command, use 'ashes2 help <cmd>'


Author: Bruno Dufour (bruno.dufour@mail.mcgill.ca)
URL: http://www.sable.mcgill.ca/~bdufou1/ashes2

The help message lists all commands that Ashes2 will recognize.

Notice that commands are organized by categories to make it easier to find a particular command. To know more about a particular command, it is possible to query Ashes2 for more details. For example, to know more about the search command, enter the following:

Listing 5

$ ashes2 help search

Ashes2 will respond with the usage for the requested command:

Listing 6

   Command: search
      Description: 
         Searches the list of repositories for a pattern. 

      Usage: search <pattern>+
      where:
         pattern: a pattern to look for

In the next section, we will look at how to query Ashes2 for benchmarks.


Back to Top


Exploring Repositories

Ashes2 uses repositories to organize benchmarks. Each repository can contain any number of suites or benchmarks.

Each repository is simply a directory which holds suites and/or benchmarks. A list of repositories is simply a list of directory names separated by the path separator character (':' on Posix systems, ';' on Microsoft Windows systems). There are several ways to tell Ashes2 the location of repositories. One way is to set the 'ASHES2_REPOS' environment variable to contain a list of repositories to use. Another way is to pass this list with the '-r' (or the equivalent '--repos') option to Ashes2 when invoked. Another way is available through the Ashes2 configuration files. FIXME.

Ashes2 supports arbitrary nesting of suites. Each repository element (either a bechmark or suite) is assigned a fully qualified name (FQN) based on place in the hierarchy. Each level in the hierarchy is separated by a dot ('.'). For example, if the a repository contains a suite named 'micro', its FQN is simply 'micro' since it has no parent element. A benchmark named 'hello' found inside the suite 'micro' will have a FQN equal to 'micro.hello', and so on.

This section will take the suite named micro as an example. It is installed by default with the full Ashes2 distribution.

Let's start by searching for the pattern 'micro':

Listing 7

$ ashes2 search micro
Searching ...
[ Results for search key micro ]
[ 3 match(es) found ]

   Suite: micro
      Aliases: small, nano, easy
      Private: No
      Location: /home/bdufour/links/ashes2/benchmarks/micro
      URL: 
      Description: A collection of micro benchmarks

   Benchmark: micro.empty
      Aliases: 
      Private: No
      Configurations: default
      Suite: micro
      Location: /home/bdufour/links/ashes2/benchmarks/micro/empty
      Language: java
      URL: 
      Description: The empty program, which simply returns from main

   Benchmark: micro.hello
      Aliases: hello_world
      Private: No
      Configurations: default
      Suite: micro
      Location: /home/bdufour/links/ashes2/benchmarks/micro/hello
      Language: java
      URL: 
      Description: The famous 'Hello World' program

We can see that Ashes2 found 3 matches for the pattern 'micro'. One of them is the suite named micro, the other two are the benchmarks it contains.

Ashes2 supports wildcards in the search pattern. For example, if we wanted to list the contents of the suite 'micro', we could use:

Listing 8

$ ashes2 search micro.*
Searching ...
[ Results for search key micro.* ]
[ 2 match(es) found ]

   Benchmark: micro.empty
      Aliases: 
      Private: No
      Configurations: default
      Suite: micro
      Location: /home/bdufour/links/ashes2/benchmarks/micro/empty
      Language: java
      URL: 
      Description: The empty program, which simply returns from main

   Benchmark: micro.hello
      Aliases: hello_world
      Private: No
      Configurations: default
      Suite: micro
      Location: /home/bdufour/links/ashes2/benchmarks/micro/hello
      Language: java
      URL: 
      Description: The famous 'Hello World' program

Warning

You may need to quote the wildcard patterns if your shell tries to expand them automatically.

The following table lists various wildcards and regular expressions that Ashes2 support when searching:

Supported Wildcards and Regular Expressions

*: Matches one level in the hierarchy
**: Matches any number of levels in the hierarchy
?: Matches any character in a string
[]: Matches any character inside of the brackets. Ranges can also be specified using a '-', as in [A-Z] which matches all uppercase letters.
[^]: Negation of the above case, i.e. matches any character not inside the brackets.
{p1,p2,...,pN}: Matches any of the patterns p1 to pN.

The 'search' command returns any repository element that matches a pattern at any position in its FQN. However, if we only wanted to look for exact matches for a given pattern, the 'describe' command is to be used. For example, let's repeat our previous search for 'micro' using this command:

Listing 9

$ ashes2 describe micro
Searching ...
[ Results for search key micro ]
[ 1 match(es) found ]

   Suite: micro
      Aliases: small, nano, easy
      Private: No
      Location: /home/bdufour/links/ashes2/benchmarks/micro
      URL: 
      Description: A collection of micro benchmarks

If we wanted to list all benchmarks, we need the 'list' command. It takes a parameter specifying what to list. In your case, we enter the following:

Listing 10

$ ashes2 list benchmarks
   Available benchmarks:
      jolden.bh
      jolden.bisort
      jolden.em3d
      jolden.health
      jolden.mst
      jolden.perimeter
      jolden.power
      jolden.treeadd
      jolden.tsp
      jolden.voronoi
      micro.empty
      micro.hello
      spec.jvm98.check
      spec.jvm98.compress
      spec.jvm98.db
      spec.jvm98.jack
      spec.jvm98.javac
      spec.jvm98.jess
      spec.jvm98.mpegaudio
      spec.jvm98.mtrt
      spec.jvm98.raytrace

   Total benchmarks: 21

Ashes2 will list all of the FQN of each benchmark that it can find. Similarly, one can list all suites:

Listing 11

$ ashes2 list suites
   Available suites:
      621
      621.2002
      621.2003
      jolden
      micro
      spec
      spec.jvm98

   Total suites: 7

Next, we will look at how the various ways to customize Ashes2.


Back to Top


Customizing Ashes2

Configuration in Ashes2 is achieved through Python scripts. Basic knowledge of the Python is therefore an advantage, although not really required for basic configurations.

Ashes2 supports system-wide configurations as well as user-level configurations. System-wide configurations are found in the config/ directory in Ashes2's installation directory. User-level configurations are found in the .ashes2 directory in a user's home directory ($HOME under Posix systems).

In order for Ashes2 to be able to compile and execute benchmarks, it has to be told where to find a supported java compiler and java virtual machine. Note that multiple tools of each kind can be configured simultaneously. In the configuration directory of your choice ($ASHES2_HOME/config or $HOME/config), make sure that the lang directory exists. Then, open or create a file named java.py inside the lang directory. Then refer to the appropriate subsection to configure Ashes2 for your system.


Configuring Ashes2 to use a standard (Sun, IBM, Blackdown, etc.) JDK

To add configure Ashes2 to use a standard JDK, add the following lines to java.py, making sure that you change the values according to your system:

Code listing 1: Standard JDK Configuration

from ashes2.Scene import Scene

manager = Scene().getLanguageManager("java")
manager.addStandardJDK(
    "sun-jdk-1.4.1.03",      # Name of the JDK as it will appear in Ashes2
    "1.4.1_03",              # Version string for the JDK
    "/opt/sun-jdk-1.4.1.03") # Path to the JDK


Configuring Ashes2 to use the Jikes compiler

To configure Ashes2 to use Jikes, add the following lines to java.py, making sure that you change the values according to your system:

Code listing 2: Jikes Configuration

from ashes2.Scene import Scene

manager = Scene().getLanguageManager("java")
manager.addCompiler(JikesCompiler(
    "jikes",  # Name of the compiler as it will appear in Ashes2
    "1.3",    # Language version supported by the compiler
    "/usr",   # Path in which the 'jikes' executable is found
    ["/opt/ibm-jdk-1.3.1/jre/lib/rt.jar"])) # Classpath to use


Configuring Ashes2 to use SableVM

To configure Ashes2 to use SableVM, add the following lines to java.py, making sure that you change the values according to your system:

Code listing 3: SableVM Configuration

from ashes2.Scene import Scene

manager = Scene().getLanguageManager("java")
manager.addVM(SableVM(
    "sablevm-inlined-1.0.9", # Name of the VM as it will appear in Ashes2
    "/home/bdufour/pkgs/sablevm/current/bin/sablevm-inlined")) # Path to the SableVM executable


Real-world configuration example

More complex configurations are possible. The following lists the contents of my personal java.py as an example:

Code listing 4: Real-world Configuration Example

# Java configuration

import os, os.path
from ashes2.Scene import Scene
from ashes2.lang.java.StandardJVM import StandardJVM
from ashes2.lang.java.SableVM import SableVM
from ashes2.lang.java.JikesCompiler import JikesCompiler
from ashes2.Ashes2IO import ashes2_debug

# Get the Java manager
manager = Scene().getLanguageManager("java")

# Standard JDK configuration
manager.addStandardJDK("sun-jdk-1.2.2", "1.2", "/opt/sun-jdk-1.2.2")
manager.addStandardJDK("sun-jdk-1.4.1.02", "1.4", "/opt/sun-jdk-1.4.1.02")
manager.addStandardJDK("sun-jdk-1.4.1.03", "1.4", "/opt/sun-jdk-1.4.1.03")
manager.addStandardJDK("sun-jdk-1.3.1", "1.3", "/opt/sun-jdk-1.3.1.08")
manager.addStandardJDK("ibm-jdk-1.3.1", "1.3", "/opt/ibm-jdk-1.3.1")
manager.addStandardJDK("ibm-jdk-1.4.1", "1.4", "/opt/ibm-jdk-1.4.1")

# For convenience: provide aliases for the most commonly used
#  virtual machines
manager.aliasVM("sun-jdk-1.2.2", "jdk1.2")
manager.aliasVM("sun-jdk-1.3.1", "jdk1.3")
manager.aliasVM("sun-jdk-1.4.1.03", "jdk1.4")

# SableVM configuration
sablevm_home = os.path.expanduser("~/pkgs/sablevm/1.0.9/")
manager.addVM(SableVM("sablevm-inlined-1.0.9",
    sablevm_home + "bin/sablevm-inlined",
    aliases = ("sablevm-inlined",)))
manager.addVM(SableVM("sablevm-debug-1.0.9",
    sablevm_home + "bin/sablevm-debug",
    aliases = ("sablevm-debug",)))
manager.addVM(SableVM("sablevm-direct-1.0.9",
    sablevm_home + "bin/sablevm-direct",
    aliases = ("sablevm-direct",)))
manager.addVM(SableVM("sablevm-nosig-1.0.9",
    sablevm_home + "bin/sablevm-nosig",
    aliases = ("sablevm-nosig",)))
manager.addVM(SableVM("sablevm-switch-1.0.9",
    sablevm_home + "bin/sablevm-switch",
    aliases = ("sablevm-switch",)))
manager.addVM(SableVM("sablevm-traditional-1.0.9",
    sablevm_home + "bin/sablevm-traditional",
    aliases = ("sablevm-traditional",)))


# Jikes configuration
manager.addCompiler(JikesCompiler("jikes", "1.3", "/usr", ["/opt/ibm-jdk-1.3.1/jre/lib/rt.jar"]))

# Set a default VM and Compiler
manager.setDefaultJDK("sun-jdk-1.4.1.03")


Setting the Repository Path

Ashes2 configuration files can be used to permanently set the repository path without requiring environment variables. The following python code (usually placed in config.py in either configuration directory) will add a directory named ashes2 located in a user's home directory at the end of the repository search path:

Code listing 5: Setting the Repository Path

import os, os.path
from ashes2.Repository import Repository

ashes2_local = os.path.join(os.path.expanduser("~"), "ashes2")
Repository().append(ashes2_local)


Setting User Preferences

There are several preferences that can be set using properties. There are two ways to set properties. One way is via the -D option on the command line. For example, if we wanted to reduce the indentation step used by Ashes2 from four spaces to only two, we could use:

Listing 12: Setting properties via the command line

$ ashes2 -D ashes2.indent.step=8 help
Ashes2 Benchmarking System

Usage: ashes2 [options] <command> [args]

Available commands:
        Admin commands:
                list
                version
        Benchmark execution commands:
                compile
                run
        Benchmark management commands:
                export
                refresh
        Help commands:
                help
        Misc commands:
                prompt
        Search commands:
                describe
                locate
                search
        Shell commands:
                cd
                exec
                java
                javac
                ls
                pwd
        Utility commands:
                repeat
                time

To get help for specific command, use 'ashes2 help <cmd>'


Author: Bruno Dufour (bruno.dufour@mail.mcgill.ca)
URL: http://www.sable.mcgill.ca/~bdufou1/ashes2

However, it may be preferable to permanently set properties in most cases. The following code (usually found in the configuration file named config.py) will do the same make the property definition permanent:

Code listing 6: Setting properties via configuration files

from ashes2.Scene import Scene

Scene().setProperty("ashes2.indent.step", 8)

Note

A property set via the command line will override any definition of the same property found in configuration files

The following table lists all recognized properties and their possible values

Table 1: Ashes2 Properties

Property Name Description Possible Values
ashes2.colors.level1 Sets the value of the level1 color used in the console output. Color.
Possible values are:
  • bold
  • teal
  • turquoise
  • fuscia
  • purple
  • blue
  • darkblue
  • green
  • darkgreen
  • yellow
  • brown
  • red
  • darkred
  • "" (Empty string)
ashes2.colors.level2 Sets the value of the level2 color used in the console output. Color
ashes2.colors.level3 Sets the value of the level3 color used in the console output. Color
ashes2.colors.message Sets the color used for displaying Ashes2 regular messages. Color
ashes2.colors.warning Sets the color used for displaying Ashes2 warnings. Color
ashes2.colors.error Setst the color used for displaying Ashes2 error messages. Color
ashes2.colors.debug Setst the color used for displaying Ashes2 debug messages. Color
ashes2.time.format Sets the format of the displayed times. Valie Python float format string (e.g. "%.3f")
ashes2.prompt.ps1 Sets the value of the level1 prompt string (like in UN*X shells). Function returning as String or String
ashes2.prompt.ps2 Sets the value of the level2 prompt string (like in UN*X shells). Function returning a String or String
ashes2.indent.step Sets the number of spaces used for each indentation level. Default is 3. Integer (or integer as String)
ashes2.list.bullet Sets the string to be used as bullet in the output from the 'list' command. Default is the empty string. String


Back to Top


Performing Benchmarking Operations


Executing Benchmarks

In order to determine of at least one proper JVM is installed correctly, and to know which VM Ashes2 will use to run benchmarks, type the following at your prompt:

Listing 13

        $ ashes2 list java

This will make Ashes2 output the name of every JVM that it knows. For my own system, the output is:

Listing 14

           Available java tools:
      [ ] ibm-jdk-1.3.1
      [ ] ibm-jdk-1.4.1
      [ ] sablevm-debug-1.0.9 (Aliases: sablevm-debug)
      [ ] sablevm-direct-1.0.9 (Aliases: sablevm-direct)
      [ ] sablevm-inlined-1.0.9 (Aliases: sablevm-inlined)
      [ ] sablevm-nosig-1.0.9 (Aliases: sablevm-nosig)
      [ ] sablevm-switch-1.0.9 (Aliases: sablevm-switch)
      [ ] sablevm-traditional-1.0.9 (Aliases: sablevm-traditional)
      [ ] sun-jdk-1.2.2 (Aliases: jdk1.2)
      [ ] sun-jdk-1.3.1 (Aliases: jdk1.3)
      [ ] sun-jdk-1.4.1.02
      [*] sun-jdk-1.4.1.03 (Aliases: jdk1.4)

Note that the currently selected JVM is marked with a '*' next to it.

To execute benchmarks, we can its fully qualified name, like so:

Listing 15

        $ ashes2 run micro.empty
        +------------------------------------------------------------------------------+
|                       Executing benchmark: micro.empty                       |
+------------------------------------------------------------------------------+
-============================= Execution starting =============================-
   Execution status: SUCCESS
-============================ Execution completed =============================-

The run command accepts an arbitrary number of parameters, so that we can run multiple benchmarks in sequence:

Listing 16

        $ ashes2 run micro.empty micro.hello
        +------------------------------------------------------------------------------+
|                       Executing benchmark: micro.empty                       |
+------------------------------------------------------------------------------+
-============================= Execution starting =============================-
   Execution status: SUCCESS
-============================ Execution completed =============================-


+------------------------------------------------------------------------------+
|                       Executing benchmark: micro.hello                       |
+------------------------------------------------------------------------------+
-============================= Execution starting =============================-
   Execution status: SUCCESS
-============================ Execution completed =============================-

The run command also accepts parameters with wildcards, so that we can achieve the same effect by using:

Listing 17

        $ ashes2 run micro.*
        +------------------------------------------------------------------------------+
|                       Executing benchmark: micro.empty                       |
+------------------------------------------------------------------------------+
-============================= Execution starting =============================-
   Execution status: SUCCESS
-============================ Execution completed =============================-


+------------------------------------------------------------------------------+
|                       Executing benchmark: micro.hello                       |
+------------------------------------------------------------------------------+
-============================= Execution starting =============================-
   Execution status: SUCCESS
-============================ Execution completed =============================-

Note the running a benchmark suite is equivalent to running all of the benchmarks and suites that it contains, so the previous command could have been shortened to ashes2 run micro.

Executing benchmarks using an alternative JVM is just as easy. The --with-<tool> option can be used to change the default virtual machine. For example, if we wanted to use a 1.3 JVM instead, we could use:

Listing 18

        $ ashes2 --with-java=jdk1.3 list java
           Available java tools:
      [ ] ibm-jdk-1.3.1
      [ ] ibm-jdk-1.4.1
      [ ] sablevm-debug-1.0.9 (Aliases: sablevm-debug)
      [ ] sablevm-direct-1.0.9 (Aliases: sablevm-direct)
      [ ] sablevm-inlined-1.0.9 (Aliases: sablevm-inlined)
      [ ] sablevm-nosig-1.0.9 (Aliases: sablevm-nosig)
      [ ] sablevm-switch-1.0.9 (Aliases: sablevm-switch)
      [ ] sablevm-traditional-1.0.9 (Aliases: sablevm-traditional)
      [ ] sun-jdk-1.2.2 (Aliases: jdk1.2)
      [*] sun-jdk-1.3.1 (Aliases: jdk1.3)
      [ ] sun-jdk-1.4.1.02
      [ ] sun-jdk-1.4.1.03 (Aliases: jdk1.4)

Note that the default JVM has now changed.


Compiling Benchmarks

The compile command is very similar to the run command. One major difference is that we seldom want to compile benchmarks in place, so that we have to create a local working copy first. This can be achieved using the export command (or the refresh command):

Listing 19

$ ashes2 export micro
   Exporting suite 'micro' to 'micro'

Compiling the benchmarks can now be done safely:

Listing 20

$ ashes2 compile micro
+------------------------------------------------------------------------------+
|                       Compiling benchmark: micro.empty                       |
+------------------------------------------------------------------------------+
-============================ Compilation starting ============================-
   Compilation status: SUCCESS
-=========================== Compilation completed ============================-


+------------------------------------------------------------------------------+
|                       Compiling benchmark: micro.hello                       |
+------------------------------------------------------------------------------+
-============================ Compilation starting ============================-
   Compilation status: SUCCESS
-=========================== Compilation completed ============================-

Warning

When working with local copies of the repositories, make sure that the current directory (.) appears first in your repository search path. Use ashes2 list repositories to make sure that it is the case.

The javac tool is used to compilation, so all commands that used the java tool can be also used in this case:

Listing 21

$ ashes2 list javac
   Available javac tools:
      [ ] ibm-jdk-1.3.1
      [ ] ibm-jdk-1.4.1
      [ ] jikes
      [ ] sun-jdk-1.2.2
      [ ] sun-jdk-1.3.1
      [ ] sun-jdk-1.4.1.02
      [*] sun-jdk-1.4.1.03

$ ashes2 --with-javac=jdk1.3 list javac
   Available javac tools:
      [*] ibm-jdk-1.3.1
      [ ] ibm-jdk-1.4.1
      [ ] jikes
      [ ] sun-jdk-1.2.2
      [ ] sun-jdk-1.3.1
      [ ] sun-jdk-1.4.1.02
      [ ] sun-jdk-1.4.1.03


Back to Top


Using additional utilities

The time command can be used to time the execution of an Ashes2 command. It takes as parameter another Ashes2 command. For example:

Listing 22

$ ashes2 time run micro.hello
+------------------------------------------------------------------------------+
|                       Executing benchmark: micro.hello                       |
+------------------------------------------------------------------------------+
-============================= Execution starting =============================-
   Execution status: SUCCESS
-============================ Execution completed =============================-



Total time: 0.281 seconds

The repeat command can be used to repeat another Ashes2 command multiple times. It takes as parameters a repetition count and another Ashes2 command. For example:

Listing 23

$ ashes2 repeat 3 time run micro.hello
-================================ Iteration 1 =================================-
+------------------------------------------------------------------------------+
|                       Executing benchmark: micro.hello                       |
+------------------------------------------------------------------------------+
-============================= Execution starting =============================-
   Execution status: SUCCESS
-============================ Execution completed =============================-



Total time: 0.269 seconds
-==============================================================================-

-================================ Iteration 2 =================================-
+------------------------------------------------------------------------------+
|                       Executing benchmark: micro.hello                       |
+------------------------------------------------------------------------------+
-============================= Execution starting =============================-
   Execution status: SUCCESS
-============================ Execution completed =============================-



Total time: 0.284 seconds
-==============================================================================-

-================================ Iteration 3 =================================-
+------------------------------------------------------------------------------+
|                       Executing benchmark: micro.hello                       |
+------------------------------------------------------------------------------+
-============================= Execution starting =============================-
   Execution status: SUCCESS
-============================ Execution completed =============================-



Total time: 0.279 seconds
-==============================================================================-


Back to Top



Maintained by Bruno Dufour