\documentclass[letterpaper]{article}
%%%%%%%%%%%%%%%%%%%%%
\topmargin     -0.5in
\oddsidemargin    0in
\evensidemargin   0in
\textwidth        6in
\textheight     9in
%%%%%%%%%%%%%%%%%%%%%

\setlength{\parskip}{2mm}
\setlength{\parindent}{0mm}

\begin{document}
\title{MetaLexer Developer Manual}
\author{Andrew Casey - 260\ 260\ 558}
\maketitle

\section{Organization}

The files of the MetaLexer project are organized as follows.

\subsection{metalexer/}

\textbf{.classpath \& .project} Eclipse project files.

\textbf{common.properties \& .xml} The Ant build file for the frontend.  Shared by all backends.

\textbf{metalexer.properties \& .xml} The Ant build file for the metalexer backend.

\textbf{jflex.properties \& .xml} The Ant build file for the jflex backend.

\textbf{bin/} The directory containing the Java class files produced the build process (either Eclipse or CLI).  If jars are produced, then they will be stored elsewhere.

\textbf{gen/} The directory containing source files produced by the build process (scanners produced by JFlex, parsers produced by Beaver, AST node classes, some JUnit test classes, etc).

\textbf{permgen/} The directory containing files that will be copied to \textbf{gen} if for some reason generation is impossible (e.g. if JFlex is missing).

\textbf{lib/} The directory containing external jars depended upon by the build process.  Note that these jars are not required by the produced jars.

\textbf{run\_targets/} Useful eclipse run targets (mostly for running tests, but also for the frontend).

\textbf{src/} Handwritten (vs generated) source code (JFlex, Beaver, JastAdd, Java, etc), not related to testing.  Must not depend on anything in \textbf{test}.

\textbf{test/} Handwritten (vs generated) testing source code.  Not included in Jars.

\textbf{\dots} Anything else is unofficial and is not required for building MetaLexer.  It is not considered to be part of the system and may not be depended upon.

\subsection{metalexer/src/ \& metalexer/test/}

\textbf{frontend} Files related to the frontend or shared by all backends.

\textbf{backend-metalexer} Files related to the MetaLexer backend (i.e. for the MetaLexer-to-MetaLexer translator).

\textbf{backend-jflex} Files related to the JFlex backend (i.e. for the MetaLexer-to-JFlex translator).

\subsection{metalexer/src/frontend/}

\textbf{metalexer} Java package.

\textbf{component.flex \& .grammar \& .ast} Specification of the component language parser.

\textbf{layout.flex \& .grammar \& .ast} Specification of the layout language parser.

\textbf{Component \& LayoutInherit.jadd} The specifications of processInheritance for components and layouts, respectively.  These are the methods that trigger processing of the raw AST from the parser.

\textbf{Component \& LayoutErrors.jrag} The specifications of getErrors for components and layouts, respectively.  These files delineate all the possible (frontend) errors that can be raised.

\textbf{\dots} The other files provide support methods for processInheritance and getErrors.

\subsection{metalexer/src/frontend/metalexer}

\textbf{CompilationProblem \& Error \& Warning} CompilationProblem is the base type of all errors and warning encountered during compilation.

\textbf{Constants} Some helpful constants.

\textbf{FileLoader} Handles loading of components and layouts from files.

\textbf{PatternType} An enumeration of possible pattern types (i.e. acyclic, cyclic, cleanup).

\subsection{metalexer/src/backend-metalexer/}

\textbf{Component \& LayoutGeneration.jadd} The printers that turn a processed AST into MetaLexer files.

\textbf{metalexer/metalexer/} Java package containing \textbf{ML2ML}, the entry point for the MetaLexer-to-MetaLexer translator.

\subsection{metalexer/src/backend-jflex/}

\textbf{Component \& LayoutGeneration.jadd} The printers that turn a processed AST into JFlex files.

\textbf{\_Generation.jadd} Hierarchically called from \textbf{Component \& LayoutGeneration.jadd}.  Generate the .flex file output by the translator (i.e. the actual lexer).

\textbf{\_MetaGeneration.jadd} Hierarchically called from \textbf{LayoutGeneration.jadd}, under \textbf{LayoutMetaGeneration}.  Generate the .macro.flex file output by the translator (i.e. the meta-lexer).

\textbf{JFlexErrors.jrag} The specification of JFlex-specific component and layout errors.  Contributes to the same collections as \textbf{Component \& LayoutErrors.jrag}.

\textbf{ReturnWrap.flex} Custom lexer for translating return statements of the form \texttt{return X;} into statements of the form \texttt{return Maybe.Just(X);}.  This is handled with a lexer so that comments and strings can be handled properly.

\textbf{PackageFind.flex} Custom lexer for finding package statements of the form \texttt{package X;} and returning \texttt{X}.  This is handled with a lexer so that comments and strings can be handled properly.

\textbf{StaticFind.flex}  Custom lexer for finding all occurrences of the keyword static.  This is handled with a lexer so that comments and strings can be handled properly.

\textbf{metalexer/jflex/} Java package containing \textbf{ML2JFlex}, the entry point for the MetaLexer-to-JFlex translator.

\textbf{\dots} Files that support generation and meta-generation.

\subsection{metalexer/test/frontend/}

\textbf{c\_scanner.testlist} List of component language scanner tests (input in .in, expected output in .out).  Output files contain a list of tokens up to either EOF or the first error.

\textbf{c\_parserpass.testlist} List of component language parser tests (input in .in, expected output in .out).  Output files contain pretty printed versions of input files.

\textbf{c\_parserfail.testlist} List of component language parser tests (input in .in, expected output in .out).  Output files contain lists of errors.

\textbf{c\_inheritancepass.testlist}  List of component language inheritance tests (input in .mlc, expected output in .out).  Output files contain collapsed (i.e. post-inheritance), pretty printed versions of input files.

\textbf{c\_inheritancefail.testlist} List of component language inheritance tests (input in .mlc, expected output in .out).  Output files contain lists of errors.

\textbf{c\_error.testlist} List of component language error tests (input in .mlc, expected output in .out).  Output files contain lists of errors.

\textbf{l\_scanner.testlist} List of layout language scanner tests (input in .in, expected output in .out).  Output files contain a list of tokens up to either EOF or the first error.

\textbf{l\_parserpass.testlist} List of layout language parser tests (input in .in, expected output in .out).  Output files contain pretty printed versions of input files.

\textbf{l\_parserfail.testlist} List of layout language parser tests (input in .in, expected output in .out).  Output files contain lists of errors.

\textbf{l\_inheritancepass.testlist}  List of layout language inheritance tests (input in .mll, expected output in .out).  Output files contain collapsed (i.e. post-inheritance), pretty printed versions of input files.

\textbf{l\_inheritancefail.testlist} List of layout language inheritance tests (input in .mll, expected output in .out).  Output files contain lists of errors.

\textbf{l\_error.testlist} List of layout language error tests (input in .mll, expected output in .out).  Output files contain lists of errors.  Note: l\_error\_helper\_ tests check the effect of the \%helper directive on errors.

\textbf{metalexer/} Java package.  Contains JastAdd files (in contrast to \textbf{src}) to keep them separate from the test input and output files.

\textbf{\dots} Test specifications (as listed in .testlist files).

\subsection{metalexer/test/frontend/metalexer/}

\textbf{Component \& LayoutPrint.jrag} Pretty printers for testing.  Produce an output that is useful for testing.  Not guaranteed to produce correct MetaLexer (e.g. may include helpful, but illegal annotations).

\textbf{FrontendTests} The top-level JUnit test suite.

\textbf{\_TestBase} Class to be extended by generated test file.

\textbf{\_TestGenerator} Class that generates a test file from a .testlist file.

\textbf{\_TestTool} Class for generating .out files automatically.

\textbf{\dots} Support files for the Base, Generator, and Tool classes.

\subsection{metalexer/test/backend-metalexer/}

\textbf{metalexer/metalexer/} Java package containing the top-level JUnit test suite, \textbf{BackendMetalexerTests}.

\textbf{in/} Metalexer specifications.

\textbf{out1/} Files from \textbf{in} that have been run through the pretty printer.

\textbf{out2/} Files from \textbf{out1} that have been run through the pretty printer.  Contents should be identical to those of \textbf{out1}.

\subsection{metalexer/test/backend-jflex/}

\textbf{c\_error.testlist} List of component language (JFlex-specific) error tests (input in .mll, expected output in .out).  Output files contain lists of errors.

\textbf{c\_error.testlist} List of layout language (JFlex-specific) error tests (input in .mll, expected output in .out).  Output files contain lists of errors.

\textbf{m\_c\_meta.testlist} List of tests.  Each test has a .in file list of meta-tokens and regions to be passed to the generated meta-lexer for the component language specification and a .out file listing the expected embedding transitions returned by the meta-lexer.

\textbf{n\_scanner.testlist} List of natlab language scanner tests.  Copied from McLab but modified by removing lines passed through the comment buffer (i.e. preceded by '\#' in the original).

\textbf{in/} Specifications of various languages given in MetaLexer.

\textbf{out/} The output from translating the specifications in \textbf{in/} to JFlex: lexer and meta-lexer JFlex files, properties files recording the characters assigned to regions and meta-tokens and the numbers assigned to embeddings, and Java scanners output by JFlex.

\textbf{out/bin/} The class files that result from compiling the Java scanners output by JFlex.  Note: this directory is on the classpath so that these scanners can be tested by the JUnit test suite.

\textbf{placeholders/} Stubs of Java classes that contain just enough to make the generated scanners work.  Most importantly, the parser stubs contain the list of tokens on which the scanners depend.  Note: subdirectories are all Java packages and are on the classpath.

\textbf{metalexer/jflex/} Java package.  Contains JastAdd files to keep them separate from test intput files.

\subsection{metalexer/test/backend-jflex/metalexer/jflex/}

\textbf{BackendJFlexTests} The top-level JUnit test suite.

\textbf{CompilationTests}  Arguably the most important test cases.  Compiles the MetaLexer specifications in \textbf{in} to JFlex, to Java, to class files.  Depended upon by the tests that exercise the generated scanners.  Note: when running in Eclipse, it is necessary to run the tests more than once, refreshing in between, because Eclipse won't pick up the changes made by this test during a single run.

\textbf{\_Tests} Handwritten (vs generated) test cases.

\textbf{\_TestBase} Class to be extended by generated test file.

\textbf{\_TestGenerator} Class that generates a test file from a .testlist file.

\textbf{\_TestTool} Class for generating .out files automatically.

\textbf{JFlexHelper} A helper class that loads the JFlex jar at runtime if it is present.

\textbf{ReflectionHelper} A helper class that loads the generated scanner files at runtime if they have been generated (i.e. if JFlex is available).

\textbf{\dots} Support files for the Base, Generator, and Tool classes.

\section{JFlex}

Unfortunately, JFlex is covered by a GPL license.  Since the MetaLexer is covered by a BSD-style license, the MetaLexer distribution cannot contain JFlex.

MetaLexer will function properly without JFlex, but you will be unable to rebuild any modified .flex files and the compilation-based tests in the JFlex backend will be unavailable.

If you wish to download JFlex on your own, it is available at \textit{http://jflex.de/}.  Just unzip the archive under the lib/ directory and point \texttt{jflex.jar.path.prop} in \textit{common.properties} at the jar file (path should be relative to lib/).

\section{Configurations}

The MetaLexer compiler supports multiple backend with a shared frontend.  The current implementation supports MetaLexer-to-MetaLexer and MetaLexer-to-JFlex.

Unfortunately, since the project uses aspects, there is interference between the backends.  As a result, only one can be built and developed at a time.  One benefit of this approach is that it keeps the release jars small by excluding code required for other backends.

To work solely on the frontend, use the \textbf{common.xml} build file.  To work on the MetaLexer backend and the frontend, use the \textbf{metalexer.xml} build file.  To work on the JFlex backend and the frontend, use the \textbf{jflex.xml} build file.

\section{Building MetaLexer}

MetaLexer developers can use either the command line or the Eclipse IDE.

\subsection{Command Line}

When executing the build files from the command line, ensure that the \textit{eclipse.running} property is not set.  If it is, then the Java compilation steps will be skipped.

The Ant build file takes care of all classpath issues.  No configuration should be required.

\subsection{Eclipse}

To build the project in Eclipse, use the same Ant targets as from the command line, but ensure that the \textit{eclipse.running} property is set.  This will allow Eclipse to build Java files that are in source path folders in the Eclipse build path.

The provided \textbf{.classpath} file takes care of all source- and classpath issues.  No configuration should be required.

When running the JUnit tests in the JUnit view, note that the frontend and backend tests must be run separately.  (In contrast, the \textit{test} Ant target runs all appropriate tests in a single pass.)

If you want a really minimal release jar, you have to build it from the command line because otherwise Eclipse will compile some extraneous Java files and they will be included.

\end{document}