Ville Oikarinen
generated on 2009-02-24
The build script of a (software) product needs to be reliable and fast. Full build is reliable but slow. Incremental build is fast, but all dependencies must be correctly declared.
For a simple product this is not usually a problem, but if it consists of several projects, dependency management becomes easily a problem.
To utilize the dependency handling mechanism of ant the targets must be defined in the same script, even if the product consists of several projects. (Naturally they can delegate details to macros or targets defined in helper scripts.)
The problem is that matching dependencies must be declared for compile-time and run-time classpaths. Certain other aspects lead easily to redundancy with a pure ant solution.
First a bit of ngremental terminology: The highest level directory of a product to build is called a workspace. The workspace consists of modules, some of which are projects while others are libraries.
The main targets of the workspace depend on individual project targets, which in turn depend on other module targets in some product-specific way.
ngremental is a combination of ant and ngrease scripts that let the user declare dependencies and other features of the modules of the workspace. The entry point ant script contains the top level targets that delegate all work to a generated ant script (which they depend on, of course).
The workspace and its modules are declared using an ngrease based Domain Specific Language. The declaration is split into many files so that each module can depend on its own declaration file. (Touching a module definition potentially changes its dependencies so all its targets should be considered outdated.)
For a quick start, please download ngremental from the SourceForge project page of ngrease and experiment with the bundled zero-configuration example.
The directory example-workspace contains a working workspace. In example-workspace/_ant run ant to build the default target is-commitable that makes sure all classes and test-reports are up to date.
Then run ant again to verify that ngremental does nothing when all targets are newer than their sources. Try touching some source files and run ant again to see how it executes the minimal set of tasks to ensure the workspace still is-commitable. You can also break some classes to make sure some test fails.
Next you can read the rest of this manual and try creating new modules and/or dependencies between them, trying out the different attributes supported by the project-def and library-def declarations and even customize the macros that define how each target type is built.
The default ant script _ant/build.xml is the entry point to ngremental and contains all the targets of the workspace. The script is an example, meant to be modified.
The targets are nouns so as a user of the script you are encouraged to say what you need and let the script determine whether it needs to do something in order to ensure the requested target is up to date and how to do it.
The main targets include
The script implementation delegates the targets to one huge main-script.xml it first generates using a workspace definition that refers to module definitions.
The generated main-script.xml utilizes the dependency mechanism of ant, which makes the targets as fast as possible without real risk of them being outdated.
Correctness is more important than speed, however, so the uptodate logic is cumulative, even for tests, because in practice bugs aren't always detected by the tests of the same project, but only by some project that depends on the buggy project.
The main responsibility of main-script.xml is the dependencies, and it delegates most of the real work to macros that are defined as separate files, _ant/import-*.xml for easy customization.
The file _ant/workspace.ngr defines the workspace by referring to all modules by name inside an workspace-ref element.
A module name is the module root directory path, relative to the workspace root and with space as the directory separator. For example module {common util} referes to a module in directory common/util under the workspace.
Example:
workspace-ref {
module {common util}
module {_lib ngrease}
}
Each module in the workspace must be defined by a project.ngr file in the module root.
There are two kinds of modules: projects and libraries. Details are defined below, in dedicated sections.
Both projects and libraries must have a name element, and its content must match the module name in workspace.ngr.
A project is a "normal" module i.e. a module that contains sources that need to be compiled into classes.
A project-def element must contain a name element and a deps element that defines the dependencies for the project.
Example:
project-def {
name {common ui}
has-tests
has-testutils
deps {
prod {common util}
test {_lib uispec4j-1.5}
}
}
Below is a list of all features project-def supports:
If a project has-tests its tested-classes target will depend on test-report which will depend on test-classes which in turn depends on the tested-classes targets of all dependencies, both prod and test type.
The has-tests element may define the name of the test source folder as its child. The default is test
So has-tests will declare tests in the default location, and for example has-tests:src/test/java overrides the default.
This optional parameter is passed to import-test-report.xml the bundled example version of which generates the test-report by running all test classes that match this pattern and don't match **/Abstract*.class
The example import-test-report.xml will use **/*Test.class as the default value for this parameter.
The testutils of a project are classes that are available to all test and testutil classes of projects that depend on the project.
The source folder of testutils is testutil
If a project has-sql its test-report will depend on a db target. The db target is built by import-sql.xml by executing the sql files under directory sql.
If the sql scripts of the project have property references that need to be resolved before execution, the properties files can be defined as Eclipse-style absolute paths, considering the workspace root as root.
Example:
sql-filters {
/common/valuetypes/src/common/valuetypes/sql-types.properties
/common/valuetypes/src/common/valuetypes/field-limits.properties
}
If the default statement delimiter ; cannot be used in the sql scripts for some reason, it can be redefined with the sql-delimiter element.
If a project has-ejbs its jar target will generate an ejb-jar instead of an ordinary jar. See import-jar.xml for details.
If a project has-custom-ejb-descriptors (under conf ), import-jar.xml will use them instead of the defaults.
A project can depend on other modules in two ways:
Note that prod implies test so you need to define only one of them for each dependency. Effectively test means test only.
There is a third dependency type, embedded , but since it was only needed because of a bug in an application server, it should be considered deprecated, unless new need arises.
If import-compile.xml cannot provide all functionality for class generation (code generation is needed etc), custom ant content can be defined inside the custom-classes-generator element.
Example:
custom-classes-generator {
raw:e {ant
a {antfile "classes.xml"}
a {dir "${dir.base}/api/fancy-api"}
a {target classes-for-ant}
a {inheritAll false}
}
}
The language for ant content will be documented later. Meanwhile this test for the target element may be helpful. You can also skip the ant level, like in the example above, with raw nxml content. See this test for an nxml e element for details.
Most probably custom-classes-generator will consume other files than src files, and these files must be declared so the generator is rerun when they are touched.
Example:
extra-classes {
classes {
dir:"${dir.base}/api/fancy-api/interface-definitions"
file:"${dir.base}/api/fancy-api/classes.xml"
}
#:"other types like test and testutil not implemented"
}
after-classes-task is like custom-classes-generator except that it doesn't replace the default class generation, it augments it.
custom-ant-content is inserted directly in the root of the generated ant script. This can be used to add targets and properties that cannot be generated by ngremental.
Certain application containers require certain parts of the application to be deployed "globally" instead of inside the ear package.
Such parts of the application are not added to the classpath of the Manifests of the depending projects.
A library provides class locations (usually jars) for projects, using an provides element.
Example:
library-def {
name {_lib ngrease-all-0.3.0}
provides {
lib/ngrease-ast-0.3.0.jar
lib/ngrease-metalanguage-0.3.0.jar
lib/ngrease-testutils-0.3.0.jar
examples
languages
}
}
A library that is provided-by-ejb-container won't be added to the Manifest of jars that depend on it.
ngremental has been developed for and used in only one product so some more parameterization is needed, for example in directory naming conventions. Some functionality, like database manipulation, is published as stub macros, because the implementations weren't generic enough.
The system is used by dozens of users daily to build a workspace of over one hundred modules so it is quite reliable, assuming no program does anything funny with timestamps (certain SVN clients are known to have questionable defaults for timestamp handling at revert, for example).
The author developed ngremental for Medici Data Oy. Thank you Juha Tenhunen of Medici Data Oy for the kind permission to publish it as part of the ngrease project.
Marko Oikarinen of Sysart Oy and Janne Heikkinen of Medici Data Oy contributed some code, especially ant import files.