• Main Menu
• Home
• Articles
• Books
• Glossary
• Links

Escape from JAR hell with Apache Ivy the agile dependency manager

The Java dependency management problem

If you took a typical enterprise Java application and decomposed it into the proprietary or third party components that were used to develop it, you could easily find upward of 100 different libraries (in the form of Java archives - or JAR files). A number of these libraries might be open source implementations, such as the Jakarta commons libraries, others might be commercial business components that you have sourced from a third party. However, since many commercial components also make extensive use of open source components themselves, you can quite easily find yourself with duplicate sets of libraries possibly at different versions!

Similarly, when you are developing your own common libraries, you will no doubt be creating multiple versions of them and will probably need to support different versions for different projects, or for projects in different phases. Indeed, your library dependencies might have dependencies on other libraries themselves - a chain of dependencies such as this  is often called a transitive dependency.  

Without some mechanism to manage and capture this kind of dependency metadata you can quite literally end up in JAR hell!

One typical approach is to create a shared repository of libraries - maybe in a version control system. This approach can work but requires a significant amount of administration effort to keep it up to date. This is especially true for transitive dependencies where you would need to somehow need to work out all the chains of dependencies and ensure you get correct and consistent versions.

Fortunately, there are now a number of Java tools that can help you solve this problem. These include Apache Maven (which is an alternative to Apache Ant for Java build and project management) and the tool that is the subject of this article: Apache Ivy. If you are an existing Apache Ant user, then Ivy should be particularly interesting because it works seamlessly with your existing Ant build scripts.

Installing Apache Ivy

Apache Ivy is distributed as a binary Windows Zip file or Linux/UNIX compressed tar file; you can download the latest version of Ivy from http://ant.ivy.org/download.html. Inside the distribution file you will find documentation and samples as well as a Java library with the name ivy-2.1.0.jar (or similar version depending on the version of Ivy you downloaded). You can either install this library to your Ant lib directory (for example, %ANT_HOME%\lib) or to a directory alongside your top-level build.xml file. I prefer to install the Java library into a directory called lib alongside a common build.xml file, and then place it under version control. In this article I will therefore assume you are doing the same.

Once you have done this you will need to change your build.xml to reference the Ivy tasks “namespace”. You can achieve this by adding the ivy namespace shown in the following to your top-level <project> element:

Namespaces allow you to reference third party libraries and their tasks in shorthand format throughout your build script. In this case the namespace being created is called “ivy” and refers to the Uniform Resource Identifier (URI) “org.apache.ivy.ant”. In order for Ant to be able to map this URI to the Java classes for the Ivy tasks, you will also need to define a Classpath for the mapping as follows:

<!-- setup reference to the Ivy Ant tasks library --> 
<path id="ivy.lib.path">
    <fileset dir="${dir.lib}" includes="ivy*.jar" />
</path>

<taskdef resource="org/apache/ivy/ant/antlib.xml"
         uri="antlib:org.apache.ivy.ant" 
         classpathref="ivy.lib.path" />

This <taskdef> element references the Ant tasks for the Ivy library that you have downloaded and installed – in this case in the location referenced by the ${dir.lib} property. Once this has been carried out then Any of the Apache Ivy tasks can be called simply by using the prefix <ivy:task_name/>, for example <ivy:publish/> to call the Ivy “publish” task.

Ivy repositories

Apache Ivy uses structured repositories to store and organize the Java archive libraries that you will be consuming as part of your build process. Apache Ivy breaks down the contents of the repository into modules and artifacts. The artifacts are the physical Java archives (JAR files) you are going to build or execute against together with the metadata files which are used to specify information about the archives and their dependencies. Ivy refers to a collection of these artifacts into a component as a module. This terminology is used because a module man contain (publish) a number of artifacts; as a simple example a module might contain artifacts for both binary and source Java archives.

You can make use of two pre-populated repositories: ivyrep (www.jayasoft.fr/org/ivyrep/) or the Apache Maven ibiblio repository located at http://ibiblio.org/maven2. You can also implement your own internal repository on a file system or local web server. An obvious reason for doing so is security – in commercial environments you probably would not want to publish your companies internally developed modules on the Internet. While you are learning about Apache Ivy making use of the Internet based repositories is fine, however at some stage I recommend that you create your own repository since you will have little or no control over the content of public ones. You also might be behind a restrictive firewall or even have no Internet access. 

<?xml version="1.0" encoding=" ISO-8859-1"?>
    <ivy-module version="1.0">  
        <info organisation="apache" module="commons-cli"  
              revision="1.0" status="release"  
              publication="20021227095900">  
            <license name="Apache" 
                     url="http://www.apache.org/licenses/LICENSE-2.0.txt"/>  
            <ivyauthor name="jayasoft" url="http://www.jayasoft.org/"/>  
            <description homepage="http://jakarta.apache.org/commons/cli/">  
                The CLI library provides a simple and easy to use API for
                working with the command line arguments and options.  
            </description>  
      </info>  
      <dependencies>  
          <dependency org="apache" name="commons-lang" rev="1.0"/>  
          <dependency org="apache" name="commons-logging" rev="1.0"/>  
      </dependencies>  
</ivy-module>

In this example you can see that commons-cli library is dependent on two other libraries commons-lang and commons-logging. Each of these libraries may also be dependent on other libraries - if so Ivy can manage the resolution of these transitive dependencies for you. Note that this example is equivalent to how your own internal repository might look. In pratice, Ivyrep only stores the metadata files (ivy-1.0.xml) whilst the maven ibiblio repository stores the physical Java archive an also a Maven POM file (i.e. commons-cli-1.0.pom) - this file details dependencies in a similar way but in Maven's own format. I recommend having a browse at the commons-cli entries in ivrep (http://www.jayasoft.fr/org/ivyrep/apache/commons-cli/) or ibiblio (http://www.ibiblio.org/maven/commons-cli/) to see what these repositories look like for yourself

Ivy module resolution

<ivysettings>
    <settings defaultResolver="happybank" />
    <property name="ibiblio-maven2-root" value="http://repo1.maven.org/maven2/" />
    <resolvers>
        <chain name="happybank">
            <filesystem name="local">
                <ivy pattern="${ivy.settings.dir}/repository/[module]/
                 ivy-[revision].xml" />
                <artifact pattern="${ivy.settings.dir}/repository/[module]/
                [artifact]-[revision].[ext]" />
            </filesystem>
            <url name="buildmeister">
               <ivy pattern="http://m2.buildmeister.com/[module]/[revision]/
                ivy-[revision].xml" />
               <artifact pattern="http://m2.buildmeister.com/[module]/[revision]/
                [artifact]-[revision].[ext]" />
            </url>  
            <ibiblio name="ibiblio" m2compatible="true" 
                     root="${ibiblio-maven2-root}" />      
        </chain>
    </resolvers>
</ivysettings>

This example "chains" together three resolvers as follows:

• The first resolver (called “local”) searches for any revisions of modules in a local project repository which is contained on the file system – in this case in a directory located at the same level as the ivysettings.xml file but more likely this would be on a shared directory somewhere on your network.
• If a module cannot be found at the first location then a second resolver (called “buildmeister”) is used to search for revisions in an Internet hosted repository that I have created at http://m2.buildmeister.com.
• Finally if no matching revisions can be found then modules are searched for on the Internet in the ibiblio repository.

You will have noticed the pattern format that Ivy uses here for matching artifacts and artifact metadata; for example the pattern "[module]/[artifact]-[revision].[ext]" will for commons-cli version 1.0 be expanded to look for a physical Java archive called "commons-cli/commons-cli-1.0.jar" in the repository.

To make use of this ivysettings.xml file you will need to include it in your build script. An example of a complete build.xml file for the HappyBank common project to achieve this would be as follows:

<?xml version="1.0">
<project name="common" default="compile" xmlns:ivy="antlib:org.apache.ivy.ant">

    <property name="dir.src"    value="src"/>
    <property name="dir.build"  value="build"/>
    <property name="dir.libs"   value="../common/libs"/>

    <!-- load ivy settings -->
    <ivy:settings file="ivysettings.xml" id="happybank.common.ivy" />

    <!-- setup reference to the Ivy Ant tasks library --> 
    <path id="ivy.lib.path">
        <fileset dir="${dir.lib}" includes="ivy*.jar" />
    </path>
    <taskdef resource="org/apache/ivy/ant/antlib.xml"
        uri="antlib:org.apache.ivy.ant"  
        classpathref="ivy.lib.path" />

    <!-- resolve dependent libraries -->
    <target name="ivy.resolve">
        <ivy:resolve file="ivy.xml" settingsRef="happybank.common.ivy" />
        <ivy:cachepath pathid="happybank.common.classpath"
                       settingsRef="happybank.common.ivy" />
    </target>

    <!-- compile the source code -->
    <target name="compile" depends="ivy.resolve">
        <javac destdir="${dir.build}"
            <src path="${dir.src}"/>
            <classpath refid="happybank.common.classpath"/>
        </javac>
    </target>

</project>

This build script uses the <settings> task to load the ivy settings file and creates a reference to it called happybank.common.ivy. This reference can then be used by other Ivy tasks as illustrated in the ivy.resolve target. This target first resolves all of the dependent modules via the <resolve> task and then uses the <cachepath> task to create a Java Classpath that will refer to the exact versions of the modules that have been resolved. Hopefully you are now starting to see some of the power of Ivy!

Let us take a look at a real world example to see what this resolution process looks like. If we had an ivy.xml file which contained the following dependencies:

<?xml version="1.0" encoding="UTF-8"?>
<ivy-module version="2.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="http://ant.apache.org/ivy/schemas/ivy.xsd">
    <info organisation="happybank" module="happybank-common" />
    <dependencies>
        <dependency org="dbdeploy" name="dbdeploy-ant" rev="3.0M1" />
        <dependency org="commons-cli" name="commons-cli" rev="1.0" />
        <dependency org="junit" name="junit" rev="3.8.2" />
    </dependencies>
</ivy-module>

then executing ant compile from the command line would display output similar to the following:

In this example you can see that all seven dependencies have been successfully resolved and downloaded from buildmeister.com. Note that by default Ivy downloads and caches JAR files and metadata to a directory called ".ivy" in your home directory. If you look at this location you should see a directory structure similar to the repository structure I described earlier.

Ivy configurations

Configurations are one of the most powerful features of Ivy, however they do take a little bit of patience and practice to get used to. Essentially Ivy configurations allow you specify how a "group of dependencies" is related and when they should be used. For example you might want to specify some dependencies as compile time only, so that they won’t be included at runtime, maybe because you are developing a web application and a number of the libraries are already provided by target web container. The examples we have seen so far have used the "default" configuration only. If we wanted to create some more advanced configurations we could augment our existing ivy.xml file as follows:

<?xml version="1.0" encoding="UTF-8"?>
<ivy-module version="2.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="http://ant.apache.org/ivy/schemas/ivy.xsd">
    <info organisation="happybank" module="happybank-common" />
    <configurations>
        <conf name="local" visibility="private" />
        <conf name="compile" description="used for building" />
        <conf name="test" extends="compile" description="used for testing" />
        <conf name="runtime" description="used for running" />
        <conf name="master" description="used for publishing" />
        <conf name="default" extends="master, runtime" />
    </configurations>
    <dependencies>
        <dependency org="dbdeploy" name="dbdeploy-ant" rev="3.0M1" 
                    conf="local->default" />
        <dependency org="commons-cli" name="commons-cli" rev="1.0" 
                    conf="compile->default;runtime->default" />
        <dependency org="junit" name="junit" rev="3.8.1" 
                    conf="local->default;compile->default;test->default" />
    </dependencies>
</ivy-module>

In this example I have defined a number of configurations which can be summarised as follows:

• local- for dependencies which are used/local to this project only; this configuration is marked as private so that it cannot be reused
• compile - for dependencies we need to compile code
• test - for dependencies we need to run against when we are executing unit test
• runtime - for dependencies we need to run against when executing the project
• master - a configuration used for publishing modules (this won't make much sense now but we will discuss publishing a bit later)
• default - if any other projects reference this project's published modules as a dependency then the default mapping is used to specify that it would need to resolve both the published and runtime libraries

In each of the dependencies you can then specify how they will be resolved according to these configurations. For example, when the project modules are resolved and built using with “compile” configuration, Ivy will include the modules for junit, commons-cli and so on in the dependency list. Any (transitive) dependencies - if they exists - will be resolved using the “default” configuration for those modules. As its name implies the “default” configuration exists by default, and is valid until you define any "conf's" explicitly in ivy.xml file. You can specify any available conf definition on both sides of  the "-> "equation. In other words, you map configurations of your project's to configurations of its dependent projects. If you omit the conf attribute in your dependency element, then it is assumed to be "*->*" by default, which means it will be resolved from any configuration to any other configuration of the same name.

So now that we have our configurations defined, let's look at what executing our ant compile command would do:

You can see now that we have resolved all of the same modules, but this time they are grouped according to their configuration. What this allows us to do is to specify a number of Classpath's based on these configurations as follows:

<target name="ivy.resolve">
    <ivy:resolve file="ivy.xml" settingsRef="happybank.common.ivy" />
    <ivy:cachepath pathid="ivy.compile.classpath" conf="compile" type="jar"
                   settingsRef="happybank.common.ivy" />
    <ivy:cachepath pathid="ivy.test.classpath" conf="test" type="jar"
                   settingsRef="happybank.common.ivy" />        
    <ivy:cachepath pathid="ivy.runtime.classpath" conf="runtime" type="jar"
                   settingsRef="happybank.common.ivy" />
</target>

As you can see configurations are very powerful and come into their own when you are creating reusable libraries. With well specified configurations consumers of your libraries can get their hands on exactly what they need depending on how they are consuming your libraries, i.e. for compile, test, runtime and so on. Obviously one thing you should probably do is agree on a naming convention for configurations. The ones I have described here are a good start.

Publishing modules

As well as consuming previously built modules you can also use Ivy to publish your own internally developed modules. To do this you use Ivy's <publish> task. First however, you need to specify what modules your project will publish. To do this you specify a <publications> entry in your ivy.xml file as follows:

<?xml version="1.0" encoding="UTF-8"?>
<ivy-module version="2.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="http://ant.apache.org/ivy/schemas/ivy.xsd">
    <info organisation="happybank" module="happybank-common" />
    <configurations>
        ...
    </configurations>
    <publications>
        <artifact conf="master" name="happybank-common" type="jar" ext="jar"/>
    </publications>
    <dependencies>
        ...
    </dependencies>
</ivy-module>

You can either publish "integration" or "release" versions of your modules. So far we have been using released version of modules only, i.e. 1.0, 1.1, 2.0 and so on. An example of an Ant target to publish a release version of the modules for this project would look similar to the following:

<target name="ivy.publish" depends="jar">
    <property name="revision" value="${build.number}"/>
    <ivy:publish artifactspattern="${dir.dist}/[artifact].[ext]" 
        resolver="local" 
        settingsRef="happybank.common.ivy"
        pubrevision="${revision}"
        status="release"/>
    <echo message="project ${ant.project.name} released with version ${revision}"/>
</target>

This example will publish any modules built in the directory specified by the dir.dist property together with an automatically resolved iv-[revision].xml containing a fixed set of module dependencies. The location to publish to is specified by the resolver, in this case it is the "local" resolver which refers to the file system directory location described in the previous section. This example publishes a specific version of the module referred to by the Ant ${revision} property and uses the "status" flag to indicate this.

If you building internal modules frequently (perhaps as part of a Continuous Integration build) you can publish integration versions rather than release. To do so you would change the publish target to the following:

<target name="ivy.publish" depends="jar">
    <ivy:publish artifactspattern="${dir.dist}/[artifact].[ext]" 
        resolver="local" 
        settingsRef="happybank.common.ivy"
        pubrevision="latest.integration" />
</target>

Note that Ivy has three pre-defined statuses "released", "milestone" and "integration" (although you can define your own). If specify latest.[status] then in order to find the latest revision with the appropriate status Ivy has to parse all the ivy files in your repository from the last one until it finds such a revision. Hence don't be surprised if the resolution slow down.

To make use of integration versions in other modules, you simply specify the dependency as follows:

<?xml version="1.0" encoding="UTF-8"?>
<ivy-module version="2.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="http://ant.apache.org/ivy/schemas/ivy.xsd">
    <info organisation="happybank" module="happybank-online" />
    <configurations>
        ...
    </configurations>
    <dependencies>
        <dependency org="happybank" name="happybank-common" changing="true"   
                    rev="latest.integration" 
                    conf="compile->master;runtime->default" />
        ...
    </dependencies>
</ivy-module>

This ivy.xml file is for a projectwhich consumes our common module's outputs; you can also see how configuration mapping is carried out across modules you create.

Other capabilities and integrations

The scenarios I have discussed above are the most common uses however Ivy has a number of additional capabilities (and Ant tasks) that are worth investigating. Examine the tutorials and documentation on the Ivy website for more details. One capability that is certainly worth mentioning is the graphical dependency reports that Ivy can automatically create via the Ivy <report> task. As an example, the report for the module used in this article is illustrated below:

These reports can give you a really good view of a projects modules and dependencies. More examples of reports such as these can be seen on the Ivy website.

Ivy also integrates well with a number of popular open source tools. Obviously, it integrates seamlessly with Apache Ant, however there is also a plugin for integrating Ivy into the Eclipse development environment (or any commercial tools based on it). This integration gives you an editor for creating Ivy files as well as a useful capability for resolving libraries and adding the resultant set of libraries to your Eclipse project's build path. This feature is a big "buggy" at the moment but I have had some success with it. The screenshot below shows the modules resolved in Eclipse by the Ivy plugin:

Finally, Ivy can also integrate with CruiseControl - the popular open source Continuous Integration toolkit. This allows you to automate the building of interdependent projects for Continuous Integration.

Summary

In summary, Ivy is an extremely powerful addition to Apache Ant for implementing Enterprise dependency management. It is relatively straightforward to get up and running with Ivy, but there is also enough depth and power to ensure that it is usable across large and multiple projects. By using Ant and Ivy together you can succesfully move away from manually managing libraries in a shared directory to a more controlled and scaleable mechanism. Finally, the combination of Ant and Ivy certainly make a compelling alternative to Apache Maven.

If you would like to see a more detailed example of how Apache Ivy can be used on a working project then visit https://sourceforge.net/projects/happybank/ and download its source code.

References

• Ivy home page
• Automation for the people: Manage dependencies with Ivy
• Nexus the Maven Repository Manager - useful for creating your own internal repositories

Comments

There are no comments on this article.

Back to Top

Submit a new comment

All fields in bold are required.

Name:

Website:

Comment:

Reload Image

Catchpa:

   Processing...