Artifactory is a shared, structured store for built software modules and Gradle is a tool which helps you package software modules and transfer them to and from such a store. (Gradle also has many other features which are not detailed in this guide.)

The Holy Gradle is a set of plugins for Gradle which extend it for

  • developing modules which contain a large number of files when built (such as C++ libraries or web applications);

  • working with multiple source code repositories;

  • delivering modules plus dependencies to developers who do not have access to Artifactory; and

  • working under Windows.

This section gives a short introduction to the main concepts and features. The following diagram shows how they fit in with other development tools.

Figure 1: Artifactory and Gradle in context
TO DO
Link to pages with more detail.

Gradle Module Concepts

In Gradle terms, a published module is a collection of files (which Gradle calls artifacts), plus metadata which identifies the module and describes how it depends on other modules. Modules may be published by many groups in different organisations, so a module identifier has three parts: group (also called organisation), name, and version. As an easy way to avoid conflicts, the group string often includes the reverse domain name of the organisation, in the Java style; for example, com.mycorp.teamA.

Modules are stored in repositories, which are usually on a shared server but can also be on the local file system. When a developer uses Gradle to download a module, the dependencies for that module are automatically also downloaded. Gradle will check that all versions are consistent, even if the same dependency is requested by several different libraries.

A module may contain several artifacts for different aspects of the same software component, if they are needed in different contexts. For example, one part may be for normal runtime use and another for debugging; or it might be a communication library with separate client and server parts. In a Gradle module these parts are called configurations. One configuration in a module can extend (include) other configurations to share artifacts. For example, the client and server parts of a communication library may need to share some definitions.

Each part may need different sets of other modules; therefore, each configuration has its own collection of dependencies. When we say "the dependencies of a module" we really mean "the dependencies of the configurations of a module".

The relationships between these concepts can be viewed as follows.

Figure 2: Gradle module concept relationships

Ignoring the detail of configurations, the relationships between some modules could be viewed as follows, where an arrow means "depends on".

Figure 3: Basic view of module dependencies

Suppose some team creates a new module N which uses A and C, giving the dependency graph below. Gradle understands that B is also needed, and will check that the version of C is consistent.

Figure 4: More complex module dependencies

Now, suppose these modules each have configurations compile and runtime. Then A would typically declare that its compile configuration depends on compile from B, and that its runtime configuration depends on B’s runtime configuration. This means that another module such as N, which declares a dependency on the runtime configuration of A, will get the runtime files for B as well as A.

Suppose N uses C as a static library, but uses A as a plugin so it only needs runtime files for A and not header and linker files from the compile configuration. Then diagram below highlights in blue the configurations which will be fetched for the configurations of N.

Figure 5: Dependencies between configurations

More details on extending configurations, and mapping sets of related configurations between modules, can be found in the Configurations section of the link:workflows.html page.

Artifactory

Artifactory is a web service for storing modules which are published using tools such as Gradle. The simplest view of it is "version control for binaries" or "FTP / file share with history". However, it has more features and some standards to follow, described in this section.

Repositories

An Artifactory server instance does not allow users to store files in any structure they choose. It stores modules in named collections, called repositories. These can only be created by server administrators. Within each repository (or "repo"), the files for each module are in a three-level folder structure: group, then name, then version.

Gradle accesses modules by URL, usually of the format https://server.example-corp.com/artifactory/repository/group/name/version/…. Within the Artifactory web UI, the contents can be seen on the "Artifacts" tab. (Note that the web UI does not exactly match the view used by tools like Gradle, as described below.)

There are three kinds of repository on a server: local, remote, and virtual. Normally only server administrators need to deal with remote repositories directly.

Local

These contain artifacts whose contents are managed by that server. The word "local" does not mean "only visible within one organisation" — visibility depends on permissions, and visibility to other organisations depends on network configuration. They are usually named myrepo-local , and both Gradle and the web UI refer to them in this way.

Remote

These contains local copies of artifacts automatically cached from a repo on another server. Cached files may be downloaded on demand or in advance, and may be automatically deleted, depending on server setup. In the "Artifacts" tab of the web UI these appear as myrepo-cache in the "Tree Browser" and show only the files which have already been cached. When using Gradle they are referenced as just myrepo , and any attempt to GET a file which is not already cached will cause it to be cached. Both views can be found under the "Simple Browser" section of the web UI. Normally you will not refer to a remote repository directly, but will use a virtual repo which includes that remote.

Virtual

These point to one or more other repos, using a list configured by admins. Requests for artifacts from these repos will search the list in order. Like remote repos, virtual repos are also named as just myrepo . These are only shown in the "Simple Browser", not in the "Tree Browser". For a given folder or file in the "Tree Browser" you can find which virtual repos include it by looking in the "Virtual Repository Associations" section on the right.

Usually a server admin will set up one or more virtual repos for a team which includes all the local and remote repos they need. This is convenient for the team, because they only need to list one repo in their build.gradle file. It is also important for admins, if they need to change repository configuration in future, for example, to change permissions or backup processes. When they make such changes they can also change the virtual repo lists, so that all requests to the virtual repo work as before.

Important

You can think of virtual repos like "interfaces", and local and remote repos like "implementation". You should always set up your build.gradle scripts to

  • get dependencies from a virtual repo (so that admins can re-arrange the other repos); and

  • publish to a local repo (because you can’t publish to the other two).

Access Control

Artifactory has access control, so different users may have different permissions to read or modify a repository. The Holy Gradle includes a feature to keep a developer’s Artifactory password securely in the Windows Credential Manager, so that it can log in automatically without storing passwords in build scripts.

Module Version Lifetimes

An important question which Artifactory cannot fully answer is, after a version of a module is published, when can it be deleted? Official release versions may need to be kept for many years but often teams also make temporary versions, and delete them to save disk space.

A common approach is to put long-term and temporary releases in different repositories: something-release-local for "permanent" versions, and something-integration-local for temporary ones. The Holy Gradle provides a plugin for deleting older versions of modules using date ranges and other rules, and it allows different rules for each repository. Typically this plugin would be run periodically using a tool like Jenkins.

Promotion

A common development process called continuous integration involves building and testing a module regularly, for example, any time a new source code change is committed. When a build passes all tests, it can be used as a release candidate.

When a repository server like Artifactory is involved, it is a good idea to publish each build to a temporary repository. This tests the publish process, and also allows the builds to be downloaded from Artifactory for manual testing. To release a successful build, you could re-run the publish process targeting a repository which will store the module permanently. However, Artifactory provides a quick way to copy or move a module to another repository on the same server.

Gradle and the Holy Gradle

The Holy Gradle provides a way to publish and use pre-built and pre-packaged software modules produced with tools like C++, HTML, and JavaScript. These tools do not provide their own module packaging system and their output form can have a large number of files. (In contrast, Java and .NET have their own packaging systems, and only a small number of output files per module). A packaged module may be a library, web component, build tool, end-user application, test data set, or any other collection of files which are useful for developing software components.

Basic Modular Development Process

The basic approach for developing with Gradle is as follows.

  • Write a build.gradle file for your module (call it A) which lists

    • the configurations of A;

    • the modules needed by each configuration (its dependencies), each with its own version (for example, B:1.0 and C:1.5);

    • the locations for downloading those dependencies;

    • the artifacts to publish for each configuration of A;

    • the location to publish A.

  • Run Gradle to download the dependencies.

  • Build and test A.

  • Run Gradle to publish A.

The downloading and publishing of modules in the above example could look like this, where an arrow means "file transfer". The group parts of the module IDs are omitted for simplicity.

Figure 6: Basic view of using Gradle

The transferred files also include a metadata file describing the module. For the Holy Gradle this is an ivy.xml file.

Holy Gradle Modules

The Holy Gradle plugins add several features to processes of fetching and publishing modules.

Packed Dependencies

The artifacts of Java-based modules are often only one or two JAR files. To build a module, the file names of these JAR dependencies are simply passed to the Java compiler on the command line.

For languages like C++, each module may provide many files — header files, static and dynamic libraries, resources, and so on. Also, some of these files may have variants such as for "release" and "debug" mode, or for 32-bit and 64-bit platforms. To build such a module, the file names from its dependencies are often too many to pass on a command line, and compilers expect to get a list of folders to search for header files etc.

The basic extension which the Holy Gradle's intrepid-plugin provides is to collect files into one or more ZIP files (packages) when publishing, and unzip those files (into your Gradle user home) when dependencies are downloaded. All packages are unpacked into the same folder, so that folder will contain the union of all those files. It also extends the Gradle dependency syntax to specify a relative folder path for each dependency, and the unzipped files appear at that location in your workspace using links. The compiler can then be configured to look for the dependency files at those locations.

Source Dependencies

In Subversion and some other source control systems, a very simple way to handle dependencies is to add them as "externals". This has many disadvantages — for example, there is no easy way use the same external at more than one place in the graph, as in the previous example where module C is used by both A and N. However, it is useful for combining source code from multiple repositories, for cases where several components are always built together, which is not well supported by some other systems such as Mercurial.

The Holy Gradle adds the notion of "source dependencies" to Gradle. "Downloading" a source dependency will check it out from source control. When the original module is published, its source dependencies will also be published, as separate modules. The metadata will be automatically filled in to describe the connections between these modules.

Source Reference Information

In the Java world, binary modules often have a matching source module, containing the same version of the source code, for example for debugging use. Java IDEs are often built to automatically download source published in this way. For C++ applications on Windows, the Visual Studio IDE instead downloads source from a Microsoft Symbol Server. Neither the Holy Gradle nor Artifactory include special support for this. However, the Holy Gradle automatically adds a build_info folder to each module, which has a reference to the source code location and revision.

Holy Gradle Plugins

The Holy Gradle is implemented as a set of Gradle plugins, plus a custom distribution of Gradle (for bootstrapping and debugging the plugins). The section A Minimal Holy Gradle Build File shows how you include these plugins in your build file.

DONE: https://bitbucket.org/nm2501/holy-gradle-plugins/wiki/Home

The usage relationship between the plugins, the custom distribution, and some other related tools, is as follows.

Figure 7: Holy Gradle component dependencies

More detail on each plugin can be found on the Holy Gradle Software Components page. The original Requirements are also part of this documentation.

TO DO
Link to doc on how to do more complex cases with dependencies which also need to be promoted.