D

dinodocs

Git repository web hook handler that builds documentation

Name Last Update
WebContent Loading commit data...
src/us/rader Loading commit data...
.gitattributes Loading commit data...
.gitignore Loading commit data...
Doxyfile Loading commit data...
LICENSE Loading commit data...
README.md Loading commit data...
build-doc.sh Loading commit data...
pom.xml Loading commit data...

Copyright © 2015-2016 Kirk Rader. All rights reserved.

dinodocs

Listen for commit hook invocations from source control management services. Automatically build Sphinx and / or Doxygen documentation fetched from source control.

Currently supports commit hook invocations from GitLab and Subversion.

Features

  • Supports multiple version control hosting servers (currently GitLab and Subversion)

  • Automatically generates latest documentation on each commit

  • Supports Sphinx mark-up based documentation

  • Supports software source code in a variety of languages, including Java and C++, with documentation comments in Javadoc / Doxygen compatible formats

  • Supports "hybrid" Sphinx / Doxygen based projects using the breathe plug-in

  • Supports Graphviz and PlantUML diagrams embedded directly in mark-up and / or source code

  • Builds and serves both HTML and PDF versions of documentation

  • Automatically builds Zip archives of the HTML output to make it easy publish outside of dinodocs, itself

Prerequisites

Note that dinodocs currently will only run on a Linux server.

TODO: create a Windows cmd script version of build-documents.sh.

  1. texlive (e.g. sudo apt-get install texlive-full)

  2. Python (e.g. sudo apt-get install python2.7)

  3. pip (e.g. sudo apt-get install python-pip)

  4. Java 7 or later (e.g. sudo apt-get install openjdk-7-jdk)

  5. JEE servlet container (e.g. sudo apt-get install tomcat7)

  6. Graphviz (e.g. sudo apt-get install graphviz)

  7. PlantUML (see Configuration )

  8. Git (e.g. sudo apt-get install git)

  9. Subversion (e.g. sudo apt-get subversion)

  10. Doxygen (e.g. sudo apt-get install doxygen)

  11. Maven (e.g. sudo apt-get install maven -- only required to build dinodocs using its pom.xml)

Building

To build dinodocs.war:

git clone https://github.com/parasaurolophus/dinodocs.git
cd dinodocs
mvn package

Configuration

Additional steps are required after installing all of the preceding packages.

JEE Servlet Container

Set up your servlet container, e.g. see http://tomcat.apache.org/tomcat-7.0-doc/index.html

Use the deployment tools of your choice to install dinodocs.war -- but note the additional configuration options that you may need described below for PlantUML and the Python virtual environment.

PlantUML

The plantuml.jar file must be in a location that is accessible to the process running dinodocs and referenced via the PLANTUML_JAR environment setting in web.xml. The default value for PLANTUML_JAR is /usr/lib/plantuml.jar:

<env-entry>
 <description>Path to plantuml.jar</description>
 <env-entry-name>PLANTUML_JAR</env-entry-name>
 <env-entry-type>java.lang.String</env-entry-type>
 <env-entry-value>/usr/lib/plantuml.jar</env-entry-value>
</env-entry>

After downloading plantuml.jar from http://plantuml.com/download.html you can place it any directory you like, but if that is anywhere other than /usr/lib/ then you will have to edit web.xml accordingly.

Sphinx Virtual Environment

Assuming that you cloned this project's repository to ~/dinodocs:

sudo pip install virtualenv
sudo virtualenv /var/opt/sphinx
sudo source /var/opt/sphinx/bin/activate
sudo pip install -r ~/dinodocs/WebContent/requirements.txt
sudo echo export SPHINX_VIRTUALENV=/var/opt/sphinx/bin/activate > /etc/profile.d/sphinx.sh

If you cloned this repository somewhere else, change the path to requirements.txt in the pip install -r ... command accordingly.

Note that you can create the Sphinx virtual environment where ever you like, so long as the process running dinodocs can access it, and give it any name. If you create this virtual environment anywhere other than /var/opt/sphinx/, however, you will have to change the value of the SPHINX_VIRTUALENV setting in web.xml accordingly:

<env-entry>
 <description>Path to Python virtual environment activation script</description>
 <!-- The Python virtual environment referenced here must include the following 
  packages: sphinx, sphinx-autobuild and sphinxcontrib-plantuml -->
 <env-entry-name>SPHINX_VIRTUALENV</env-entry-name>
 <env-entry-type>java.lang.String</env-entry-type>
 <env-entry-value>/var/opt/sphinx/bin/activate</env-entry-value>
</env-entry>
<env-entry>

Projects Directory

You must specify a directory in web.xml that Dinodocs will use as its working storage. The servlet container process must have read and write access to the specified directory.

<env-entry>
   <description>Path to projects directory</description>
   <env-entry-name>DINODOCS_PROJECTS_DIR</env-entry-name>
   <env-entry-type>java.lang.String</env-entry-type>
   <env-entry-value>/var/opt/dinodocs/projects</env-entry-value>
</env-entry>

This may be a relative or absolute path name. If relative, it will be resolved against the dinodocs servlet's webapp directory. This must agree with the value configured for DINODOCS_PROJECTS_URI!

Projects URL

In order to serve the output built by Dinodocs you must specify the base URL by which the projects directory -- see Projects Directory -- can be accessed.

<env-entry>
  <description>Base URL for project output</description>
  <env-entry-name>DINODOCS_PROJECTS_URL</env-entry-name>
  <env-entry-type>java.lang.String</env-entry-type>
  <env-entry-value>file:///var/opt/dinodocs/projects/</env-entry-value>
</env-entry>

This may be a relative URI or absolute URL. If relative, it will be resolved against the dinodocs servlet request's URL. This must agree with the value configured for DINODOCS_PROJECTS_DIR!

Operation

The servlets in dinodocs.war perform four distinct operations depending on the URL and HTTP request method by which they are invoked. To understand how each of these operations work, here is the terminology used in the source code and documentation comments:

Terminology:

Project Directory

A directory containing working directories corresponding to a set of branches in a SCM repository. The name of the project will be based on the URL of the remote repository specified in a commit hook invocation.

Branch Directory

Each project directory contains a set of sub-directories corresponding to various branches in the remote repository. For projects based on SCM systems like Subversion, each project directory will contain only a single sub-directory named branch. Note that a project directory created using a commit hook will typically contain only a single branch directory, but it is possible to pass query string parameters that specify particular branches for SCM systems that support this functionality.

Project Descriptor

Each branch sub-directory contains a project.xml file that stores the parameters necessary to drive the response to HTML GET and PUT requests.

Source Directory

Each branch sub-directory contains a source sub-directory that is the actual local working directory for the corresponding branch of the project's SCM repository. I.e. the source sub-directory is the destination of a particular Git clone or pull invocation, a particular Subversion checkout invocation etc.

Build Directory

The script used by dinodocs to generate HTML and PDF output places all output in a sub-directory named build under the corresponding branch directory (i.e. beside that branch's source directory).

POST

There are two servlet entry points for handling POST requests. One handles "web hook" requests from a GitLab server. The other handles post-commit hook invocations from Subversion.

GitLab

When invoked via POST, the GitLabBuilder servlet assumes that it is being called by a GitLab server as a commit web hook while SVNBuilder assumes it is being invoked by a Subversion post-commit hook. For each such commit hook invocation, the corresponding servlet:

  • Creates a local project directory whose name is based on the value of the remote repository URL specified in the JSON request message (see below)

  • Creates a "master" branch sub-directory for Git repositores or a "branch" sub-directory for Subversion repositories.

  • Pulls the latest version of the specified repository into a "source" sub-directory of the branch directory

  • If the "source" directory contains a file named Doxyfile, invokes doxygen -- but note that this requires a few specific options to be set in Doxyfile as described below

  • If the "source" directory contains a file named conf.py, generates the Sphinx HTML and PDF outputs using corresponding invocations of sphinx-build

Here is an example of a typical web hook request message from GitLab:

{"object_kind":"push",
 "before":"2bd12191f8bb991fa98a469ebfa7f6a1279e2b14",
 "after":"b4a9cfad235f000f6f53155a1ff01e70ff17b3d4",
 "ref":"refs/heads/master",
 "checkout_sha":"b4a9cfad235f000f6f53155a1ff01e70ff17b3d4",
 "message":null,
 "user_id":1,
 "user_name":"jdoe",
 "user_email":"jdoe@example.com",
 "project_id":2,
 "repository":{"name":"sphinx-examples",
               "url":"git@gitlab.example.com:root/sphinx-examples.git",
               "description":"Sphinx examples",
               "homepage":"http://gitlab.example.com/root/sphinx-examples",
               "git_http_url":"http://gitlab.example.com/root/sphinx-examples.git",
               "git_ssh_url":"git@gitlab.example.com:root/sphinx-examples.git",
               "visibility_level":20},
 "commits":[{"id":"b4a9cfad235f000f6f53155a1ff01e70ff17b3d4",
             "message":"include :label: parameter in discussion of .. math:: directive\n",
             "timestamp":"2015-11-12T02:27:20-08:00",
             "url":"http://gitlab.example.com/root/sphinx-examples/commit/b4a9cfad235f000f6f53155a1ff01e70ff17b3d4",
             "author":{"name":"John Doe","email":"jdoe@example.com"}}],
 "total_commits_count":1}

Subversion

The SVNBuilder entry point servlet expects a POST request containing a single JSON object. That object must contain three attributes corresponding to the three arguments passed to a post-commit hook by svn.

Here is and example of a post-commit hook bash script that meets the criteria expected by SVNBuilder:

#!/bin/sh

# svn post-commit hook script

REPOS="$1"
REV="$2"
TXN_NAME="$3"

curl -X POST -H "Content-Type: application/json" -d "{\"repository\":\"$SVN_URL$REPOS\",\"revision\":\"$REV\",\"transaction\":\"$TXN_NAME\"}'" $DINODOCS_URL

The preceding script assumes that the $SVNURL environment variable will be set to the prefix to prepend to the value passed by svn commit as the first argument to post-commit to turn it into a remote repository URL. It also assumes that $DINODOCS_URL will be set to the URL for the SVNBuilder entry point for your dinodocs server.

For example, in a simple svnserver based configuration, you might put something like this in your repository's conf/hooks-env file:

[post-commit]
SVN_URL=svn://<your svnserver host name>
DINODOCS_URL=http://<your dinodocs server host name>/dinodocs/SVNBuilder

After replacing and with the real values for your set up, each time you commit changes to your Subversion repository it should send a HTML POST request to dinodocs with a JSON message something like:

{"repository":"svn:<your svnserver host name>/<your repo path>","revision":"<current revision>","transcation","<current transaction name>"}

Caveats

GitLabBuilder only pays attention to the contents of the "repository" object, while ignoring the specific "commits." It processes the "repository" information either by cloning or pulling the latest version of the "master" branch from the "origin" repository, as necessary, specified by the value of the git\_http\_url attribute into a sub-directory of the local builds directory.

Similarly, SVNBuilder ignores the "revision" and "transaction" JSON attributes and simply checks out the latest sources each time it is invoked.

TODO: This implies that GitLabBuilder cannot be used effectively as part of a work flow that relies on more than one active branch. Consider using the "commits" array to build the particular version(s) that actually triggered the web hook. This will require a much more complex set of mechanisms for mapping and managing the contents of the projects directory.

GitLabBuilder bases the local directory name on the remote repository's URL. This means that your repository naming conventions must conform to the requirements of directory names in the Linux file system.

TODO: Consider generating unique local directory names, decoupling them from the remote repository names or URL's.

GET

The Projects servlet responds to GET requests with a JSON array of objects, each of which corresponds to the generated output for one of the repositories that has been built by any of the "builder" servlets. Each such JSON object contains:

  • The name of the project

  • An array of "branches", each of which corresponds to the contents of a branch directory within the given project

Each branch object contains:

  • The time stamp at which the source files were built

  • The remote repository home page URL

  • The URL's of any index.html found by searching in the project's output directory

  • The URL of the project's PDF output file

For example:

[{"name":"root-dinodocs-docs",
  "branches":[{"name":"master",
               "lastBuilt":"2015-12-13T18:21:28.028-08:00",
               "homepage":"http://ozymandias/root/dinodocs-docs",
               "html":["http://ozymandias:8000/dinodocs/projects/root-dinodocs-docs/master/build/html/index.html",
                       "http://ozymandias:8000/dinodocs/projects/root-dinodocs-docs/master/build/api/index.html"],
               "pdf":["http://ozymandias:8000/dinodocs/projects/root-dinodocs-docs/master/build/latex/graphviz-ba1594f423fdf362e6a78c0cea924fab46ec1a43.pdf",
                      "http://ozymandias:8000/dinodocs/projects/root-dinodocs-docs/master/build/latex/graphviz-84a80a84671b46807903981bf63cc9a93b3f6076.pdf",
                      "http://ozymandias:8000/dinodocs/projects/root-dinodocs-docs/master/build/latex/graphviz-1e3f6fafc188871ea0d6ac2ca4942d8252c549bc.pdf",
                      "http://ozymandias:8000/dinodocs/projects/root-dinodocs-docs/master/build/latex/dinodocs.pdf",
                      "http://ozymandias:8000/dinodocs/projects/root-dinodocs-docs/master/build/latex/graphviz-bd5485ce64a53d721c1c47ff2b6e56ccd3112831.pdf",
                      "http://ozymandias:8000/dinodocs/projects/root-dinodocs-docs/master/build/latex/graphviz-71949a57b3c51cc30d500c7e8dc37af2db8fb798.pdf"]}]}]

TODO: refactor home page URL to be at the project level.

DELETE

Projects handles HTTP DELETE requests by deleting the branch(es) directory specified by the (required) "projectName" and (optional) "branchName" query string parameters. Specifying only a project name deletes the entire project while also specifying a branch name deletes only that branch directory.

For example, the following jQuery-based code will delete a project directory from the server then refresh the web page on which it was invoked:

function deleteProject(projectName) {

    $.ajax({

        url : "`Projects`?projectName=" + encodeURI(projectName),
        type : "DELETE",

        success : function(data) {

            document.location.reload(true);

        },

        error : function(xhr, status, error) {

            alert("Error deleting " + projectName);

        }

    });

}

The same function could be used to delete only a single branch by adding a "branchName" parameter to the query string in the URL passed to JQuery's ajax function.

PUT

TODO: Currently, neither Projects nor any of the "builder" servlets support HTML PUT requests. The plan is to add the ability to force a rebuild of a given branch directory's output by sending PUT requests to the Projects servlet. This will require adding more meta-data in the project descriptor denoting which SCM to poll.

Doxygen

As noted previously, the dinodocs servlet will look for a file named Doxyfile among the files pulled from GitLab for your projct. If present, it is assumed to be a Doxygen configuration file. There are two general approaches to making your Doxygen based documentation compatible with dinodocs.

Here are the guidelines common to both approaches:

  1. INPUT must specify the location of your source code within your project's repository, using a path name that is relative to the root of your repository, (e.g. "src," "java/src," "include" etc.)

  2. Both OUTPUT_DIRECTORY and IMAGE_PATH must be $(OUTPUT_DIRECTORY) to match the directory structure on the dinodocs server (i.e. dinodocs uses the $OUTPUT_DIRECTORY envirnonment variable to specify the location of generated files when invoking doxygen)

  3. Unless the directory specified by the value for INPUT is perfectly flat, you must set RECURSIVE = YES

"Pure" Doxygen Projects

If you have Doxygen-compatible comments in source files but no files that use Sphinx features, simply omit any conf.py in your project. You can still use Graphviz and PlantUML features in your comment blocks as documented at https://www.stack.nl/~dimitri/doxygen/manual/diagrams.html and http://plantuml.com/doxygen.html

If you wish to embed PlantUML directly in your comment blocks to provide design information beyond the DOT-based "UML" generated automatically by Doxygen, add the following ALIASES to Doxyfile:

ALIASES                = "startuml{1}=\image html \1\n\image latex \1\n\if DontIgnorePlantUMLCode"
ALIASES                += "enduml=\endif"

Append a unique PNG file name parameter to each @startuml invocation, as in:

@startuml{sequence1.png}
participant "Actor 1" as actor1
participant "Actor 2" as actor2
actor1 -> actor2: request
actor1 <- actor2: response
@enduml

See https://github.com/parasaurolophus/dinodocs/blob/master/src/us/rader/dinodocs/utility/CommandQueue.java for an example of placing UML design diagrams directly in javadoc-style, Doxygen-compatible comment blocks using the startuml{1} and enduml aliases.

See http://www.kirkrader.com/dinodocs/ for an example of "pure" Doxygen output generated using dinodocs.

"Hybrid" Sphinx / Doxygen Projects

If you wish to include Doxygen output directly in Sphinx based reStructuredText code, in addition to the common guidelines, above:

  1. Use the same value for PROJECT_NAME as in your Sphinx conf.py file.

  2. Use the same value for PROJECT_NUMBER as in your Sphinx conf.py file.

  3. Set GENERATE_XML = YES since dinodocs relies on the XML output

  4. Set XML_OUTPUT = doxygen as per the expectations of the dinodocs build process

  5. Set the options to generate any other type of output (HTML, LaTeX etc.) to NO since dinodocs will automatically build both HTML and PDF output in any case.

See the Breathe documentation at http://breathe.readthedocs.org/en/stable/ for documentation on the directives you can use in your Sphinx documentation to include content from Doxygen comment blocks.

Instead of the ALIASES described above for the "pure" Doxygen approach, here are the ALIASES that work with "hybrid" projects. (You only need to define the ones listed below that are appropriate for your source code commenting style):

# use \rst ... \endrst to wrap embedded Sphinx directives
# when you *don't* use leading asterisks or slashes on each line
# of your doxygen comment blocks
ALIASES                = "rst=\verbatim embed:rst"
ALIASES                += "endrst=\endverbatim"

# use \rststar ... \endrststar to wrap embedded Sphinx directives
# when you use leading asterisks on each line of your doxygen
# comment blocks
ALIASES                += "rststar=\verbatim embed:rst:leading-asterisk"
ALIASES                += "endrststar=\endverbatim"

# use \rstslash ... \endrstslash to wrap embedded Sphinx directives
# when you use leading slashes on each line of your doxygen
# comment blocks
ALIASES                += "rstslashes=\verbatim embed:rst:leading-slashes"
ALIASES                += "endrsslashes=\endverbatim"

See https://github.com/parasaurolophus/dinodocs-docs/blob/master/src/us/rader/example/CommandQueue.java for an example using \rststar and \endrststar to place UML design diagrams directly in javadoc-style, Doxygen-compatible comment blocks.

See http://breathe.readthedocs.org/en/stable/ for more information on how to use the preceding ALIASES as well as more detailed tutorial information on how to incorporate Doxygen based API documentation into your Sphinx projects.

See https://github.com/parasaurolophus/dinodocs/blob/master/Doxyfile for a complete example of a dinodocs compatible Doxyfile.

See http://www.kirkrader.com/uml/ for an example of "hybrid" Sphinx and Doxygen output generated using dinodocs.