...
 
Commits (35)
image: maven:3.5.0-jdk-8
image: maven:3.5.4-jdk-8
variables:
MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode"
......
# Version 1.0
# Version 1.1.0 (not yet released)
- Support Java 9 and above
- Upgrade DKPro Core to 1.10.0
- Upgrade dependencies
## nlp-machen-plugin
- Fixes #31
# Version 1.0.1
- Upgrade DKPro Core to 1.9.2
## nlp-maven-plugin
- Add logging information for isolated NLP tool test
- Fix loading of DKPro trainer classes
- Add target/classes as resource path during training NLP tools
## quick-pad-tagger
- Fixes #28
# Version 1.0.0
## io-odt
......
[![build status](https://gitlab.com/schrieveslaach/NLPf/badges/master/build.svg)](https://gitlab.com/schrieveslaach/NLPf/commits/master)
[![quality gate](https://sonarcloud.io/api/badges/gate?key=de.schrieveslaach.nlpf%3Amaster)](https://sonarcloud.io/dashboard?id=de.schrieveslaach.nlpf%3Amaster)
[![quality gate](https://sonarcloud.io/api/project_badges/measure?project=de.schrieveslaach.nlpf%3Amaster&metric=alert_status)](https://sonarcloud.io/dashboard?id=de.schrieveslaach.nlpf%3Amaster)
[![License: LGPL v3](https://img.shields.io/badge/License-LGPL%20v3-blue.svg)](https://www.gnu.org/licenses/lgpl-3.0)
[![Maven Central](https://maven-badges.herokuapp.com/maven-central/de.schrieveslaach.nlpf/nlpf-parent/badge.svg)](https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22de.schrieveslaach.nlpf%22)
......@@ -7,23 +7,28 @@
![NLP Lean Programming framework (NLPf)](images/NLPf.svg.png)
As a developer, have you ever wondered how you build your domain-specific Natural Language Processing (NLP) models? For example, you need to write on Information Extraction system which requires a custom set of Named Entity Recognition (NER) models but nowhere in the World Wide Web you are able to find any tool or model. The project _“Towards Effective NLP Application Development”_ provides the required tools in a familiar environment to build and use your custom NLP models very efficiently.
As a developer, have you ever wondered how you build your domain-specific Natural Language Processing (NLP) models? For example, you need to write on Information Extraction system which requires a custom set of Named Entity Recognition (NER) models but nowhere in the World Wide Web you are able to find any tool or model. The project _“Towards Effective NLP Application Development”_ provides the required tools in a familiar environment to build and use your custom NLP models very efficiently.
This project is based on research results of the collaboration [ETL Quadrat](http://www.softwaresysteme.pt-dlr.de/media/content/Infoblatt_ETL_QUADRAT.pdf).
This project is an Open Source rewrite of the previous implementation and uses [DKPro Core™][1] to leverage the benefits to a wide variaty of
projects using DKPro Core™.
Klick on the image below to watch a tutorial on YouTube:
[![NLPf Tutorial](https://img.youtube.com/vi/44UJspVebTA/0.jpg)](https://www.youtube.com/watch?v=44UJspVebTA)
## Project Goals
- Build your domain-specific corpus more effectively
* Effective project management through Maven project management
* Integrated annotation tool support through the Quick Pad Tagger
- Determine your best-performing NLP pipeline through Continuous Integration
- Determine your best-performing NLP pipeline through build automation
* Add NLP tools as Maven dependency and determine the best-performing NLP pipeline by `mvn test`
* Deploy the best-performing NLP pipeline with its Models as Maven dependency into a Maven repository
- Integrate the best-performing NLP pipeline as Maven dependency
Following sections provide exemplary how you can use Maven and the annotation to build your NLP models more efficiently.
Following sections provide an example setup how you can use Maven and the annotation to build your NLP models more efficiently.
Make sure that you meet the [requirements](#requirements).
## Create Your Corpus
......@@ -43,7 +48,7 @@ Create an empty directory and create a `pom.xml` file:
<plugin>
<groupId>de.schrieveslaach.nlpf</groupId>
<artifactId>nlp-maven-plugin</artifactId>
<version>1.0-beta07</version>
<version>1.0.0</version>
<extensions>true</extensions>
</plugin>
</plugins>
......@@ -67,7 +72,7 @@ contains the corresponding reader and writer implementations for DKPro Core™.
<dependency>
<groupId>de.schrieveslaach.nlpf</groupId>
<artifactId>io-odt</artifactId>
<version>1.0-beta07</version>
<version>1.0.0</version>
</dependency>
</dependencies>
</project>
......@@ -77,8 +82,9 @@ Furthermore, you need to place your test documents into `src/test/corpus`.
## Annotate Your Corpus
[Download the Quick Pad Tagger](https://gitlab.com/schrieveslaach/NLPf/builds/artifacts/master/browse?job=integrationTests). After that, open the `quick-pad-tagger-1.0-beta07.exe` file or execute `java -jar quick-pad-tagger-1.0-beta07.jar` to start the annotation tool.
The Quick Pad Tagger has been tested on Linux, Windows 10, and macOS.
[Download the Quick Pad Tagger](https://gitlab.com/schrieveslaach/NLPf/-/jobs/53855551/artifacts/browse/quick-pad-tagger/target/).
After that, open the `quick-pad-tagger-1.0.0.exe` file or execute `java -jar quick-pad-tagger-1.0.0.jar` to start the annotation tool.
The Quick Pad Tagger has been tested on Linux, Windows 8 & 10, and macOS.
Use the [cheat sheet](quick-pad-tagger/Cheat Sheet.md) to look up the controls of the Quick Pad Tagger. Following videos show the capabilities of the annotation tool.
......@@ -111,7 +117,7 @@ dependencies to your `pom.xml` and execute the commands below.
<dependency>
<groupId>de.tudarmstadt.ukp.dkpro.core</groupId>
<artifactId>de.tudarmstadt.ukp.dkpro.core.opennlp-asl</artifactId>
<version>1.9.0</version>
<version>1.9.2</version>
</dependency>
</dependencies>
</project>
......@@ -132,11 +138,12 @@ mvn deploy
NLPf supports multiple NLP frameworks. Following table provides an overview.
| Tool | Maven Artifact Id | Segmenter Training | Named Entity Recognition Trainer | POS Tagging Trainer |
| ---------------- | ----------------------------------------------- | -------------------- | -------------------------------- | -------------------- |
| OpenNLP | `de.tudarmstadt.ukp.dkpro.core.opennlp-asl` | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| Stanford CoreNLP | `de.tudarmstadt.ukp.dkpro.core.stanfordnlp-gpl` | :white_large_square: | :white_check_mark: | :white_check_mark: |
| LingPipe | `de.tudarmstadt.ukp.dkpro.core.lingpipe-gpl` | :white_large_square: | :white_check_mark: | :white_large_square: |
| Tool | Maven Artifact Id | Segmenter Training | Named Entity Recognition Trainer | POS Tagging Trainer |
| ---------------- | ----------------------------------------------- | -------------------- | -------------------------------- | ---------------------------------------- |
| OpenNLP | `de.tudarmstadt.ukp.dkpro.core.opennlp-asl` | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| Stanford CoreNLP | `de.tudarmstadt.ukp.dkpro.core.stanfordnlp-gpl` | :white_large_square: | :white_check_mark: | :white_check_mark: |
| LingPipe | `de.tudarmstadt.ukp.dkpro.core.lingpipe-gpl` | :white_large_square: | :white_check_mark: | :white_large_square: |
| ArkTweet | `de.tudarmstadt.ukp.dkpro.core.arktools-gpl` | :white_large_square: | :white_large_square: | :white_check_mark: (since version 1.1.0) |
If you want to add further support for more trainers to DKPro Core™, see this
[pull request on Github](https://github.com/dkpro/dkpro-core/pull/1080) for
......@@ -228,7 +235,7 @@ an NLP pipeline:
<dependency>
<groupId>de.schrieveslaach.nlpf</groupId>
<artifactId>plumbing</artifactId>
<version>1.0-beta07</version>
<version>1.0.0</version>
</dependency>
<!-- your corpus -->
......@@ -322,5 +329,13 @@ docker run --rm -it \
sonarqube:6.7-alpine
```
# How to Cite?
<p>M. Schreiber, B. Kraft, and A. Zündorf. (2018).
<strong>“NLP Lean Programming Framework: Devel-oping NLP Applications More Effectively”</strong>. In Proceedings of the 2018 Conference of
the North American Chapter of the Association for Computational Linguistics: Demonstrations, p 1-5, New Orleans, Louisiana.
<a href="http://dx.doi.org/10.18653/v1/N18-5001">(pdf)</a> <a href="https://aclanthology.coli.uni-saarland.de/papers/N18-5001/n18-5001.bib">(bib)</a>
</p>
[1]: https://github.com/dkpro/dkpro-core
[2]: https://en.wikipedia.org/wiki/OpenDocument
......@@ -6,7 +6,7 @@
<parent>
<groupId>de.schrieveslaach.nlpf</groupId>
<artifactId>nlpf-parent</artifactId>
<version>1.0-beta08-SNAPSHOT</version>
<version>1.1.0-SNAPSHOT</version>
</parent>
<artifactId>acceptance-tests</artifactId>
......
......@@ -4,7 +4,7 @@
<parent>
<groupId>de.schrieveslaach.nlpf</groupId>
<artifactId>nlpf-parent</artifactId>
<version>1.0-beta08-SNAPSHOT</version>
<version>1.1.0-SNAPSHOT</version>
</parent>
<artifactId>io-odt</artifactId>
......@@ -19,6 +19,13 @@
<groupId>org.apache.odftoolkit</groupId>
<artifactId>simple-odf</artifactId>
<version>0.8.2-incubating</version>
<exclusions>
<exclusion>
<groupId>org.apache.odftoolkit</groupId>
<artifactId>taglets</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
......@@ -81,7 +88,7 @@
<plugin>
<groupId>org.apache.uima</groupId>
<artifactId>jcasgen-maven-plugin</artifactId>
<version>2.10.0</version>
<version>2.10.2</version>
<configuration>
<typeSystemIncludes>
......
......@@ -4,13 +4,18 @@
<parent>
<groupId>de.schrieveslaach.nlpf</groupId>
<artifactId>nlpf-parent</artifactId>
<version>1.0-beta08-SNAPSHOT</version>
<version>1.1.0-SNAPSHOT</version>
</parent>
<artifactId>nlp-maven-plugin</artifactId>
<packaging>maven-plugin</packaging>
<dependencies>
<dependency>
<groupId>net.bytebuddy</groupId>
<artifactId>byte-buddy</artifactId>
</dependency>
<!-- Maven plugin API -->
<dependency>
<groupId>org.apache.maven</groupId>
......@@ -25,6 +30,10 @@
<artifactId>maven-plugin-annotations</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.maven.reporting</groupId>
<artifactId>maven-reporting-impl</artifactId>
</dependency>
<dependency>
<groupId>org.apache.maven</groupId>
<artifactId>maven-archiver</artifactId>
......@@ -154,7 +163,7 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-plugin-plugin</artifactId>
<version>3.3</version>
<version>3.5.2</version>
<configuration>
<goalPrefix>nlp</goalPrefix>
</configuration>
......
......@@ -23,6 +23,8 @@ package de.schrieveslaach.nlpf.maven.plugin;
*/
import com.fasterxml.jackson.databind.ObjectMapper;
import de.schrieveslaach.nlpf.maven.plugin.models.TestResult;
import de.schrieveslaach.nlpf.testing.annotators.MyPosTagger;
import org.apache.maven.plugin.testing.MojoRule;
import org.junit.Rule;
import org.junit.Test;
......@@ -47,7 +49,7 @@ public class IsolatedToolsTestMojoIT extends BaseMojoIT {
rule.executeMojo(testProjectBaseDir, "train");
rule.executeMojo(testProjectBaseDir, "tools-test");
File testDataFile = new File(testProjectBaseDir, "target/isolated-tools-test-data/15DD7719CA9B2ABFD8725C112768F3FFF21CFBE94525919D0DAB41A82B8FFC2EB3A9BDBFEA7A299264D646A55D65326E2990554F00195F0690FD09B8AD088C81.json");
File testDataFile = new File(testProjectBaseDir, "target/isolated-tools-test-data/" + MyPosTagger.DEFAULT_HASH_CODE + ".json");
assertThat(testDataFile, is(anExistingFile()));
ObjectMapper mapper = new ObjectMapper();
......
......@@ -23,7 +23,7 @@ package de.schrieveslaach.nlpf.maven.plugin;
*/
import com.fasterxml.jackson.databind.ObjectMapper;
import de.schrieveslaach.nlpf.maven.plugin.TestResult;
import de.schrieveslaach.nlpf.maven.plugin.models.TestResult;
import org.apache.maven.plugin.testing.MojoRule;
import org.junit.Rule;
import org.junit.Test;
......
......@@ -35,13 +35,13 @@
<dependency>
<groupId>de.tudarmstadt.ukp.dkpro.core</groupId>
<artifactId>de.tudarmstadt.ukp.dkpro.core.io.xmi-asl</artifactId>
<version>1.9.0</version>
<version>1.10.0</version>
</dependency>
<dependency>
<groupId>de.tudarmstadt.ukp.dkpro.core</groupId>
<artifactId>de.tudarmstadt.ukp.dkpro.core.opennlp-asl</artifactId>
<version>1.9.0</version>
<version>1.10.0</version>
</dependency>
</dependencies>
</project>
......@@ -24,6 +24,7 @@ package de.schrieveslaach.nlpf.maven.plugin;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.Sets;
import de.schrieveslaach.nlpf.maven.plugin.models.TestResult;
import de.schrieveslaach.nlpf.maven.plugin.service.AnalysisEngineDescriptionService;
import de.schrieveslaach.nlpf.maven.plugin.service.ClassLoaderService;
import de.schrieveslaach.nlpf.maven.plugin.service.CollectionReaderDescriptionService;
......
......@@ -23,6 +23,7 @@ package de.schrieveslaach.nlpf.maven.plugin;
*/
import com.google.common.reflect.ClassPath;
import de.schrieveslaach.nlpf.maven.plugin.service.ClassLoaderService;
import de.schrieveslaach.nlpf.maven.plugin.service.DirectoryService;
import de.schrieveslaach.nlpf.plumbing.util.Uima;
import lombok.SneakyThrows;
......@@ -45,7 +46,9 @@ import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLClassLoader;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
import java.util.Set;
import java.util.stream.Collectors;
@Mojo(name = "execute-engine-factories", requiresDependencyResolution = ResolutionScope.TEST)
......@@ -57,17 +60,22 @@ public class ExecuteEngineFactoryMojo extends AbstractMojo {
@Inject
private DirectoryService directoryService;
@Inject
private ClassLoaderService classLoaderService;
@Override
@SneakyThrows({IOException.class})
public void execute() throws MojoExecutionException, MojoFailureException {
Uima.swallowLogging();
ClassLoader classLoader = getTestClasspathLoader();
Thread.currentThread().setContextClassLoader(classLoader);
List<? extends Class<?>> engineFactoryClasses = ClassPath.from(classLoader).getAllClasses().stream()
Set<? extends Class<?>> engineFactoryClasses = ClassPath.from(classLoader).getAllClasses().stream()
.filter(ci -> ci.getName().endsWith("EngineFactory"))
.map(ClassPath.ClassInfo::load)
.collect(Collectors.toList());
.collect(Collectors.toSet());
getLog().debug("found " + engineFactoryClasses.size() + " engine factories");
......@@ -79,7 +87,7 @@ public class ExecuteEngineFactoryMojo extends AbstractMojo {
}
}
private List<AnalysisEngineDescription> createAnalysisEngineDescriptions(List<? extends Class<?>> engineFactoryClasses) {
private List<AnalysisEngineDescription> createAnalysisEngineDescriptions(Collection<? extends Class<?>> engineFactoryClasses) {
List<AnalysisEngineDescription> engineDescriptions = new ArrayList<>();
for (Class<?> engineFactoryClass : engineFactoryClasses) {
......@@ -119,6 +127,8 @@ public class ExecuteEngineFactoryMojo extends AbstractMojo {
@SneakyThrows(MalformedURLException.class)
private ClassLoader getTestClasspathLoader() throws MojoExecutionException {
try {
ClassLoader classLoader = classLoaderService.getDependencyClassLoader();
List<String> testClasspathElements = project.getTestClasspathElements();
URL[] urls = new URL[testClasspathElements.size()];
int i = 0;
......@@ -127,7 +137,8 @@ public class ExecuteEngineFactoryMojo extends AbstractMojo {
urls[i] = new File(entry).toURI().toURL();
i++;
}
return new URLClassLoader(urls, getClass().getClassLoader());
return new URLClassLoader(urls, classLoader);
} catch (DependencyResolutionRequiredException e) {
throw new MojoExecutionException("Could not resolve test classpath elements", e);
}
......
......@@ -24,7 +24,9 @@ package de.schrieveslaach.nlpf.maven.plugin;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializationFeature;
import de.schrieveslaach.nlpf.maven.plugin.models.TestResult;
import de.schrieveslaach.nlpf.maven.plugin.reader.ClearTypesCombinationReader;
import de.tudarmstadt.ukp.dkpro.core.api.parameter.ComponentParameters;
import lombok.AllArgsConstructor;
import lombok.SneakyThrows;
import org.apache.maven.plugin.MojoExecutionException;
......@@ -33,6 +35,7 @@ import org.apache.maven.plugins.annotations.ResolutionScope;
import org.apache.uima.UIMAException;
import org.apache.uima.analysis_engine.AnalysisEngineDescription;
import org.apache.uima.collection.CollectionReaderDescription;
import org.apache.uima.fit.descriptor.ResourceMetaData;
import org.apache.uima.fit.descriptor.TypeCapability;
import org.apache.uima.jcas.JCas;
......@@ -62,6 +65,8 @@ public class IsolatedToolsTestMojo extends AbstractTestMojo {
@SneakyThrows({MojoExecutionException.class, UIMAException.class})
private TestResultWrapper testNlpTool(AnalysisEngineDescription engineDescription, List<JCas> testData) {
logNlpToolsInfo(engineDescription);
String[] outputTypes = getOutputTypesOfDescription(engineDescription);
CollectionReaderDescription testReaderDescription = collectionReaderDescriptionService.createTestReaderDescriptions(
......@@ -79,6 +84,26 @@ public class IsolatedToolsTestMojo extends AbstractTestMojo {
return new TestResultWrapper(testResult, engineDescription);
}
private void logNlpToolsInfo(AnalysisEngineDescription engineDescription) {
Class<?> implClass = analysisEngineService.loadAnalysisEngineImplementation(engineDescription);
String name;
ResourceMetaData metaData = implClass.getAnnotation(ResourceMetaData.class);
if (metaData != null) {
name = metaData.name();
} else {
name = engineDescription.getAnnotatorImplementationName();
}
String variant = null;
Object parameterVariant = engineDescription.getAnalysisEngineMetaData().getConfigurationParameterSettings().getParameterValue(ComponentParameters.PARAM_VARIANT);
if (parameterVariant != null) {
variant = parameterVariant.toString();
}
getLog().info(String.format("Testing %s with variant %s in isolation", name, variant));
}
private String[] getOutputTypesOfDescription(AnalysisEngineDescription engineDescription) {
Class<?> implementationClass = analysisEngineService.loadAnalysisEngineImplementation(engineDescription);
TypeCapability typeCapability = implementationClass.getAnnotation(TypeCapability.class);
......
package de.schrieveslaach.nlpf.maven.plugin;
/*-
* ========================LICENSE_START=================================
* nlp-maven-plugin
* %%
* Copyright (C) 2017 - 2018 Schrieveslaach
* %%
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Lesser General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Lesser Public License for more details.
*
* You should have received a copy of the GNU General Lesser Public
* License along with this program. If not, see
* <http://www.gnu.org/licenses/lgpl-3.0.html>.
* =========================LICENSE_END==================================
*/
import com.fasterxml.jackson.databind.ObjectMapper;
import de.schrieveslaach.nlpf.maven.plugin.models.Measure;
import de.schrieveslaach.nlpf.maven.plugin.models.TestResult;
import de.schrieveslaach.nlpf.maven.plugin.service.DirectoryService;
import lombok.Cleanup;
import org.apache.maven.doxia.sink.Sink;
import org.apache.maven.plugins.annotations.LifecyclePhase;
import org.apache.maven.plugins.annotations.Mojo;
import org.apache.maven.plugins.annotations.ResolutionScope;
import org.apache.maven.reporting.AbstractMavenReport;
import org.apache.maven.reporting.MavenReportException;
import org.apache.uima.UIMAFramework;
import org.apache.uima.analysis_engine.AnalysisEngineDescription;
import org.apache.uima.resource.metadata.NameValuePair;
import org.apache.uima.util.InvalidXMLException;
import org.apache.uima.util.XMLInputSource;
import org.apache.uima.util.XMLParser;
import javax.inject.Inject;
import java.io.File;
import java.io.IOException;
import java.util.HashMap;
import java.util.Locale;
import java.util.Map;
@Mojo(name = "tools-report",
defaultPhase = LifecyclePhase.SITE,
requiresDependencyResolution = ResolutionScope.COMPILE,
threadSafe = true)
public class IsolatedToolsTestReport extends AbstractMavenReport {
@Inject
protected DirectoryService directoryService;
@Override
protected void executeReport(Locale locale) throws MavenReportException {
Sink sink = getSink();
// Page title
sink.head();
sink.title();
sink.text("Isolated Tools Test Report for " + project.getName() + " " + project.getVersion());
sink.title_();
sink.head_();
sink.body();
Map<TestResult, AnalysisEngineDescription> resultsAndEngineDescriptions = loadTestResultsAndEngineDescriptions();
for (Map.Entry<TestResult, AnalysisEngineDescription> e : resultsAndEngineDescriptions.entrySet()) {
sink.section1();
TestResult testResult = e.getKey();
sink.sectionTitle1();
sink.text(testResult.toString());
sink.sectionTitle1_();
writeMeasureSection(sink, testResult);
writeParametersSections(sink, e);
sink.section1_();
}
sink.body_();
}
private void writeParametersSections(Sink sink, Map.Entry<TestResult, AnalysisEngineDescription> e) {
sink.section2();
sink.sectionTitle2();
sink.text("Parameters");
sink.sectionTitle2_();
sink.table();
sink.tableCaption();
sink.tableHeaderCell();
sink.text("Parameter Name");
sink.tableHeaderCell_();
sink.tableHeaderCell();
sink.text("Value");
sink.tableHeaderCell_();
sink.tableCaption_();
AnalysisEngineDescription engineDescription = e.getValue();
for (NameValuePair nvp : engineDescription.getAnalysisEngineMetaData()
.getConfigurationParameterSettings()
.getParameterSettings()) {
sink.tableRow();
sink.tableCell();
sink.text(nvp.getName());
sink.tableCell_();
sink.tableCell();
sink.text(nvp.getValue().toString());
sink.tableCell_();
sink.tableRow_();
}
sink.table_();
sink.section2_();
}
private void writeMeasureSection(Sink sink, TestResult testResult) {
sink.section2();
sink.sectionTitle2();
sink.text("Measure");
sink.sectionTitle2_();
sink.table();
sink.tableCaption();
sink.tableHeaderCell();
sink.text("NLP Tool");
sink.tableHeaderCell_();
sink.tableHeaderCell();
sink.text("Output Type");
sink.tableHeaderCell_();
sink.tableHeaderCell();
sink.text("Variant");
sink.tableHeaderCell_();
sink.tableHeaderCell();
sink.text("F-Measure");
sink.tableHeaderCell_();
sink.tableCaption_();
for (Measure m : testResult.getMeasures()) {
sink.tableRow();
sink.tableCell();
sink.text(m.getAnalysisEngineName());
sink.tableCell_();
sink.tableCell();
sink.text(m.getOutputType());
sink.tableCell_();
sink.tableCell();
sink.text(m.getVariant());
sink.tableCell_();
sink.tableCell();
sink.text(String.format("%.3f", m.getFScore().getFMeasure()));
sink.tableCell_();
sink.tableRow_();
}
sink.table_();
sink.section2_();
}
private Map<TestResult, AnalysisEngineDescription> loadTestResultsAndEngineDescriptions() throws MavenReportException {
ObjectMapper mapper = new ObjectMapper();
XMLParser xmlParser = UIMAFramework.getXMLParser();
Map<TestResult, AnalysisEngineDescription> resultsAndDescriptions = new HashMap<>();
for (File testDataFile : directoryService.getIsolatedToolsTestDataDirectory().listFiles()) {
TestResult testResult;
try {
testResult = mapper.readValue(testDataFile, TestResult.class);
} catch (IOException e) {
getLog().warn("Cannot load test data of " + testDataFile);
continue;
}
File descriptorFile = new File(directoryService.getDescriptorsDirectory(),
testResult.getPipelineDescriptorNames().iterator().next());
try {
@Cleanup
XMLInputSource s = new XMLInputSource(descriptorFile);
resultsAndDescriptions.put(testResult, xmlParser.parseAnalysisEngineDescription(s));
} catch (IOException | InvalidXMLException e) {
throw new MavenReportException("Could not read engine description from " + descriptorFile, e);
}
}
return resultsAndDescriptions;
}
@Override
public String getOutputName() {
return "isolated-tools-test-report";
}
@Override
public String getName(Locale locale) {
return "Isolated Tools Test Report";
}
@Override
public String getDescription(Locale locale) {
return "This report summarizes the results of the isolated tools test goal";
}
}
......@@ -23,6 +23,7 @@ package de.schrieveslaach.nlpf.maven.plugin;
*/
import com.fasterxml.jackson.databind.ObjectMapper;
import de.schrieveslaach.nlpf.maven.plugin.models.TestResult;
import de.schrieveslaach.nlpf.maven.plugin.service.DirectoryService;
import de.schrieveslaach.nlpf.plumbing.util.Uima;
import org.apache.commons.io.FileUtils;
......
......@@ -28,6 +28,7 @@ import com.google.common.collect.HashBasedTable;
import com.google.common.collect.Lists;
import com.google.common.collect.Sets;
import com.google.common.collect.Table;
import de.schrieveslaach.nlpf.maven.plugin.models.TestResult;
import de.schrieveslaach.nlpf.maven.plugin.reader.ClearTypesCombinationReader;
import de.schrieveslaach.nlpf.maven.plugin.service.DirectoryService;
import de.schrieveslaach.nlpf.plumbing.util.AnalysisEngineDescriptionNode;
......@@ -58,6 +59,7 @@ import java.util.Collections;
import java.util.Deque;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import java.util.Set;
......@@ -172,13 +174,15 @@ public class PipelinesTestMojo extends AbstractTestMojo {
List<List<AnalysisEngineDescription>> buildPipelineCombinations() throws ResourceInitializationException, MojoExecutionException {
List<AnalysisEngineDescription> annotatorDescriptions = loadEngineDescriptions();
// Groups all engine description by the output value of @TypeCapability
List<List<EngineDescriptionElement>> groupingByOutputCapability = new ArrayList<>(annotatorDescriptions.stream()
Map<EngineDescriptionElement, List<EngineDescriptionElement>> m = annotatorDescriptions.stream()
.map(EngineDescriptionElement::new)
.collect(Collectors.groupingBy(
Function.identity(),
Collectors.toList()
)).values());
Collectors.<EngineDescriptionElement>toList()
));
// Groups all engine description by the output value of @TypeCapability
List<List<EngineDescriptionElement>> groupingByOutputCapability = new ArrayList<>(m.values());
List<List<AnalysisEngineDescription>> pipelines = new ArrayList<>();
buildAllPipelineCombinations(groupingByOutputCapability, 0, pipelines, new LinkedList<>());
......
package de.schrieveslaach.nlpf.maven.plugin.models;
/*-
* ========================LICENSE_START=================================
* nlp-maven-plugin
* %%
* Copyright (C) 2017 - 2018 Schrieveslaach
* %%
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Lesser General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Lesser Public License for more details.
*
* You should have received a copy of the GNU General Lesser Public
* License along with this program. If not, see
* <http://www.gnu.org/licenses/lgpl-3.0.html>.
* =========================LICENSE_END==================================
*/
import de.tudarmstadt.ukp.dkpro.core.eval.measure.FMeasure;
import lombok.AllArgsConstructor;
import lombok.Getter;
@AllArgsConstructor
public class FScore {
@Getter
private final double precision;
@Getter
private final double recall;
@Getter
private final double fMeasure;
FScore(FMeasure fMeasure) {
this.fMeasure = fMeasure.getFMeasure();
this.recall = fMeasure.getRecall();
this.precision = fMeasure.getPrecision();
}
}
package de.schrieveslaach.nlpf.maven.plugin.models;
/*-
* ========================LICENSE_START=================================
* nlp-maven-plugin
* %%
* Copyright (C) 2017 - 2018 Schrieveslaach
* %%
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Lesser General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Lesser Public License for more details.
*
* You should have received a copy of the GNU General Lesser Public
* License along with this program. If not, see
* <http://www.gnu.org/licenses/lgpl-3.0.html>.
* =========================LICENSE_END==================================
*/
import lombok.AllArgsConstructor;
import lombok.EqualsAndHashCode;
import lombok.Getter;
import lombok.NonNull;
@AllArgsConstructor
@EqualsAndHashCode(exclude = "fScore")
public class Measure implements Comparable<Measure> {
@Getter
@NonNull
private final String analysisEngineName;
@Getter
@NonNull
private final String outputType;
@Getter
private final String variant;
@Getter
@NonNull
private FScore fScore;
@Override
public int compareTo(Measure measure) {
int cmp = outputType.compareTo(measure.outputType);
if (cmp == 0) {
if (variant != null && measure.variant != null) {
cmp = variant.compareTo(measure.variant);
} else if (variant == null && measure.variant != null) {
cmp = -1;
} else if (variant != null && measure.variant == null) {
cmp = 1;
}
}
if (cmp == 0) {
cmp = analysisEngineName.compareTo(measure.analysisEngineName);
}
return cmp;
}
}
package de.schrieveslaach.nlpf.maven.plugin;
package de.schrieveslaach.nlpf.maven.plugin.models;
/*-
* ========================LICENSE_START=================================
......@@ -36,10 +36,7 @@ import com.fasterxml.jackson.databind.node.ArrayNode;
import com.fasterxml.jackson.databind.node.ObjectNode;
import com.google.common.collect.ImmutableList;
import de.tudarmstadt.ukp.dkpro.core.eval.measure.FMeasure;
import lombok.AllArgsConstructor;
import lombok.EqualsAndHashCode;
import lombok.Getter;
import lombok.NonNull;
import org.apache.uima.analysis_engine.AnalysisEngineDescription;
import java.io.IOException;
......@@ -47,6 +44,7 @@ import java.util.ArrayList;
import java.util.List;
import java.util.SortedSet;
import java.util.TreeSet;
import java.util.stream.Collectors;
import static de.schrieveslaach.nlpf.plumbing.util.AnalysisEngineDescriptionUtil.hash;
import static java.util.Arrays.asList;
......@@ -54,7 +52,7 @@ import static java.util.Arrays.asList;
@JsonDeserialize(using = TestResult.TestResultDeserializer.class)
@JsonSerialize(using = TestResult.TestResultSerializer.class)
@EqualsAndHashCode
class TestResult implements Comparable<TestResult> {
public class TestResult implements Comparable<TestResult> {
private final SortedSet<Measure> measures = new TreeSet<>();
......@@ -63,11 +61,11 @@ class TestResult implements Comparable<TestResult> {
private TestResult() {
}
TestResult(AnalysisEngineDescription engineDescription) {
public TestResult(AnalysisEngineDescription engineDescription) {
this(asList(engineDescription));
}
TestResult(Iterable<AnalysisEngineDescription> pipeline) {
public TestResult(Iterable<AnalysisEngineDescription> pipeline) {
for (AnalysisEngineDescription aed : pipeline) {
pipelineDescriptors.add(hash(aed) + ".xml");
}
......@@ -81,6 +79,18 @@ class TestResult implements Comparable<TestResult> {
return ImmutableList.copyOf(pipelineDescriptors);
}
public ImmutableList<Measure> getMeasures() {
return ImmutableList.copyOf(measures);
}
@Override
public String toString() {
return measures.stream()
.map(measure -> measure.getAnalysisEngineName())
.distinct()
.collect(Collectors.joining(", "));
}
@Override
public int compareTo(TestResult other) {
int compare = Integer.compare(measures.size(), other.measures.size());
......@@ -92,12 +102,12 @@ class TestResult implements Comparable<TestResult> {
return -1 * Double.compare(calculateHarmonicMean(), other.calculateHarmonicMean());
}
private double calculateHarmonicMean() {
public double calculateHarmonicMean() {
double sum = 0.0;
int cnt = 0;
for (Measure measure : measures) {
double fMeasure = measure.fScore.fMeasure;
double fMeasure = measure.getFScore().getFMeasure();
if (fMeasure < 0) {
// punish measures without any hits
fMeasure = 1 / 1_000_000.0;
......@@ -113,60 +123,6 @@ class TestResult implements Comparable<TestResult> {
return cnt / sum;
}
@AllArgsConstructor
private static class FScore {
@Getter
private final double precision;
@Getter
private final double recall;
@Getter
private final double fMeasure;
private FScore(FMeasure fMeasure) {
this.fMeasure = fMeasure.getFMeasure();
this.recall = fMeasure.getRecall();
this.precision = fMeasure.getPrecision();
}
}
@AllArgsConstructor
@EqualsAndHashCode(exclude = "fScore")
private static class Measure implements Comparable<Measure> {
@NonNull
private final String analysisEngineName;
@NonNull
private final String outputType;
private final String variant;
@NonNull
private FScore fScore;
@Override
public int compareTo(Measure measure) {
int cmp = outputType.compareTo(measure.outputType);
if (cmp == 0) {
if (variant != null && measure.variant != null) {
cmp = variant.compareTo(measure.variant);
} else if (variant == null && measure.variant != null) {
cmp = -1;
} else if (variant != null && measure.variant == null) {
cmp = 1;
}
}
if (cmp == 0) {
cmp = analysisEngineName.compareTo(measure.analysisEngineName);
}
return cmp;
}
}
public static class TestResultSerializer extends JsonSerializer<TestResult> {
@Override
......@@ -186,22 +142,22 @@ class TestResult implements Comparable<TestResult> {
jsonGenerator.writeStartObject();
jsonGenerator.writeFieldName("analysisEngineName");
jsonGenerator.writeString(measureEntry.analysisEngineName);
jsonGenerator.writeString(measureEntry.getAnalysisEngineName());
jsonGenerator.writeFieldName("outputType");
jsonGenerator.writeString(measureEntry.outputType);
jsonGenerator.writeString(measureEntry.getOutputType());
jsonGenerator.writeFieldName("variant");
jsonGenerator.writeString(measureEntry.variant);
jsonGenerator.writeString(measureEntry.getVariant());
jsonGenerator.writeFieldName("measure");
jsonGenerator.writeStartObject();
jsonGenerator.writeFieldName("f-measure");
jsonGenerator.writeNumber(measureEntry.fScore.getFMeasure());
jsonGenerator.writeNumber(measureEntry.getFScore().getFMeasure());
jsonGenerator.writeFieldName("precision");
jsonGenerator.writeNumber(measureEntry.fScore.getPrecision());
jsonGenerator.writeNumber(measureEntry.getFScore().getPrecision());
jsonGenerator.writeFieldName("recall");
jsonGenerator.writeNumber(measureEntry.fScore.getRecall());
jsonGenerator.writeNumber(measureEntry.getFScore().getRecall());
jsonGenerator.writeEndObject();
jsonGenerator.writeEndObject();
......
......@@ -23,12 +23,14 @@ package de.schrieveslaach.nlpf.maven.plugin.service;
*/
import lombok.Getter;
import lombok.SneakyThrows;
import org.apache.maven.plugin.logging.Log;
import org.apache.maven.plugin.logging.SystemStreamLog;
import org.apache.maven.project.MavenProject;
import javax.inject.Inject;
import javax.inject.Singleton;
import java.io.File;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLClassLoader;
......@@ -56,6 +58,7 @@ public class ClassLoaderService {
*
* @return
*/
@SneakyThrows(MalformedURLException.class)
private ClassLoader initDependencyClassLoader() {
List<URL> paths = new ArrayList<>();
......@@ -69,6 +72,8 @@ public class ClassLoaderService {
}
});
paths.add(new File(mavenProject.getBuild().getOutputDirectory()).toURI().toURL());
return new URLClassLoader(paths.toArray(new URL[paths.size()]), getClass().getClassLoader());
}
......
......@@ -30,9 +30,11 @@
<configuration>
<phases>
<process-resources>
org.apache.maven.plugins:maven-resources-plugin:resources,
de.schrieveslaach.nlpf:nlp-maven-plugin:validate-annotations
</process-resources>
<compile>
org.apache.maven.plugins:maven-compiler-plugin:compile,
de.schrieveslaach.nlpf:nlp-maven-plugin:train
</compile>
<test-compile>
......
......@@ -23,6 +23,7 @@ package de.schrieveslaach.nlpf.maven.plugin;
*/
import de.schrieveslaach.nlpf.maven.plugin.factories.CustomEngineFactory;
import de.schrieveslaach.nlpf.maven.plugin.service.ClassLoaderService;
import de.schrieveslaach.nlpf.maven.plugin.service.DirectoryService;
import de.schrieveslaach.nlpf.testing.annotators.MyPosTagger;
import de.schrieveslaach.nlpf.testing.annotators.MySegmenter;
......@@ -30,6 +31,7 @@ import org.apache.maven.artifact.Artifact;
import org.apache.maven.artifact.DependencyResolutionRequiredException;
import org.apache.maven.plugin.MojoExecutionException;
import org.apache.maven.project.MavenProject;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
......@@ -63,9 +65,17 @@ public class ExecuteEngineFactoryMojoTest {
@Mock
private DirectoryService directoryService;
@Mock
private ClassLoaderService classLoaderService;
@Rule
public final ExpectedException expectedException = ExpectedException.none();
@Before
public void setUpClassLoaderService() {
when(classLoaderService.getDependencyClassLoader()).thenReturn(getClass().getClassLoader());
}
@Test
public void shouldExecuteEngineFactories_FromMavenTestClassPathElements() throws Exception {
mockTestClasspathElements(CustomEngineFactory.class);
......
......@@ -27,6 +27,8 @@ import de.schrieveslaach.nlpf.testing.annotators.MyPosTagger;
import de.tudarmstadt.ukp.dkpro.core.api.parameter.ComponentParameters;
import de.tudarmstadt.ukp.dkpro.core.opennlp.OpenNlpNamedEntityRecognizer;
import org.apache.commons.io.FileUtils;
import org.apache.uima.analysis_engine.AnalysisEngineDescription;
import org.apache.uima.resource.ResourceInitializationException;
import org.json.JSONObject;
import org.junit.Before;
import org.junit.Test;
......@@ -37,6 +39,7 @@ import org.mockito.junit.MockitoJUnitRunner;
import java.io.File;
import java.nio.charset.Charset;
import static de.schrieveslaach.nlpf.plumbing.util.AnalysisEngineDescriptionUtil.hash;
import static java.util.Arrays.asList;
import static net.javacrumbs.jsonunit.JsonMatchers.jsonEquals;
import static org.apache.uima.fit.factory.AnalysisEngineFactory.createEngineDescription;
......@@ -48,6 +51,22 @@ import static org.mockito.Mockito.when;
@RunWith(MockitoJUnitRunner.class)
public class IsolatedToolsTestMojoTest extends AbstractTestMojoTest {
private static final AnalysisEngineDescription OPEN_NLP_PERSON_DESCRIPTION;
private static final AnalysisEngineDescription MY_PERSON_DESCRIPTION;
private static final AnalysisEngineDescription OPEN_NLP_ORGANIZATION_DESCRIPTION;
private static final AnalysisEngineDescription MY_ORGANIZATION_DESCRIPTION;
static {
try {
OPEN_NLP_PERSON_DESCRIPTION = createEngineDescription(OpenNlpNamedEntityRecognizer.class, ComponentParameters.PARAM_VARIANT, "person");
MY_PERSON_DESCRIPTION = createEngineDescription(MyNamedEntityRecognizer.class, ComponentParameters.PARAM_VARIANT, "person");
OPEN_NLP_ORGANIZATION_DESCRIPTION = createEngineDescription(OpenNlpNamedEntityRecognizer.class, ComponentParameters.PARAM_VARIANT, "organization");
MY_ORGANIZATION_DESCRIPTION = createEngineDescription(MyNamedEntityRecognizer.class, ComponentParameters.PARAM_VARIANT, "organization");
} catch (ResourceInitializationException e) {
throw new AssertionError(e);
}
}
@InjectMocks
private IsolatedToolsTestMojo mojo;
......@@ -73,7 +92,7 @@ public class IsolatedToolsTestMojoTest extends AbstractTestMojoTest {
public static String jsonOpenNlpPersonTestResult() {
return new JSONObject()
.put("descriptors", asList(
"933D4E2CC3B73F2359007BA5990F68AFFEFCA031E364D5354A3F7DE9B8308C6CCD89B5E89FF169D72798D92C80C5DBB458BF6FE914779B5E4EE94D6A7F18EEB8.xml"
hash(OPEN_NLP_PERSON_DESCRIPTION) + ".xml"
))
.put("measures", asList(
new JSONObject()
......@@ -119,19 +138,19 @@ public class IsolatedToolsTestMojoTest extends AbstractTestMojoTest {
@Test
public void shouldExecutePOSTaggerIsolated_OpenNlpNER() throws Exception {
mockEngineDescriptionsInDescriptorsDirectory(
createEngineDescription(OpenNlpNamedEntityRecognizer.class, ComponentParameters.PARAM_VARIANT, "person"),
createEngineDescription(MyNamedEntityRecognizer.class, ComponentParameters.PARAM_VARIANT, "person"),
createEngineDescription(OpenNlpNamedEntityRecognizer.class, ComponentParameters.PARAM_VARIANT, "organization"),
createEngineDescription(MyNamedEntityRecognizer.class, ComponentParameters.PARAM_VARIANT, "organization")
OPEN_NLP_PERSON_DESCRIPTION,
MY_PERSON_DESCRIPTION,
OPEN_NLP_ORGANIZATION_DESCRIPTION,
MY_ORGANIZATION_DESCRIPTION
);
provideTestJCas();
mojo.execute();
assertThatIsolatedTestDataFileExists("2CE76C3CC0D9A18DB4A36D5C53C84C45E84B1610ACD5005026DD70F7B02E17789160772FEC566E612D655D93A8C4561DC027E649C40E82F479532B0729309399");
assertThatIsolatedTestDataFileExists("21C50931D58D96FA33EAB7208AE1278ACE1F22D0DEFC7587F5DE1851013B2270C9BFCD9A7414769EBEE1BCF9733F02AA2E8FE4F1C9BDD44B2ED474BDF1CCB4D0");
assertThatIsolatedTestDataFileExists("DCA1B1EA9F8F959CD4F15CFF0724A11583977970F20E086B59DC614B76BECF75D745576C2DEC217DD191AF994325AED4CEC0A3CF960C1329E929F66338C5E5F");
File openNerPersonTestResult = assertThatIsolatedTestDataFileExists("933D4E2CC3B73F2359007BA5990F68AFFEFCA031E364D5354A3F7DE9B8308C6CCD89B5E89FF169D72798D92C80C5DBB458BF6FE914779B5E4EE94D6A7F18EEB8");
assertThatIsolatedTestDataFileExists(hash(OPEN_NLP_ORGANIZATION_DESCRIPTION));
assertThatIsolatedTestDataFileExists(hash(MY_ORGANIZATION_DESCRIPTION));
assertThatIsolatedTestDataFileExists(hash(MY_PERSON_DESCRIPTION));
File openNerPersonTestResult = assertThatIsolatedTestDataFileExists(hash(OPEN_NLP_PERSON_DESCRIPTION));
String json = FileUtils.readFileToString(openNerPersonTestResult, Charset.defaultCharset());
assertThat(json, jsonEquals(jsonOpenNlpPersonTestResult()).withTolerance(0.001));
......
package de.schrieveslaach.nlpf.maven.plugin;
package de.schrieveslaach.nlpf.maven.plugin.models;
/*-
* ========================LICENSE_START=================================
......@@ -46,6 +46,7 @@ import java.util.Map;
import static java.util.Arrays.asList;
import static net.javacrumbs.jsonunit.JsonMatchers.jsonEquals;
import static org.apache.uima.fit.factory.AnalysisEngineFactory.createEngineDescription;
import static org.hamcrest.Matchers.closeTo;
import static org.hamcrest.Matchers.contains;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.hasEntry;
......@@ -296,6 +297,23 @@ public class TestResultTest {
assertThat(map, not(hasKey(createOpenNlpNerPersonTestResult())));
}
@Test
public void shouldCalculateHarmonicMean_WithoutMeasures() throws Exception {
TestResult testResult = new TestResult(asList(
createEngineDescription(MySegmenter.class),
createEngineDescription(OpenNlpNamedEntityRecognizer.class, OpenNlpNamedEntityRecognizer.PARAM_VARIANT, "person")
));
assertThat(testResult.calculateHarmonicMean(), is(equalTo(0.0)));
}
@Test
public void shouldCalculateHarmonicMean_WithMeasures() throws Exception {
TestResult testResult = createOpenNlpNerPersonTestResult();
assertThat(testResult.calculateHarmonicMean(), is(closeTo(0.9, 0.0001)));
}
@SneakyThrows(UIMAException.class)
private TestResult createOpenNlpNerTestResult() {
return createTestResult(
......
......@@ -23,7 +23,9 @@ package de.schrieveslaach.nlpf.maven.plugin.service;
*/
import org.apache.maven.artifact.Artifact;
import org.apache.maven.model.Build;
import org.apache.maven.project.MavenProject;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.InjectMocks;
......@@ -39,6 +41,8 @@ import java.util.Set;
import static java.util.Arrays.asList;
import static org.hamcrest.Matchers.contains;
import static org.hamcrest.Matchers.empty;
import static org.hamcrest.Matchers.hasItem;
import static org.hamcrest.Matchers.hasSize;
import static org.hamcrest.Matchers.instanceOf;
import static org.hamcrest.Matchers.is;
import static org.junit.Assert.assertThat;
......@@ -54,6 +58,13 @@ public class ClassLoaderServiceTest {
@Mock
private MavenProject mavenProject;
@Before
public void setUpMavenProject() {
Build build = mock(Build.class);
when(build.getOutputDirectory()).thenReturn(new File(".").getAbsolutePath());
when(mavenProject.getBuild()).thenReturn(build);
}
@Test
public void shouldReturnClassLoader_WithCompileArtifacts() throws Exception {
mockArtifact("compile");
......@@ -62,7 +73,7 @@ public class ClassLoaderServiceTest {
assertThat(dependencyClassLoader, instanceOf(URLClassLoader.class));
URLClassLoader classLoader = (URLClassLoader) dependencyClassLoader;
assertThat(asList(classLoader.getURLs()), contains(new URL("file:/tmp/test.jar")));
assertThat(asList(classLoader.getURLs()), hasItem(new URL("file:/tmp/test.jar")));
}
@Test
......@@ -73,7 +84,7 @@ public class ClassLoaderServiceTest {
assertThat(dependencyClassLoader, instanceOf(URLClassLoader.class));
URLClassLoader classLoader = (URLClassLoader) dependencyClassLoader;
assertThat(asList(classLoader.getURLs()), is(empty()));
assertThat("has only build directory as resource", asList(classLoader.getURLs()), hasSize(1));
}
@Test
......@@ -84,7 +95,7 @@ public class ClassLoaderServiceTest {
assertThat(dependencyClassLoader, instanceOf(URLClassLoader.class));
URLClassLoader classLoader = (URLClassLoader) dependencyClassLoader;
assertThat(asList(classLoader.getURLs()), contains(new URL("file:/tmp/test.jar")));
assertThat(asList(classLoader.getURLs()), hasItem(new URL("file:/tmp/test.jar")));
}
private void mockArtifact(String scope) {
......
......@@ -7,7 +7,7 @@
<parent>
<groupId>de.schrieveslaach.nlpf</groupId>
<artifactId>nlpf-parent</artifactId>
<version>1.0-beta08-SNAPSHOT</version>
<version>1.1.0-SNAPSHOT</version>
</parent>
<artifactId>nlp-sonarqube-plugin</artifactId>
......@@ -132,6 +132,11 @@
<artifactId>json</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>net.javacrumbs.json-unit</groupId>
<artifactId>json-unit</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
......
......@@ -10,34 +10,48 @@ package de.schrieveslaach.nlpf.sonarqube.plugin.measures;
* it under the terms of the GNU Lesser General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Lesser Public License for more details.
*
*
* You should have received a copy of the GNU General Lesser Public
* License along with this program. If not, see
* <http://www.gnu.org/licenses/lgpl-3.0.html>.
* =========================LICENSE_END==================================
*/
import de.tudarmstadt.ukp.dkpro.core.api.ner.type.NamedEntity;
import de.tudarmstadt.ukp.dkpro.core.api.parameter.ComponentParameters;
import de.tudarmstadt.ukp.dkpro.core.api.segmentation.type.Sentence;
import de.tudarmstadt.ukp.dkpro.core.api.segmentation.type.Token;
import de.tudarmstadt.ukp.dkpro.core.opennlp.OpenNlpNamedEntityRecognizer;
import de.tudarmstadt.ukp.dkpro.core.opennlp.OpenNlpSegmenter;
import lombok.SneakyThrows;
import org.apache.commons.io.FileUtils;
import org.apache.uima.jcas.tcas.Annotation;
import org.apache.uima.resource.ResourceInitializationException;
import org.json.JSONArray;
import org.json.JSONObject;
import org.junit.Test;
import org.sonar.api.batch.fs.FileSystem;
import org.sonar.api.batch.fs.InputModule;
import org.sonar.api.batch.sensor.SensorContext;
import org.sonar.api.batch.sensor.measure.NewMeasure;
import sun.security.pkcs11.Secmod;
import java.io.File;
import java.nio.charset.Charset;
import static de.schrieveslaach.nlpf.sonarqube.plugin.measures.NlpToolMetrics.ISOLATED_TOOLS_TEST;
import static de.schrieveslaach.nlpf.testing.EngineDescriptionHashing.hash;
import static de.schrieveslaach.nlpf.testing.JsonDataFactory.createNlpToolTestResultOfOpenNlpSegmenterAndOpenNlpNamedEntityRecognizerWithPersonAndModelVariant;
import static net.javacrumbs.jsonunit.JsonMatchers.jsonEquals;
import static org.apache.uima.fit.factory.AnalysisEngineFactory.createEngineDescription;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
import static org.mockito.hamcrest.MockitoHamcrest.argThat;
public class IsolatedToolsTestSensorTest extends SensorTest {
......@@ -61,9 +75,32 @@ public class IsolatedToolsTestSensorTest extends SensorTest {
verify(context).newMeasure();
verify(newMeasure).forMetric(ISOLATED_TOOLS_TEST);
verify(newMeasure).on(inputModule);
verify(newMeasure).withValue(
"[{\"descriptors\":[\"E9F898F2DAB703DD603DBE66410600DEF500740E9436D1BD445AC925327A803F40FD14C9D094E91FC2D884EC0C4DCD832B7080023D0ED4A605314B1B6CA12ABC.xml\",\"933D4E2CC3B73F2359007BA5990F68AFFEFCA031E364D5354A3F7DE9B8308C6CCD89B5E89FF169D72798D92C80C5DBB458BF6FE914779B5E4EE94D6A7F18EEB8.xml\",\"DCA1B1EA9F8F959CD4F15CFF0724A11583977970F20E086B59DC614B76BECF75D745576C2DEC217DD191AF994325AED4CEC0A3CF960C1329E929F66338C5E5F.xml\"],\"measures\":[{\"outputType\":\"de.tudarmstadt.ukp.dkpro.core.api.ner.type.NamedEntity\",\"variant\":\"organization\",\"f-measure\":0.5714,\"analysisEngineName\":\"OpenNlpNamedEntityRecognizer\"},{\"outputType\":\"de.tudarmstadt.ukp.dkpro.core.api.ner.type.NamedEntity\",\"variant\":\"person\",\"f-measure\":0.5,\"analysisEngineName\":\"OpenNlpNamedEntityRecognizer\"},{\"outputType\":\"de.tudarmstadt.ukp.dkpro.core.api.segmentation.type.Sentence\",\"variant\":null,\"f-measure\":1.0,\"analysisEngineName\":\"OpenNlpSegmenter\"},{\"outputType\":\"de.tudarmstadt.ukp.dkpro.core.api.segmentation.type.Token\",\"variant\":null,\"f-measure\":1.0,\"analysisEngineName\":\"OpenNlpSegmenter\"}]}]"
);
verify(newMeasure).withValue(argThat(jsonEquals(createIsolatedToolTestValue())));
}
@SneakyThrows(ResourceInitializationException.class)
private static JSONArray createIsolatedToolTestValue() {
return new JSONArray()
.put(new JSONObject()
.put("descriptors", new JSONArray()
.put(hash(createEngineDescription(OpenNlpSegmenter.class)) + ".xml")
.put(hash(createEngineDescription(OpenNlpNamedEntityRecognizer.class, ComponentParameters.PARAM_VARIANT, "person")) + ".xml")
.put(hash(createEngineDescription(OpenNlpNamedEntityRecognizer.class, ComponentParameters.PARAM_VARIANT, "organization")) + ".xml")
)
.put("measures", new JSONArray()
.put(createMeasure(NamedEntity.class, "organization", 0.5714, "OpenNlpNamedEntityRecognizer"))
.put(createMeasure(NamedEntity.class, "person", 0.5, "OpenNlpNamedEntityRecognizer"))
.put(createMeasure(Sentence.class, null, 1.0, "OpenNlpSegmenter"))
.put(createMeasure(Token.class, null, 1.0, "OpenNlpSegmenter"))
)
);
}
private static JSONObject createMeasure(Class<? extends Annotation> annotationClass, String variant, Double fMeasure, String analysisEngineName) {
return new JSONObject()
.put("outputType", annotationClass.getCanonicalName())
.put("variant", variant != null ? variant : JSONObject.NULL)
.put("f-measure", fMeasure)
.put("analysisEngineName", analysisEngineName);
}
}
......@@ -10,12 +10,12 @@ package de.schrieveslaach.nlpf.sonarqube.plugin.measures;
* it under the terms of the GNU Lesser General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Lesser Public License for more details.
*
*
* You should have received a copy of the GNU General Lesser Public
* License along with this program. If not, see
* <http://www.gnu.org/licenses/lgpl-3.0.html>.
......@@ -25,6 +25,8 @@ package de.schrieveslaach.nlpf.sonarqube.plugin.measures;
import de.tudarmstadt.ukp.dkpro.core.api.segmentation.type.Token;
import org.apache.commons.io.FileUtils;
import org.apache.uima.jcas.tcas.Annotation;
import org.json.JSONArray;
import org.json.JSONObject;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
......@@ -40,10 +42,12 @@ import java.util.List;
import static de.schrieveslaach.nlpf.testing.JsonDataFactory.createNlpToolTestResultOfMySegmenterAndMyPosTagger;
import static de.schrieveslaach.nlpf.testing.JsonDataFactory.createNlpToolTestResultOfOpenNlpSegmenterAndOpenNlpNamedEntityRecognizerWithPersonAndModelVariant;
import static net.javacrumbs.jsonunit.JsonMatchers.jsonEquals;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
import static org.mockito.hamcrest.MockitoHamcrest.argThat;