Commit 1d44eed3 authored by 35V LG84's avatar 35V LG84

Initial import

Signed-off-by: 35V LG84's avatar35V LG84 <35vlg84-x4e6b92@e257.fi>
parents
/**/lib_managed/
/**/src_managed/
/**/target/
/tests/**/out.*
/.idea/
*.swp
*~
*.sc
= DirSuite releases
Current published release is:
libraryDependencies += "fi.e257.testing" %% "dirsuite" % "0.20.0"
See link:./UPGRADE.adoc[Upgrade Guide] for needed changes between versions.
== Release history
=== 0.20.0
This is same as DirSuite v0.7.0. The difference is that code is
hosted at gitlab.com and moved under new namespace and groupId.
==== Fixes
None
==== Development
Updated deps and tools:
* build
** sbt: 1.2.8
** sbt-sonatype: 2.3
= Contributors Guide
Contributions to the project are welcome!
Before starting working on something, please first check by opening Gitlab issue
if your plan is feasible and viable from project's perspective.
== How to build and test
* You do have sbt, don't you? If not, get it from here:
link:http://www.scala-sbt.org/[http://www.scala-sbt.org/]
=== Get source code
----
git clone https://gitlab.com/e257/testing/dirsuite.git
----
=== Build and test
----
sbt clean test
----
=== Generate test coverage reports
----
sbt clean coverage test coverageReport
sbt coverageAggregate
----
== Explore code
Directory structure follows maven's layout:
* DirSuite main code: link:./src/main/scala/fi/e257/testing[src -> main -> fi -> e257 -> testing]
* DirSuite tests
** link:./tests/dirsuite[tests -> dirsuite] is collection of tests, which are used to test DirSuite
** link:./src/test/scala/fi/e257/testing/DirSuiteLikeTest.scala[src -> tests -> DirSuiteLikeTest]
contains DirSuite's own tests. E.g. it's used to test DirSuite's own code.
These tests are a bit convoluted, because they are testing test-framework itself.
* Example dirsuite setups: link:./examples[examples]
== Developer Certificate of Origin
Your pull requests and patches can be merged only if you can certify
the link:./DCO[Developer Certificate of Origin (DCO), Version 1.1].
By signing-off your commit you certify DCO and acknowlegde that you have
the rights to submit your pull requsted or patch as an open-source contribution,
as stated in link:./DCO[Developer Certificate of Origin].
Certifying DCO is done by adding a `Signed-off-by` line
to **every** git commit message:
Signed-off-by: gitlab-account <your.real@email.address>
If you set your `user.name` and `user.email` correctly in git config,
then git will automatically include that line for you with `git commit -s`.
You can also create a git `commit.template` with appropriate content. These
settings can be done per repository basis, so they don't have to be globally
set in your system.
Please make sure that you sign-off all your PR's commits.
Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.
This diff is collapsed.
DirSuite
Copyright 2016-2019 E257.FI Contributors
This product includes software developed at
e257.fi (https://e257.fi/devel/).
-------------------------------------------------------------
Scalatest
-------------------------------------------------------------
This product includes software developed by
Artima, Inc. (http://www.artima.com/).
image:https://gitlab.com/e257/testing/dirsuite/badges/master/build.svg["Build status", link="https://gitlab.com/e257/testing/dirsuite/-/jobs/"]
image:https://gitlab.com/e257/testing/dirsuite/badges/master/coverage.svg["Coverage", link="https://gitlab.com/e257/testing/dirsuite/-/jobs/"]
image:https://maven-badges.herokuapp.com/maven-central/fi.e257.testing/dirsuite_2.12/badge.svg["Maven Central", link="https://maven-badges.herokuapp.com/maven-central/fi.e257.testing/dirsuite_2.12"]
image:https://javadoc.io/badge/fi.e257.testing/dirsuite_2.12.svg?color=blue["javadoc", link="https://javadoc.io/page/fi.e257.testing/dirsuite_2.12/latest/fi/e257/testing/dirsuite/index.html"]
= DirSuite test framework and add-on for ScalaTest
This is DirSuite add-on to ScalaTest. DirSuite can run (huge) collection of tests which are defined on filesystem.
Arguments, inputs and expected reference output files are defined in the test directory.
== Documentation
* link:./docs/dirsuite.adoc[DirSuite Reference] contains reference information for dirsuite
* link:./docs/howto.adoc[DirSuite HOWTO] has examples how to use dirsuite
* Example DirSuite test setup:
** link:./examples/[examples] Demo setup with example test corpus.
** link:./examples/src/test/scala/DirSuiteDemo.scala[examples -> src -> tests -> DirSuiteDemo] DirSuite tests against example corpus.
** link:./examples/tests/[examples -> tests] Test corpus for example project
* link:https://gitlab.com/e257/accounting/tackler[Tackler Project] has extensive set of tests based on DirSuite
** link:https://gitlab.com/e257/accounting/tackler/tree/master/tests[Tackler's DirSuite tests]
** link:https://gitlab.com/e257/accounting/tackler/blob/master/project/TacklerTests.scala[Tackler's clean up setup for DirSuite tests]
== Releases
Dependency settings for SBT are:
libraryDependencies += "fi.e257.testing" %% "dirsuite" % "version-number"
DirSuite artifacts are released on Maven Central Repository.
For release information and version history details, see link:./CHANGELOG.adoc[CHANGELOG]. For Upgrade instructions,
see link:./UPGRADE.adoc[UPGRADE].
== Contributing to DirSuite
Contributions to the project are welcome. See
link:./CONTRIBUTING.adoc[CONTRIBUTING] how you could help.
Your pull requests and patches can be merged only if you can certify
the link:./DCO[Developer Certificate of Origin (DCO), Version 1.1].
By signing-off your commit you certify DCO and acknowlegde that you have
the rights to submit your pull requsted or patch as an open-source contribution,
as stated in link:./DCO[Developer Certificate of Origin].
Certifying DCO is done by adding a `Signed-off-by` line
to **every** git commit message:
Signed-off-by: gitlab-account <your.real@email.address>
If you set your `user.name` and `user.email` correctly in git config,
then git will automatically include that line for you with `git commit -s`.
You can also create a git `commit.template` with appropriate content. These
settings can be done per repository basis, so they don't have to be globally
set in your system.
Please make sure that you sign-off all your PR's commits.
== Credits
Without link:http://www.scalatest.org/[ScalaTest] this project
would not exists. See link:./THANKS.adoc[THANKS] for full list of credits.
== License
....
Copyright 2016-2019 E257.FI Contributors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
....
= Credits
DirSuite has been originally written by e257.fi from scratch.
Following projects have provided valuable contributions,
feedback, inspiration and ideas for this project. Thank you for your help!
Components (used with DirSuite):
* link:http://www.scalatest.org/[ScalaTest]
* link:https://github.com/pathikrit/better-files[Better-files]
* link:https://github.com/jsuereth/scala-arm[Scala-ARM (automatic resource mgmt)]
Build-tools (used to build DirSuite):
* Build, Release and Publish
** link:http://www.scala-sbt.org/[sbt]
** link:https://github.com/xerial/sbt-sonatype[sbt-sonatype]
* QA
** link:https://github.com/scoverage/sbt-coveralls[sbt-coveralls]
** link:https://github.com/scoverage/sbt-scoverage[sbt-scoverage]
** link:https://github.com/wartremover/wartremover[WartRemover]
Tests:
* link:http://www.scalatest.org/[ScalaTest]
Ideas and inspiration:
* link:https://github.com/hrj/abandon[Abandon]
= DirSuite upgrade guide
== DirSuite 0.20.0
New toplevel namespace is `fi.e257.testing`.
No other changes should be needed.
import Dependencies._
/**
* Build settings for DirSuite
*/
lazy val commonSettings = Seq(
organization := "fi.e257.testing",
scalaVersion := "2.12.3",
scalacOptions ++= Seq(
"-Xlint",
"-deprecation",
"-feature",
"-unchecked",
"-Xfatal-warnings"),
wartremoverErrors ++= Warts.allBut(
Wart.Throw //https://github.com/puffnfresh/wartremover/commit/869763999fcc1fd685c1a8038c974854457b608f
),
publishTo := sonatypePublishTo.value,
)
lazy val dirsuite = (project in file(".")).
settings(commonSettings: _*).
settings(
name := "dirsuite",
version := "0.20.0",
fork in run := true,
libraryDependencies += betterFiles,
libraryDependencies += scalaArm,
libraryDependencies += scalatest
)
= DirSuite Reference
DirSuite is an add-on to ScalaTest which makes it possible to run and test
corpus of tests which are defined in as files on directory tree.
Test interface is simple. Test cases are define as special `exec-file`.
Exec-files describes how many times your test function is executed
per test, and what are the arguments for each execution step.
Your testFunction must take arguments as an array of strings.
Return values, possibly thrown exceptions etc. can be defined
and validated (asserted) by using normal
link:http://www.scalatest.org/user_guide/using_assertions[ScalaTest's assertions].
DirSuite also supports automatic validation of potential output.
Currently supported formats are txt and xml-output, but there are
extension points, where it is possible to add more
different kind of Validators.
It is possible to run multiple execution steps per one test case.
For example, test case for `jgit commit` operation could look
something like that:
`init`, `add file.txt`, `commit -m "msg"`
if test function is jgit's main method.
== Runnable Example Project
There are three demo DirSuite-setups under `examples`.
These are runnable examples, which demonstrate different aspects of DirSuite:
* link:../examples/src/main/scala/DemoApp.scala[DemoApp] which is used in tests as an example app.
It can count arguments, write arguments to files, and it will fail if told so.
* link:../examples/tests/[Test corpus] actual test case steps (exec-files), inputs and reference outputs are stored here.
* link:../examples/src/test/scala/DirSuiteDemo.scala[DirSuiteDemo]
This is DirSuite test, which tests "normal" operation of demo app.
* link:../examples/src/test/scala/MapArgsDemo.scala[MapArgsDemo]
This is DirSuite test, which manipulates test case arguments from DirSuite. This could be used for example
to provide fixed configuration arguments to the test app.
* link:../examples/src/test/scala/FailureDemo.scala[FailureDemo] These are tests which all will fail.
Currently all of these are ignored, but if you run them, they will provide example output of failure cases.
=== How to run examples?
Install SBT, if you don't have it, and then:
cd examples
sbt test
== Example of simple DirSuite test suite
Here is an example of DirSuite test case, which first runs tests which validates
successful operation, and after that check that application fails in expected way
(with invalid input).
Example of exec-file for success case, this would be located under
`tests/success/basic01.exec` in actual system:
# format: exec
exec:do;something;
exec:do;more;
Example of exec-file for failure case, this would be located under
`tests/failure/runtime01.exec` in actual system:
# format: exec
exec:fail;this;is;stupid;
Convention of `tests/success`, `tests/failure` is not mandated by DirSuite,
but it would be good idea to group different kind of tests to separate folders.
With above exec-file setup, we could write following DirSuite test.
class MyDirSuite extends DirSuiteLike {
val testdir = Paths.get("tests").toAbsolutePath.normalize
/*
* find all test cases which match glob: "tests/" + "success/basic*.exec"
* add assert that they return success status code.
*/
runDirSuiteTestCases(testdir, Glob("success/basic*.exec")) { args: Array[String] =>
assertResult(DemoApp.SUCCESS) {
DemoApp.doMain(args)
}
}
/*
* find all test cases wich match glob: "tests/" + "failure/runtimeEx*.exec"
* and assert that they throw runtime exception
*/
runDirSuiteTestCases(testdir, Glob("failure/runtimeEx*.exec")) { args: Array[String] =>
assertThrows[RuntimeException] {
DemoApp.run(args)
}
}
}
So this would run all `basic*.exec` tests and assert successful run for each basic.*exec steps and test cases
(if multiple `basic*.exec` files are found).
After that all `runtimeEx*.exec` test will run, and test framework will assert that each execution step will
throw `RuntimeException`.
=== Multistep failure tests
Dirsuite supports also multistep failure mode testing.
This is a test where first steps must behave one way (succeed),
and then the last step must behave differently (fail).
For example there is following Exec-file content:
# format: exec
exec:1;do;something;fun;
exec:2;do;more;fun;
exec:3;fail;this;is;stupid;
In this configuration, one test case will execute all these steps,
and first two steps (1-2) must succeed, but then the last,
step 3, must fail.
Multistep DirSuite configuration for above test case:
class MyDirSuite extends DirSuiteLike {
val testdir = Paths.get("tests").toAbsolutePath.normalize
/*
* find all test cases wich match glob: "tests/" + "failure/multistep*.exec"
* and assert that they throw runtime exception
*/
runDualAssertionDirSuiteTestCases(testdir, Glob("failure/multistep*.exec")) { args: Array[String] =>
assertResult(DemoApp.SUCCESS) {
DemoApp.doMain(args)
} { args: Array[String] =>
assertThrows[RuntimeException] {
DemoApp.run(args)
}
}
}
}
In that case test framework will find all `multistep*.exec` test cases, and then run those tests.
For each test it will assert that first steps (steps 1-2) will be successful and last execution
step (step 3) will throw `RuntimeException`.
== Exec file format
Exec file format is following:
# format: exec
# first line content must be above, and this line is comment
# next line is exec step without any arguments
exec:
# next step is exec step with one argument "a b c"
exec:a b c;
# next step is exec with 3 args "a", "b", "c"
exec:a;b;c;
# next step is exec with 3 empty args "", "", ""
exec:;;;
# next step is exec with arg: "\"qoute\""
exec:"qoute";
If exec separator `";"` is not suitable in your situation,
then it could be easily changed by overriding `getExecArgumentSeparator` method.
Separator is string, so it could consist of multiple characters, and default `tokenizer`
implementation will use it with `String::split`.
Example of changing default sepator to `"|"`:
class MyDirSuite extends DirSuiteLike {
// change exec separator to "|"
override protected
def getExecArgumentSeparator: String = "|"
runDirSuiteTestCases(testdir, Glob("barsep/sep[0-9]*.exec")) { args: Array[String] =>
assertResult(DemoApp.SUCCESS) {
app.doTxt(args)
}
}
}
If exec arguments must contain new line characters or something more complex,
then exec line tokenizer or the whole exec arg parser can be overridden:
- `tokenizer` tokenize line after "exec:" to the end of line
- `parseExec` parses whole exec file
See link:http://javadoc.io/doc/fi.e257/dirsuite_2.12[API Docs] or source code for exact arguments and return values for these two.
=== Regex or Glob to find exec-files
DirSuite Regex and Glob behaves as Java Regex and Glob, with twist.
* link:https://docs.oracle.com/javase/tutorial/essential/regex/index.html[Java Regex]
* link:https://docs.oracle.com/javase/tutorial/essential/io/fileOps.html#glob[Java Glob]
The Twist: DirSuite uses `better-files.glob and regex` to find files.
If pattern is NOT absolute path, and it does not start with Regex/Glob special character,
then basepath and pattern are combined together. That means that you don't need wildcard
matching at the beginning of your pattern. E.g.
// testdir == "/path/to/some/dir/tests"
val testdir = Paths.get("tests").toAbsolutePath.normalize
// this will match
// "/path/to/some/dir/tests/success/basic*.exec"
findFiles(testdir, "success/basic*.exec")
== Validating output
DirSuite can validate automatically output for test case, if reference
and output files follow certain naming scheme.
=== Reference files
Reference files are found automatically based on following logic:
Basename of exec-file is used as test case basename. This basename is
appended with `.ref.*` glob, and this pattern is used
in testcase's test directory.
Testcase: tests/subfolder/test01.exec
References: tests/subfolder/test01.ref.*
If this logic is not sufficient, you can override `findReferences` method.
=== Output files
Expected Output files are found automatically, if files
conform this naming scheme:
For each reference file, output name is constructed by:
Basename of exec-file is used as basename for output files.
This basename is prefixed with `out.` and after that
reference file's uniq part of filename is appended to output name.
Uniq part of reference name is suffix when `basename.ref.`
prefix removed from name.
Example:
Testcase: tests/subfolder/test01.exec
References: tests/subfolder/test01.ref.*
For reference file-1:
tests/subfolder/test01.ref.file-1.txt
Output would be:
tests/subfolder/out.test01.file-1.txt
If this logic is not sufficient, you can override `mapOutput` method.
=== Selecting Validator
Default validator is selected based on reference file extension.
`TestValidator.txtValidator` will be used for text files,
and `TestValidator.xmlValidator` for xml-files.
This validator selection logic and default validators can be replaced by overriding
`selectValidator` method.
=== Final words
Comments, Questions? Open a ticket at link:https://gitlab.com/e257/testing/dirsuite[].
Happy testing!
= How to use DirSuite
Under examples are three demo DirSuites, which demonstrates
different aspects of DirSuite. These are runnable projects,
just run `sbt test` on `examples`-directory.
* link:../examples/tests/[test cases] Test cases which are used by examples.
* link:../examples/src/test/scala/DirSuiteDemo.scala[DirSuiteDemo]
Example of typical usage of DirSuite
* link:../examples/src/test/scala/MapArgsDemo.scala[MapArgsDemo]
Example of how to map and change test's arguments by DirSuite test
(e.g. to provide extra arguments for conf file in actual test code).
* link:../examples/src/test/scala/FailureDemo.scala[FailureDemo]
Example how to ignore tests and examples of actual error messages when tests fail.
These ignored tests will all fail, if you enable them by changing
`ignore` prefix to `run` prefix.
/**/lib_managed/
/**/src_managed/
/**/target/
/tests/**/out.*
/.idea/
*.swp
*~
name := "DirSuiteExample"
scalaVersion := "2.12.3"
libraryDependencies += "fi.e257.testing" %% "dirsuite" % "0.20.0" % "test"
/*
* Copyright 2016-2019 E257.FI
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
import java.nio.charset.StandardCharsets
import java.nio.file.{Files, Path, Paths}
object DemoApp {
//
val SUCCESS = 127
val FAILURE = 255
}
class DemoApp(val testdir: Path) {
def doSuccess(args: Array[String]): Int = {
DemoApp.SUCCESS
}
def doFailure(args: Array[String]): Int = {
DemoApp.FAILURE
}
def doFlaky(args: Array[String]): Int = {
if (args(0) == "bang") {
throw new RuntimeException("BANG!")
}
else if (args(0) == "fail") {
DemoApp.FAILURE
} else {
DemoApp.SUCCESS
}
}
def doArgsCount(args: Array[String]): Int = {
args.length
}
def doTxt(args: Array[String]): Int = {
val output = Paths.get(testdir.toString, args(0))
Files.write(output, args
.mkString("hello\n", "\n", "\nworld\n")
.getBytes(StandardCharsets.UTF_8))
DemoApp.SUCCESS
}
def doXml(args: Array[String]): Int = {
val output = Paths.get(testdir.toString, args(0))
Files.write(output, args
.mkString("<hello><arg>", "</arg><arg>", "</arg></hello>\n")
.getBytes(StandardCharsets.UTF_8))
DemoApp.SUCCESS
}
def doTxtXml(args: Array[String]): Int = {
val result =
if (args(1) == "txt") {
doTxt(args)
} else if (args(1) == "xml") {
doXml(args)
} else {
DemoApp.FAILURE
}