Commit 34979d4f authored by William Arndt's avatar William Arndt

update workflow tools page with current state of workflows working group progress

parent 138e1294
# Workflow tools
Supporting data-centric science often involves the movement of data
across file systems, multi-stage analytics and visualization. Workflow
technologies can improve the productivity and efficiency of
data-centric science by orchestrating and automating these
steps. NERSC provides support for the TaskFarmer, Swift and Fireworks
tools. We also maintain other packages like Tigres that can help users
build workflows.
# Workflow Mangement Tools
Supporting data-centric science involves the movement of data, multi-stage
processing, and visualization at scales where manual control becomes
prohibitive and automation is needed. Workflow technologies can improve the
productivity and efficiency of data-centric science by orchestrating and
automating these steps.
A NERSC working group review and refresh of this content is currently in
progress; an initial update is expected by April 2020. In the meantime we
request the following of users considering workflow management solutions:
* Please do not write your own workflow manager. More than 200 such solutions
already exist and almost certainly at least 3 of them can be found which will
fit your needs.
* Before you begin developing infrastructure targeting a particular workflow
manager, please contact NERSC consultants to confirm it can be effectively
used at NERSC. Some tools have infrastructure needs or operate in a manner
which is fundamentaly incompatible with NERSC systems and we'd like to
prevent wasted effort if we can.
  • Maybe scratch

    and we'd like to prevent wasted effort if we can.

    cause it sounds kinda unfriendly and annoyed.

    Edited by Bjoern Enders
Please register or sign in to reply
## TaskFarmer
[TaskFarmer](workflow/taskfarmer.md) is a utility developed at NERSC to distribute single-node tasks across
a set of compute nodes - these can be single- or multi-core tasks. TaskFarmer tracks which
tasks have completed successfully, and allows straightforward re-submission of failed or un-run jobs from a checkpoint file.
[TaskFarmer](workflow/taskfarmer.md) is a utility developed at NERSC to
distribute single-node tasks across a set of compute nodes - these can be
single- or multi-core tasks. TaskFarmer tracks which tasks have completed
successfully, and allows straightforward re-submission of failed or un-run jobs
from a checkpoint file.
## Swift
The [Swift](workflow/swift.md)
scripting language provides a simple, compact way to write
parallel scripts that run many copies of ordinary programs
concurrently in various workflow patterns, reducing the need for
complex parallel programming or arcane scripting. Swift is very
general, and is in use in domains ranging from earth systems to
bioinformatics to molecular modeling.
The [Swift](workflow/swift.md) scripting language provides a simple, compact
way to write parallel scripts that run many copies of ordinary programs
concurrently in various workflow patterns, reducing the need for complex
parallel programming or arcane scripting. Swift is very general, and is in use
in domains ranging from earth systems to bioinformatics to molecular modeling.
## Fireworks
[FireWorks](workflow/fireworks.md) is a free, open-source code for defining, managing, and
executing scientific workflows. It can be used to automate
calculations over arbitrary computing resources, including those that
have a queueing system. Some features that distinguish FireWorks are
dynamic workflows, failure-detection routines, and built-in tools and
execution modes for running high-throughput computations at large
computing centers. It uses a centralized server model, where the
server manages the workflows and workers run the jobs.
[FireWorks](workflow/fireworks.md) is a free, open-source code for defining,
managing, and executing scientific workflows. It can be used to automate
calculations over arbitrary computing resources, including those that have a
queueing system. Some features that distinguish FireWorks are dynamic
workflows, failure-detection routines, and built-in tools and execution modes
for running high-throughput computations at large computing centers. It uses a
centralized server model, where the server manages the workflows and workers
run the jobs.
## Other Workflow Tools
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment