Commit 4b1808eb authored by Colin Watson's avatar Colin Watson
Browse files

Flesh out README

* README: Move to ...
* ... here.  Reformat as Markdown.
(Using the library): New section, mainly borrowed from the project
parent 06f54396
Pipeline #221364482 passed with stages
in 3 minutes and 17 seconds
libpipeline, a pipeline manipulation library
# libpipeline, a pipeline manipulation library
Git repository:
libpipeline is a C library for setting up and running pipelines of
processes, without needing to involve shell command-line parsing which is
......@@ -10,51 +8,170 @@ often error-prone and insecure. This alleviates programmers of the need to
laboriously construct pipelines using lower-level primitives such as fork(2)
and execve(2).
Full programmers' documentation may be found using 'man libpipeline'.
Full programmers' documentation may be found using `man libpipeline`, and
the [project homepage]( has more background.
## Installation
If you need to install libpipeline starting from source code, then you will
need these separate packages installed before configuring libpipeline in
order to run its test suite:
pkg-config (
check >= 0.9.10 (
* [pkg-config](
* [check >= 0.9.10](
See the INSTALL file for general installation instructions.
Building programs with libpipeline
## Using the library
When the author took over [man-db]( in 2001, one
of the major problems that became evident after maintaining it for a while
was the way it handled subprocesses. The nature of man and friends means
that it spends a lot of time calling sequences of programs such as `zsoelim
< input-file | tbl | nroff -mandoc -Tutf8`. Back then, it was using C
library facilities such as `system` and `popen` for all this, and there were
several bugs where those functions were being called with untrusted input as
arguments without properly escaping metacharacters. Of course it was
possible to chase around every such call inserting appropriate escaping
functions, but this was always bound to be error-prone and one of the tasks
that rapidly became important was arranging to start subprocesses in a way
that was fundamentally immune to this kind of bug.
In higher-level languages, there are usually standard constructs which are
safer than just passing a command line to the shell. For example, in Perl
you can use something like this to invoke a program with arguments without
the interference of the shell:
system([$command, $arg1, $arg2, ...])
[perlipc(1)]( describes various facilities
for connecting them together.
In Python, the
[subprocess]( module
allows you to create pipelines easily and safely (as long as you remember
By contrast, C has the `fork` and `execve` primitives, but assembling these
to construct full-blown pipelines correctly is difficult and error-prone, so
many programmers don't bother and use the simple but unsafe library
facilities instead.
libpipeline solves this problem. In the following examples, function names
starting with `pipecmd_` or `pipeline_` are real functions in the library,
while any other function names are pseudocode.
Constructing the simplified example pipeline from the first paragraph above
using this library looks like this:
pipeline *p;
int status;
p = pipeline_new ();
pipeline_want_infile (p, "input-file");
pipeline_command_args (p, "zsoelim", NULL);
pipeline_command_args (p, "tbl", NULL);
pipeline_command_args (p, "nroff", "-mandoc", "-Tutf8", NULL);
status = pipeline_run (p);
You might want to construct a command more dynamically:
pipecmd *manconv = pipecmd_new_args ("manconv", "-f", from_code,
"-t", "UTF-8", NULL);
if (quiet)
pipecmd_arg (manconv, "-q");
pipeline_command (p, manconv);
Perhaps you want an environment variable set only while running a certain
pipecmd *less = pipecmd_new ("less");
pipecmd_setenv (less, "LESSCHARSET", lesscharset);
You might find yourself needing to pass the output of one pipeline to
several other pipelines, in a "tee" arrangement:
pipeline *source, *sink1, *sink2;
source = make_source ();
sink1 = make_sink1 ();
sink2 = make_sink2 ();
pipeline_connect (source, sink1, sink2, NULL);
/* Pump data among these pipelines until there's nothing left. */
pipeline_pump (source, sink1, sink2, NULL);
pipeline_free (sink2);
pipeline_free (sink1);
pipeline_free (source);
Maybe one of your commands is actually an in-process function, rather than
an external program:
pipecmd *inproc = pipecmd_new_function ("in-process", &func, NULL, NULL);
pipeline_command (p, inproc);
Sometimes your program needs to consume the output of a pipeline, rather
than sending it all to some other subprocess:
pipeline *p = make_pipeline ();
const char *line;
pipeline_want_out (p, -1);
pipeline_start (p);
line = pipeline_peekline (p);
if (!strstr (line, "coding: UTF-8"))
printf ("Unicode text follows:\n");
while (line = pipeline_readline (p))
printf (" %s", line);
pipeline_free (p);
## Building programs with libpipeline
libpipeline supplies a pkg-config file which lists appropriate compiler and
linker flags for building programs using it. The output of 'pkg-config
--cflags libpipeline' should be passed to the compiler (typically CFLAGS)
and the output of 'pkg-config --libs libpipeline' should be passed to the
linker (typically LDFLAGS).
linker flags for building programs using it. The output of `pkg-config
--cflags libpipeline` should be passed to the compiler (typically `CFLAGS`)
and the output of `pkg-config --libs libpipeline` should be passed to the
linker (typically `LDFLAGS`).
If your program uses the GNU Autotools, then you can put this in
PKG_CHECK_MODULES([libpipeline], [libpipeline])
PKG_CHECK_MODULES([libpipeline], [libpipeline])
... and this in the appropriate (replacing 'program' with the
Automake-canonicalised name for your program):
AM_CFLAGS = $(libpipeline_CFLAGS)
program_LDADD = $(libpipeline_LIBS)
AM_CFLAGS = $(libpipeline_CFLAGS)
program_LDADD = $(libpipeline_LIBS)
The details may vary for particular build systems, but this should be a
reasonable start.
When building with GCC, you should use at least the -Wformat option
(included in -Wall) to ensure that the 'sentinel' function attribute is
When building with GCC, you should use at least the `-Wformat` option
(included in `-Wall`) to ensure that the 'sentinel' function attribute is
checked. This means that your program will produce a warning if it calls
any of the several libpipeline functions that require a trailing NULL
without passing that trailing NULL.
Copyright and licensing
## Copyright and licensing
Copyright (C) 1994 Markus Armbruster.
Copyright (C) 1989, 1990, 1991, 1992, 2000, 2001, 2002, 2003, 2004, 2005,
......@@ -77,8 +194,7 @@ along with libpipeline; if not, write to the Free Software
Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301
Note on GPL versions
## Note on GPL versions
(This note is informative, and if it conflicts with the terms of the licence
then the licence is correct. See the full text of the licence in the
......@@ -106,21 +222,14 @@ write programs for your own use that use libpipeline without needing to
license them under GPL v3-compatible terms. If you distribute these
programs to others, then you must take care to use compatible licensing.
## Credits
Thanks to Scott James Remnant for code review, Ian Jackson for an extensive
design review, and Kees Cook and Matthias Klose for helpful conversations.
Bug reporting
## Bug reporting
You can report bugs here:
Bugs from before the migration to GitLab may be found here:
-- Colin Watson <>
You can [report bugs on
GitLab](, or see [bugs from
before the migration to
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment