doclifter.xml

<!DOCTYPE refentry PUBLIC 
   "-//OASIS//DTD DocBook XML V4.1.2//EN"
   "docbook/docbookx.dtd">
<refentry id='doclifter.1'>
<refmeta>
<refentrytitle>doclifter</refentrytitle>
<manvolnum>1</manvolnum>
<refmiscinfo class='date'>Aug 16 2001</refmiscinfo>
<refmiscinfo class='source'>doclifter</refmiscinfo>
<refmiscinfo class='productname'>doclifter</refmiscinfo>
<refmiscinfo class='manual'>Documentation Tools</refmiscinfo>
</refmeta>
<refnamediv id='name'>
<refname>doclifter</refname>
<refpurpose>translate troff requests into DocBook</refpurpose>
</refnamediv>
<refsynopsisdiv id='synopsis'>

<cmdsynopsis>
  <command>doclifter</command>  
  <arg choice='opt'>-o <replaceable>output-location</replaceable></arg>
  <arg choice='opt'>-e <replaceable>output-encoding</replaceable></arg>
  <arg choice='opt'>-i <replaceable>input-encodings</replaceable></arg>
  <arg choice='opt'>-h <replaceable>hintfile</replaceable></arg>
  <arg choice='opt'>-q</arg>
  <arg choice='opt'>-x</arg>
  <arg choice='opt'>-v</arg>
  <arg choice='opt'>-w</arg>
  <arg choice='opt'>-V</arg>
  <arg choice='opt'>-D <replaceable>token=type</replaceable></arg>
  <arg choice='opt'>-I <replaceable>path</replaceable></arg>
  <arg choice='opt'>-S <replaceable>spoofname</replaceable></arg>
  <arg choice='plain' rep='repeat'><replaceable>file</replaceable></arg>
</cmdsynopsis>

</refsynopsisdiv>

<refsect1><title>Description</title>
<para><command>doclifter</command>
translates documents written in troff macros to DocBook.  Structural
subsets of the requests in
<citerefentry><refentrytitle>man</refentrytitle><manvolnum>7</manvolnum></citerefentry>,
<citerefentry><refentrytitle>mdoc</refentrytitle><manvolnum>7</manvolnum></citerefentry>,
<citerefentry><refentrytitle>ms</refentrytitle><manvolnum>7</manvolnum></citerefentry>,
<citerefentry><refentrytitle>me</refentrytitle><manvolnum>7</manvolnum></citerefentry>,
<citerefentry><refentrytitle>mm</refentrytitle><manvolnum>7</manvolnum></citerefentry>,
and
<citerefentry><refentrytitle>troff</refentrytitle><manvolnum>1</manvolnum></citerefentry>
are supported.</para>

<para>The translation brings over all the structure of the original
document at section, subsection, and paragraph level.  Command and C
function synopses are translated into DocBook markup, not just a
verbatim display.  Tables (TBL markup) are translated into DocBook
table markup. PIC diagrams are translated into SVG.  Troff-level
information that might have structural implications is preserved in
XML comments.</para>

<para>Where possible, font-change macros are translated into
structural markup.  <command>doclifter</command> recognizes
stereotyped patterns of markup and content (such as the use of italics
in a FILES section to mark filenames) and lifts them.  A means to
edit, add, and save semantic hints about highlighting is supported.</para>

<para>Some cliches are recognized and lifted to structural markup 
even without highlighting.  Patterns recognized include
such things as URLs, email addresses, man page references, and
C program listings.</para>

<para>The tag <markup>.in</markup> and <markup>.ti</markup> requests are
passed through with complaints. They indicate presentation-level
markup that <command>doclifter</command> cannot translate into 
structure; the output will require hand-fixing.</para>

<para>The tag <markup>.ta</markup> is passed through with a complaint
unless the immediarely following by text lines contains a tab, in
which case the following span of lines containing tabs is lifted to a
table.</para>

<para>Under some circumstances, <command>doclifter</command> can even
lift formatted manual pages and the text output produced by
<citerefentry><refentrytitle>lynx</refentrytitle><manvolnum>1</manvolnum></citerefentry>
from HTML.  If it finds no macros in the input, but does find a NAME
section header, it tries to interpret the plain text as a manual page
(skipping boilerplate headers and footers generated by
<citerefentry><refentrytitle>lynx</refentrytitle><manvolnum>1</manvolnum></citerefentry>).
Translations produced in this way will be prone to miss structural
features, but this fallback is good enough for simple man
pages.</para>

<para><command>doclifter</command> does not do a perfect job, merely
a surprisingly good one.  Final polish should be applied by a human
being capable of recognizing patterns too subtle for a computer.  But
<command>doclifter</command> will almost always produce translations
that are good enough to be usable before hand-hacking.</para>

<para>See the <link linkend="troubleshooting">Troubleshooting</link>
section for discussion of how to solve document conversion
problems.</para>

</refsect1>

<refsect1><title>Options</title>
<para>If called without arguments <command>doclifter</command> acts as
a filter, translating troff source input on standard input to DocBook
markup on standard output.  If called with arguments, each argument
file is translated separately (but hints are retained, see below); the
suffix <filename>.xml</filename> is given to the translated output.</para>
<variablelist>
<varlistentry>
<term>-o</term>
<listitem>
<para>Set the output location where files will be saved. Defaults to current
working directory.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-h</term>
<listitem>
<para>Name a file to which information on semantic hints gathered
during analysis should be written.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-D</term>
<listitem>
<para>The <option>-D</option> allows you to post a hint. This may be
useful, for example, if <command>doclifter</command> is mis-parsing 
a synopsis because it doesn't recognize a token as a command.  This
hint is merged after hints in the input source have been read.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-I</term>
<listitem>
<para>The <option>-I</option> option adds its argument to the include
path used when docfilter searches for inclusions.  The include path is
initially just the current directory.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-S</term>
<listitem>
<para>Set the filename to be used in error and warning messages. This
is mainly inttended for use by test scripts.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-e</term>
<listitem>
<para>The <option>-e</option> allows you to set the output encoding of
the XML and the encoding field to be emitted in its header.  It
defaults to UTF-8.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-i</term>
<listitem>
<para>The <option>-i</option> allows you to set a comma-separated list of
encodings to be looked for in the input. The default is
"ISO-8859-1,UTF-8", which should cover almost all cases.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-q</term>
<listitem>
<para>Normally, requests that <command>doclifter</command> could not
interpret (usually because they're presentation-level) are passed
through to XML comments in the output.  The -q option suppresses
this.  It also suppresses listing of macros.  Messages about requests
that are unrecognized or cannot be translated go to standard error
whatever the state of this option.  This option is intended to reduce 
clutter when you believe you have a clean lift of a document and want
to lose the troff legacy.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-x</term>
<listitem>
<para>The -x option requests that <command>doclifter</command>
generate DocBook version 5 compatible xml content, rather than its
default DocBook version 4.4 output. Inclusions and entities
may not be handled correctly with this switch enabled.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-v</term>
<listitem>
<para>The -v option makes <command>doclifter</command>
noisier about what it's doing.  This is mainly useful for debugging.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-w</term>
<listitem>
<para>Enable strict portability checking.  Multiple instances of
-w increase the strictness.  See <xref linkend='portability'/>.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-V</term>
<listitem>
<para>With this option, the program emits a version message and exits.</para>
</listitem>
</varlistentry>
</variablelist>
</refsect1>

<refsect1><title>Translation Rules</title> 

<para>Overall, you can expect that font changes will be turned into
<sgmltag class='element'>Emphasis</sgmltag> macros with a <sgmltag
class='attribute'>Remap</sgmltag> attribute taken from the troff font
name.  The basic font names are R, I, B, U, CW, and SM.</para>

<para>Troff and macro-package special character escapes are mapped into ISO
character entities.</para>

<para>When <command>doclifter</command> encounters a
<markup>.so</markup> directive, it searches for the file.  If it can
get read access to the file, and open it, and the file consists
entirely of command lines and comments, then it is included.  If any
of these conditions fails, an entity reference for it is
generated.</para>

<para><command>doclifter</command> performs special parsing when it
recognizes a display such as is generated by
<markup>.DS/.DE</markup>. It repeatedly tries to parse first a
function synopsis, and then plain text off what remains in the
display.  Thus, most inline C function prototypes will be
lifted to structured markup.</para>

<para>Some notes on specific translations:</para>

<refsect2><title>Man Translation</title>

<para><command>doclifter</command> does a good job on most man pages,
It knows about the extended
<markup>UR</markup>/<markup>UE</markup>/<markup>UN</markup> and
<markup>URL</markup> requests supported under Linux.  If any
<markup>.UR</markup> request is present, it will translate these but
not wrap URLs outide them with <sgmltag>Ulink</sgmltag> tags. It also
knows about the extended <markup>.L</markup> (literal) font markup
from Bell Labs Version 8, and its friends.</para>

<para>The <markup>.TH</markup> macro is used to generate a <sgmltag
class='element'>RefMeta</sgmltag> section.  If present, the
date/source/manual arguments (see
<citerefentry><refentrytitle>man</refentrytitle><manvolnum>7</manvolnum></citerefentry>)
are wrapped in <sgmltag class='element'>RefMiscInfo</sgmltag> tag
pairs with those class attributes.  Note that
<command>doclifter</command> does not change the date.</para>

<para><command>doclifter</command> performs special parsing when it
recognizes a synopsis section. It repeatedly tries to parse first a
function synopsis, then a command synopsis, and then plain text off
what remains in the section.</para>

<para>The following man macros are translated into emphasis tags with
a remap attribute: <markup>.B</markup>, <markup>.I</markup>,
<markup>.L</markup>, <markup>.BI</markup>, <markup>.BR</markup>,
<markup>.BL</markup>, <markup>.IB</markup>, <markup>.IR</markup>,
<markup>.IL</markup>, <markup>.RB</markup>, <markup>.RI</markup>,
<markup>.RL</markup>, <markup>.LB</markup>, <markup>.LI</markup>,
<markup>.LR</markup>, <markup>.SB</markup>, <markup>.SM</markup>.
Some stereotyped patterns involving these macros are recognized and
turned into semantic markup.</para>

<para>The following macros are translated into paragraph breaks:
<markup>.LP</markup>, <markup>.PP</markup>, <markup>.P</markup>,
<markup>.HP</markup>, and the single-argument form
of <markup>.IP</markup>.</para>

<para>The two-argument form of <markup>.IP</markup>
is translated either as a <sgmltag
class='element'>VariableList</sgmltag> (usually) or <sgmltag
class='element'>ItemizedList</sgmltag> (if the tag is the troff bullet
or square character).</para>

<para>The following macros are translated semantically:
<markup>.SH</markup>,<markup>.SS</markup>, <markup>.TP</markup>,
<markup>.UR</markup>, <markup>.UE</markup>, <markup>.UN</markup>,
<markup>.IX</markup>.  A <markup>.UN</markup> call just before
<markup>.SH</markup> or <markup>.SS</markup> sets the ID for the
new section.</para>

<para>The <markup>\*R</markup>, <markup>\*(Tm</markup>,
<markup>\*(lq</markup>, and <markup>\*(rq</markup> symbols are
translated.</para>

<para>The following (purely presentation-level) macros are ignored:
<markup>.PD</markup>,<markup>.DT</markup>.</para>

<para>The <markup>.RS</markup>/<markup>.RE</markup> macros are
translated differently depending on whether or not they precede
list markup.  When <markup>.RS</markup> occurs just before <markup>.TP</markup>
or <markup>.IP</markup> the result is nested lists. Otherwise, the
<markup>.RS</markup>/<markup>.RE</markup> pair is translated into a
<sgmltag>Blockquote</sgmltag> tag-pair.</para>

<para><markup>.DS</markup>/<markup>.DE</markup> is not part of the
documented man macro set, but is recognized because it shows up with
some frequency on legacy man pages from older Unixes.  <!-- It triggers
display processing. --></para>

<para>Certain extension macros originally defined under Ultrix are
translated structurally, including those that occasionally show up on
the manual pages of Linux and other open-source Unixes.
<markup>.EX</markup>/<markup>.EE</markup> (and the synonyms
<markup>.Ex</markup>/<markup>.Ee</markup>),
<markup>.Ds</markup>/<markup>.De</markup>, <!-- cause display
parsing. --> <markup>.NT</markup>/<markup>.NE</markup>,
<markup>.PN</markup>, and <markup>.MS</markup> are translated
structurally.</para>

<para>The following extension macros used by the X distribution are
also recognized and translated structurally: <markup>.FD</markup>,
<markup>.FN</markup>, <markup>.IN</markup>, <markup>.ZN</markup>,
<markup>.hN</markup>, and <markup>.C{</markup>/<markup>.C}</markup>
<!-- triggers display parsing.--> The <markup>.TA</markup> and
<markup>.IN</markup> requests are ignored.</para>

<para>When the man macros are active, any <markup>.Pp</markup> macro
definition containing the request <markup>.PP</markup> will be
ignored. and all instances of <markup>.Pp</markup> replaced with
<markup>.PP</markup>.  Similarly, <markup>.Tp</markup> will be
replaced with <markup>.TP</markup>.  This is the least painful way to
deal with some frequently-encountered stereotyped wrapper definitions
that would otherwise cause serious interpretation problems</para>

<para>Known problem areas with man translation:</para>
<itemizedlist>
<listitem><para>Weird uses of <markup>.TP</markup>.
These will sometime generate invalid XML and sometimes result in
a FIXME comment in the generated XML (a warning message will
also go to standard error).</para></listitem>

<listitem><para>It is debatable how the man macros
<markup>.HP</markup> and <markup>.IP</markup> without tag should be
translated.  We treat them as an ordinary paragraph break. We could
visually simulate a hanging paragraph with list markup, but this would
not be a structural translation.</para></listitem>
</itemizedlist>
</refsect2>

<refsect2><title>Pod2man Translation</title>
<para><command>doclifter</command>
recognizes the extension macros produced by
<command>pod2man</command>
(<markup>.Sh</markup>, <markup>.Sp</markup>, <markup>.Ip</markup>, <markup>.Vb</markup>, <markup>.Ve</markup>) and translates them
structurally.</para>

<para>The results of lifting pages produced by
<command>pod2man</command> should be checked carefully by eyeball,
especially the rendering of command and function
synopses. <command>Pod2man</command> generates rather perverse markup;
<command>doclifter</command>'s struggle to untangle it is sometimes in
vain.</para>

<para>If possible, generate your DocBook from the POD sources.  There
is a <application>pod2docbook</application> module on CPAN that does
this.</para>
</refsect2>

<refsect2><title>Tkman Translation</title>
<para><command>doclifter</command> recognizes the extension macros
used by the Tcl/Tk documentation system: <markup>.AP</markup>,
<markup>.AS</markup>, <markup>.BS</markup>, <markup>.BE</markup>,
<markup>.CS</markup>, <markup>.CE</markup>, <markup>.DS</markup>,
<markup>.DE</markup>, <markup>.SO</markup>, <markup>.SE</markup>,
<markup>.UL</markup>, <markup>.VS</markup>, <markup>.VE</markup>.  The
<markup>.AP</markup>, <markup>.CS</markup>, <markup>.CE</markup>,
<markup>.SO</markup>, <markup>.SE</markup>, <markup>.UL</markup>,
<markup>.QW</markup> and <markup>.PQ</markup>
macros are translated structurally.
<!-- <markup>.CS</markup>/<markup>.CE</markup> triggers display processing. -->
</para>
</refsect2>

<refsect2><title>Mandoc Translation</title>
<para><command>doclifter</command> should be able to do an excellent
job on most
<citerefentry><refentrytitle>mdoc</refentrytitle><manvolnum>7</manvolnum></citerefentry>
pages, because this macro package expresses a lot of semantic
structure.</para>

<para>Known problems with mandoc translation: All
<markup>.Bd</markup>/<markup>.Ed</markup> display blocks are
translated as <sgmltag class='element'>LiteralLayout</sgmltag> tag
pairs<!-- (and trigger display processing) -->.</para>
</refsect2>

<refsect2><title>Ms Translation</title>
<para><command>doclifter</command> does a good job on most ms pages.
One weak spot to watch out for is the generation of Author and
Affiliation tags.  The heuristics used to mine this information out of
the <markup>.AU</markup> section work for authors
who format their names in the way usual for English
(e.g. "M. E. Lesk", "Eric S. Raymond") but are quite brittle.</para>

<para>For a document to be recognized as containing ms markup, it must
have the extension <filename>.ms</filename>.  This avoids problems 
with false positives.</para>

<para>The <markup>.TL</markup>, <markup>.AU</markup>,
<markup>.AI</markup>, and <markup>.AE</markup> macros turn into
article metainformation in the expected way.  The
<markup>.PP</markup>, <markup>.LP</markup>, <markup>.SH</markup>, and
<markup>.NH</markup> macros turn into paragraph and section structure.
The tagged form of <markup>.IP</markup> is translated either as a
<sgmltag class='element'>VariableList</sgmltag> (usually) or <sgmltag
class='element'>ItemizedList</sgmltag> (if the tag is the troff bullet
or square character); the untagged version is treated as an ordinary
paragraph break.</para>

<para>The <markup>.DS</markup>/<markup>.DE</markup> pair is translated
to a <sgmltag class='element'>LiteralLayout</sgmltag> tag pair<!-- and
triggers display processing -->.  The
<markup>.FS</markup>/<markup>.FE</markup> pair is translated to a
<sgmltag class='element'>Footnote</sgmltag> tag pair.  The
<markup>.QP</markup>/<markup>.QS</markup>/<markup>.QE</markup>
requests define <sgmltag class='element'>BlockQuotes</sgmltag>.</para>

<para>The <markup>.UL</markup> font change is mapped to U.
<markup>.SM</markup> and <markup>.LG</markup> become numeric plus or
minus size steps suffixed to the <sgmltag
class='attribute'>Remap</sgmltag> attribute.</para>

<para>The <markup>.B1</markup> and <markup>.B2</markup> box macros are
translated to a <sgmltag class='element'>Sidebar</sgmltag> tag
pair.</para>

<para>All macros relating to page footers, multicolumn mode, and keeps
are ignored (<markup>.ND</markup>, <markup>.DA</markup>,
<markup>.1C</markup>, <markup>.2C</markup>, <markup>.MC</markup>,
<markup>.BX</markup>, <markup>.KS</markup>, <markup>.KE</markup>,
<markup>.KF</markup>).  The <markup>.R</markup>, <markup>.RS</markup>,
and <markup>.RE</markup> macros are ignored as well.
</para>
</refsect2>

<refsect2><title>Me Translation</title>
<para>Translation of me documents tends to produce crude results that need
a lot of hand-hacking.  The format has little usable structure, and
documents written in it tend to use a lot of low-level troff macros;
both these properties tend to confuse <command>doclifter</command>.</para>

<para>For a document to be recognized as containing me markup, it must
have the extension <filename>.me</filename>.  This avoids problems 
with false positives.</para>

<para>The following macros are translated into paragraph breaks:
<markup>.lp</markup>, <markup>.pp</markup>.  The <markup>.ip</markup>
macro is translated into a <sgmltag
class='element'>VariableList</sgmltag>.  The <markup>.bp</markup>
macro is translated into an <sgmltag
class='element'>ItemizedList</sgmltag>.  The <markup>.np</markup>
macro is translated into an <sgmltag
class='element'>OrderedList</sgmltag>.</para>

<para>The b, i, and r fonts are mapped to emphasis tags with B, I, and
R <sgmltag class='attribute'>Remap</sgmltag> attributes.  The
<markup>.rb</markup> ("real bold") font is treated the same as
<markup>.b</markup>.</para>

<para><markup>.q(</markup>/<markup>.q)</markup> is translated
structurally <!-- triggers display processing -->.</para>

<para>Most other requests are ignored.</para>
</refsect2>

<refsect2><title>Mm Translation</title>
<para>Memorandum Macros documents translate well, as these macros
carry a lot of structural information.  The translation rules are
tuned for Memorandum or Released Paper styles; information associated
with external-letter style will be preserved in comments.</para>

<para>For a document to be recognized as containing mm markup, it must
have the extension <filename>.mm</filename>.  This avoids problems 
with false positives.</para>

<para>The following highlight macros are translated int Emphasis tags:
<markup>.B</markup>, <markup>.I</markup>, <markup>.R</markup>,
<markup>.BI</markup>, <markup>.BR</markup>, <markup>.IB</markup>,
<markup>.IR</markup>, <markup>.RB</markup>, <markup>.RI</markup>.
</para>

<para>The following macros are structurally translated:
<markup>.AE</markup>, <markup>.AF</markup>, <markup>.AL</markup>,
<markup>.RL</markup>, <markup>.APP</markup>, <markup>.APPSK</markup>,
<markup>.AS</markup>, <markup>.AT</markup>, <markup>.AU</markup>,
<markup>.B1</markup>, <markup>.B2</markup>, <markup>.BE</markup>,
<markup>.BL</markup>, <markup>.ML</markup>, <markup>.BS</markup>,
<markup>.BVL</markup>, <markup>.VL</markup>, <markup>.DE</markup>,
<markup>.DL</markup> <markup>.DS</markup>, <markup>.FE</markup>,
<markup>.FS</markup>, <markup>.H</markup>, <markup>.HU</markup>,
<markup>.IA</markup>, <markup>.IE</markup>, <markup>.IND</markup>,
<markup>.LB</markup>, <markup>.LC</markup>, <markup>.LE</markup>,
<markup>.LI</markup>, <markup>.P</markup>, <markup>.RF</markup>,
<markup>.SM</markup>, <markup>.TL</markup>, <markup>.VERBOFF</markup>,
<markup>.VERBON</markup>, <markup>.WA</markup>, <markup>.WE</markup>.
<!-- <markup>.DS</markup>/<markup>.DE</markup> triggers display processing. -->
</para>

<para>The following macros are ignored:</para>

<para>&nbsp;<markup>.)E</markup>, <markup>.1C</markup>,
<markup>.2C</markup>, <markup>.AST</markup>, <markup>.AV</markup>,
<markup>.AVL</markup>, <markup>.COVER</markup>,
<markup>.COVEND</markup>, <markup>.EF</markup>, <markup>.EH</markup>,
<markup>.EDP</markup>, <markup>.EPIC</markup>, <markup>.FC</markup>,
<markup>.FD</markup>, <markup>.HC</markup>, <markup>.HM</markup>,
<markup>.GETR</markup>, <markup>.GETST</markup>, <markup>.HM</markup>,
<markup>.INITI</markup>, <markup>.INITR</markup>,
<markup>.INDP</markup>, <markup>.ISODATE</markup>,
<markup>.MT</markup>, <markup>.NS</markup>, <markup>.ND</markup>,
<markup>.OF</markup>, <markup>.OH</markup>, <markup>.OP</markup>,
<markup>.PGFORM</markup>, <markup>.PGNH</markup>,
<markup>.PE</markup>, <markup>.PF</markup>, <markup>.PH</markup>,
<markup>.RP</markup>, <markup>.S</markup>, <markup>.SA</markup>,
<markup>.SP</markup>, <markup>.SG</markup>, <markup>.SK</markup>,
<markup>.TAB</markup>, <markup>.TB</markup>, <markup>.TC</markup>,
<markup>.VM</markup>, <markup>.WC</markup>.</para>

<para>The following macros generate warnings: <markup>.EC</markup>,
<markup>.EX</markup>, <markup>.GETHN</markup>,
<markup>.GETPN</markup>, <markup>.GETR</markup>,
<markup>.GETST</markup>, <markup>.LT</markup>, <markup>.LD</markup>,
<markup>.LO</markup>, <markup>.MOVE</markup>, <markup>.MULB</markup>,
<markup>.MULN</markup>, <markup>.MULE</markup>,
<markup>.NCOL</markup>, <markup>.nP</markup>, <markup>.PIC</markup>,
<markup>.RD</markup>, <markup>.RS</markup>, <markup>.RE</markup>, 
<markup>.SETR</markup>
</para>

<para>Pairs of <markup>.DS</markup>/<markup>.DE</markup> are
interpreted as informal figures.  If an <markup>.FG</markup> is
present it becomes a caption element.</para>

<para>&nbsp;<markup>.BS</markup>/<markup>.BE</markup> and
<markup>.IA</markup>/<markup>.IE</markup> pairs are passed through.
The text inside them may need to be deleted or moved.</para>

<para>The mark argument of <markup>.ML</markup> is
ignored; the following list id formatted as a normal <sgmltag
class='element'>ItemizedList</sgmltag>.</para>

<para>The contents of <markup>.DS</markup>/<markup>.DE</markup> or
<markup>.DF</markup>/<markup>.DE</markup> gets turned into a <sgmltag
class='element'>Screen</sgmltag> display.  Arguments controlling
presentation-level formatting are ignored.</para>

</refsect2>

<refsect2><title>Mwww Translation</title> <para>The mwww macros are an
extension to the man macros supported by
<citerefentry><refentrytitle>groff</refentrytitle><manvolnum>1</manvolnum></citerefentry>
for producing web pages.</para>

<para>The <markup>URL</markup>, <markup>FTP</markup>,
<markup>MAILTO</markup>, <markup>FTP</markup>, <markup>IMAGE</markup>,
<markup>TAG</markup> tags are translated structurally.  The
<markup>HTMLINDEX</markup>, <markup>BODYCOLOR</markup>,
<markup>BACKGROUND</markup>, <markup>HTML</markup>, and
<markup>LINE</markup> tags are ignored.</para>
</refsect2>

<refsect2><title>TBL Translation</title> 

<para>All structural features of TBL tables are translated, including
both horizontal and vertical spanning with &lsquo;s&rsquo; and
&lsquo;^&rsquo;.  The &lsquo;l&rsquo;, &lsquo;r&rsquo;, and
&lsquo;c&rsquo; formats are supported; the &lsquo;n&rsquo; column
format is rendered as &lsquo;r&rsquo;. Line continuations with
<sgmltag class='element'>T{</sgmltag> and <sgmltag
class='element'>T}</sgmltag> are handled correctly.  So is
<markup>.TH</markup>.</para>

<para>The <markup>expand</markup>, <markup>box</markup>,
<markup>doublebox</markup>, <markup>allbox</markup>,
<markup>center</markup>, <markup>left</markup>, and
<markup>right</markup> options are supported.  The GNU synonyms
<markup>frame</markup> and <markup>doubleframe</markup> are also
recognized.  But the distinction between single and double rules and
boxes is lost.</para>

<para>Table continuations (<sgmltag class='element'>.T&amp;</sgmltag>)
are not supported.</para>

<para>If the first nonempty line of text immediately before a table is
boldfaced, it is interpreted as a title for the table and the table
is generated using a <sgmltag class='element'>table</sgmltag> and
<sgmltag class='element'>title</sgmltag>.  Otherwise the table is
translated with <sgmltag
class='element'>informaltable</sgmltag>.</para>

<para>Most other presentation-level TBL commands are ignored.
The &lsquo;b&rsquo; format qualifier is processed, but point size and width 
qualifiers are not.</para>
</refsect2>

<refsect2><title>Pic Translation</title>
<para>PIC sections are translated to SVG.
<application>doclifter</application> calls out to
<citerefentry><refentrytitle>pic2plot</refentrytitle><manvolnum>1</manvolnum></citerefentry>
to accomplish this; you must have that utility installed for PIC
translation to work.</para>
</refsect2>

<refsect2><title>Eqn Translation</title> <para>EQN sections are
filtered into embedded MathML with 
<command>eqn -TMathML</command> if possible, otherwise passed
through enclosed in <sgmltag class='element'>LiteralLayout</sgmltag> tags.
After a delim statement has been seen, inline eqn delimiters are
translated into an XML processing instruction. Exception: inline
eqn equations consisting of a single character are translated to an
<sgmltag>Emphasis</sgmltag> with a Role attribute of eqn.</para>
</refsect2>

<refsect2><title>Troff Translation</title>
<para>The troff translation is meant only to support interpretation of the
macro sets. It is not useful standalone.</para>

<para>The <markup>.nf</markup> and <markup>.fi</markup> macros are
interpreted as literal-layout boundaries.  Calls to the
<markup>.so</markup> macro either cause inclusion or are translated
into XML entity inclusions (see above).  Calls to the
<markup>.ul</markup> and <markup>.cu</markup> macros cause following
lines to be wrapped in an <sgmltag class='element'>Emphasis</sgmltag>
tag with a <sgmltag class='attribute'>Remap</sgmltag> attribute of
"U".  Calls to <markup>.ft</markup> generate corresponding start or
end emphasis tags.  Calls to <markup>.tr</markup> cause character
translation on output. Calls to <markup>.bp</markup> generate a
<sgmltag class='element'>BeginPage</sgmltag> tag (in paragraphed text
only). Calls to <markup>.sp</markup> generate a paragraph break (in
paragraphed text only).  Calls to <markup>.ti</markup> wrap the
following line in a <sgmltag class='element'>BlockQuote</sgmltag>
These are the only troff requests we translate to DocBook.  The rest
of the troff emulation exists because macro packages use it internally
to expand macros into elements that might be structural.</para>

<para>Requests relating to macro definitions and strings
(<markup>.ds</markup>, <markup>.as</markup>, <markup>.de</markup>,
<markup>.am</markup>, <markup>.rm</markup>, <markup>.rn</markup>,
<markup>.em</markup>) are processed and expanded.  The
<markup>.ig</markup> macro is also processed.</para>

<para>Conditional macros (<markup>.if</markup>, <markup>.ie</markup>,
<markup>.el</markup>) are handled.  The built-in conditions o, n, t,
e, and c are evaluated as if for <application>nroff</application> on
page one of a document.  The m, d, and r troff conditionals
are also interpreted. String comparisons are evaluated by straight
textual comparison.  All numeric expressions evaluate to true. </para>

<para>The extended <application>groff</application> requests
<markup>cc</markup>, <markup>c2</markup>, 
<markup>ab</markup>, <markup>als</markup>, <markup>do</markup>,
<markup>nop</markup>, and <markup>return</markup> and
<markup>shift</markup> are interpreted. Its <markup>.PSPIC</markup>
extension is translated into a <sgmltag
class='element'>MediaObject</sgmltag>.</para>

<para>The <markup>.tm</markup> macro writes its arguments to standard
error (with <option>-t</option>).  The <markup>.pm</markup> macro
reports on defined macros and strings.  These facilities may aid in
debugging your translation.</para>

<para>Some troff escape sequences are lifted:</para>

<orderedlist>
<listitem><para>The \e and \\ escapes become a bare backslash, \. a
period, and \- a bare dash.</para></listitem>

<listitem><para>The troff escapes \^, \`, \' \&amp;, \0, and \| are lifted 
to equivalent ISO special spacing characters.</para></listitem>

<listitem><para>A \ followed by space is translated to an ISO non-breaking 
space entity.</para></listitem>

<listitem><para>A \~ is also translated to an ISO non-breaking space
entity; properly this should be a space that can't be used for a
linebreak but stretches like ordinary whitepace during line
adjustment, but there is no ISO or Unicode entity for
that.</para></listitem>

<listitem><para>The \u and \d half-line motion vertical motion
escapes, when paired, become <markup>Superscript</markup> or
<markup>Subscript</markup> tags.</para></listitem>

<listitem><para>The \c escape is handled as a line continuation. in
circumstances where that matters (e.g. for
token-pasting).</para></listitem>

<listitem><para>The \f escape for font changes is translated in
various context-dependent ways. First, <command>doclifter</command>
looks for cliches involving font changes that have semantic meaning,
and lifts to a structural tag.  If it can't do that, it generates an
<sgmltag>Emphasis</sgmltag> tag.</para></listitem>

<listitem><para>The \m[] extension is translated into a
<sgmltag>phrase</sgmltag> span with a remap attribute carrying the
color.  Note: Stylesheets typically won't render
this!</para></listitem>

<listitem><para>Some uses of the \o request are translated: pairs with
a letter followed by one of the characters ` ' : ^ o ~ are translated
to combining forms with diacriticals acute, grave, umlaut, circumflex,
ring, and tilde respectively if the corresponding Latin-1 or Latin-2
character exists as an ISO literal.</para></listitem>

</orderedlist>

<para>Other escapes than these will yield warnings or errors.</para>

<para>All other troff requests are ignored but passed through into
XML comments.  A few (such as <markup>.ce</markup>) also trigger
a warning message.</para>
</refsect2>
</refsect1>

<refsect1 id="portability"><title>Portability Checking</title>

<para>When portability checking is enabled,
<command>doclifter</command> emits portability warnings about markup
which it can handle but which will break various other viewers and
interpreters.</para>

<orderedlist>
<listitem><para>At level 1, it will warn about constructions that
would break 
<citerefentry><refentrytitle>man2html</refentrytitle><manvolnum>1</manvolnum></citerefentry>,
(the C program distributed with Linux
<citerefentry><refentrytitle>man</refentrytitle><manvolnum>1</manvolnum></citerefentry>,
not the older and much less capable Perl script).  A close derivative of this
code is used in GNOME <application>yelp</application>.  This should
be the minimum level of portability you aim for, and corresponds to what is 
recommended on the 
<citerefentry><refentrytitle>groff_man</refentrytitle><manvolnum>7</manvolnum></citerefentry>
manual page.
</para></listitem>
<listitem><para>At level 2, it will warn about constructions that will
break portability back to the Unix classic tools (including long macro
names and glyph references with \[]).</para></listitem>
</orderedlist>
</refsect1>

<refsect1 id="hints"><title>Semantic analysis</title>
<para><command>doclifter</command> keeps two lists of semantic hints
that it picks up from analyzing source documents (especially from
parsing command and function synopses).  The local list includes:</para>

<itemizedlist>
<listitem><para>Names of function formal arguments</para></listitem>
<listitem><para>Names of command options</para></listitem>
</itemizedlist>

<para>Local hints are used to mark up the individual page from 
which they are gathered. The global list includes:</para>

<itemizedlist>
<listitem><para>Names of functions</para></listitem>
<listitem><para>Names of commands</para></listitem>
<listitem><para>Names of function return types</para></listitem>
</itemizedlist>

<para>If <command>doclifter</command> is applied to multiple files,
the global list is retained in memory.  You can dump a report of
global hints at the end of the run with the <option>-h</option>
option.  The format of the hints is as follows:</para>

<programlisting>
&nbsp;.\&quot; | mark &lt;phrase&gt; as &lt;markup&gt;
</programlisting>

<para>where <userinput>&lt;phrase&gt;</userinput> is an item of text
and <userinput>&lt;markup&gt;</userinput> is the DocBook markup text
it should be wrapped with whenever it appeared either highlighted
or as a word surrounded by whitespace in the source text.</para>

<para>Hints derived from earlier files are also applied to later ones.
This behavior may be useful when lifting collections of documents that
apply to a function or command library.  What should be more useful is
the fact that a hints file dumped with <option>-h</option> can be one of
the file arguments to <command>doclifter</command>; the code detects
this special case and does not write XML output for such a file.
Thus, a good procedure for lifting a large library is to generate a 
hints file with a first run, inspect it to delete false positives, and
use it as the first input to a second run.</para>  

<para>It is also possible to include a hints file directly in a troff
sourcefile.  This may be useful if you want to enrich the file by
stages before converting to XML.</para>

</refsect1>

<refsect1 id="troubleshooting"><title>Troubleshooting</title>

<para><command>doclifter</command> tries to warn about problems that
it can can diagnose but not fix by itself.  When it says
<computeroutput>"look for FIXME"</computeroutput>, do that in the
generated XML; the markup around that token may be wrong.</para>

<para>Occasionally (less than 2% of the time)
<command>doclifter</command> will produce invalid DocBook markup even
from correct troff markup.  Usually this results from strange
constructions in the source page, or macro calls that are beyond the
ability of <command>doclifter</command>'s macro processor to get
right.  Here are some things to watch for, and how to fix them:</para>

<refsect2><title>Malformed command synopses.</title>

<para>If you get a
message that says <computeroutput>"command synopsis parse
failed"</computeroutput>, try rewriting the synopsis in your manual
page source.  The most common cause of failure is unbalanced []
groupings, a bug that can be very difficult to notice by eyeball.  To
assist with this, the error message includes a token number in
parentheses indicating on which token the parse failed.</para>  

<para>For more information, use the -v option.  This will trigger
a dump telling you what the command synopsis looked like after
preprocessing, and indicate on which token the parse failed (both with
a token number and a caret sign inserted in the dump of the synopsis
tokens).  Try rewriting the synopsis in your manual page source.  The
most common cause of failure is unbalanced [] groupings, a bug that
can be very difficult to notice by eyeball.  To assist with this, the
error token dump tries to insert &lsquo;$&rsquo; at the point of the
last nesting-depth increase, but the code that does this is
failure-prone.</para>

</refsect2>
<refsect2><title>Confusing macro calls.</title>

<para>Some manual page authors replace standard requests (like
<markup>.PP</markup>, <markup>.SH</markup> and <markup>.TP</markup>)
with versions that do different things in <command>nroff</command> and
<command>troff</command> environments.  While
<command>doclifter</command> tries to cope and usually does a good
job, the quirks of [nt]roff are legion and confusing macro calls
sometimes lead to bad XML being generated. A common symptom of such
problems is unclosed <sgmltag class='element'>Emphasis</sgmltag>
tags.</para>

</refsect2>
<refsect2><title>Malformed list syntax.</title>

<para> The manual-page parser can be confused by <markup>.TP</markup>
constructs that have header tags but no following body.  If the XML
produced doesn't validate, and the problem seems to be a misplaced
<sgmltag>listitem</sgmltag> tag, try using the verbose (-v) option.
This will enable line-numbered warnings that may help you zero in on
the problem.</para>

</refsect2>
<refsect2><title>Section nesting problems with SS.</title>

<para>The message <computeroutput>"possible section nesting
error"</computeroutput> means that the program has seen two adjacent
subsection headers.  In man pages, subsections don't have a depth
argument, so <command>doclifter</command> cannot be certain how 
subsections should be nested. Any subsection heading between the
indicated line and the beginning of the next top-level section might
be wrong and require correcting by hand.</para>

</refsect2>
<refsect2><title>Bad output with no doclifter error message</title>

<para>If you're translating a page that uses user-defined macros, and
doclifter fails to complain about it but you get bad output, the first
thing to do is simplify or eliminate the user-defined macros.  Replace
them with stock requests where possible.</para>

</refsect2>
</refsect1>

<refsect1 id='improving'><title>Improving Translation Quality</title> 

<para>There are a few constructions that are a good idea to check by hand
after lifting a page.</para>

<para>Look near the <sgmltag class='element'>BlockQuote</sgmltag> tags.
The troff temporary indent request (<markup>.ti</markup>) is translated
into a <sgmltag class='element'>BlockQuote</sgmltag> wrapper around the
following line.  Sometimes <sgmltag class='element'>LiteralLayout</sgmltag>
or <sgmltag class='element'>ProgramListing</sgmltag> would be a better
translation, but <command>doclifter</command> has no way to know this.
</para>

<para>It is not possible to unambiguously detect candidates for
wrapping in a DocBook <sgmltag>option</sgmltag> tag in running
text. If you care, you'll have to check for these and fix them by
hand.</para>

<!-- para>The troff-level <markup>.nf</markup>/<markup>.fi</markup> macros
don't trigger display parsing (doing so would wildly complicate macro
interpretation).  If you are translating a document that uses them to wrap
function synopses, you can improve the translation by hand-hacking
them to <markup>.DS</markup>/<markup>.DE</markup> or the equivalent in
whatever macro set is active.</para -->

</refsect1>

<refsect1><title>Bugs And Limitations</title>

<para>About 3% of man pages will either make this program throw error status
1 or generate invalid XML. In almost all such cases the misbehavior is
triggered by markup bugs in the source that are too severe to be 
coped with.</para>

<para>Equation number arguments of EQN calls are ignored.</para>

<para>Semicolon used as a TBL field separator will lead to garbled
tables. The easiest way to fix this is by patching the source.</para>

<para>The function-synopsis parser is crude (it's not a compiler) and
prone to errors.  Function-synopsis markup should be checked carefully
by a human.</para>

<para>If a man page has both paragraphed text in a Synopsis section
and also a body section before the Synopis section, bad things will
happen.</para>

<para>Running text (e.g., explanatory notes) at the end of a Synopsis
section cannot reliably be distinguished from synopsis-syntax
markup. (This problem is AI-complete.)</para>

<para>Some firewalls put in to cope with common malformations in troff
code mean that the tail end of a span between two
<markup>\f{B,I,U,(CW}</markup> or <markup>.ft</markup> highlight
changes may not be completely covered by corresponding <sgmltag
class='element'>Emphasis</sgmltag> macros if (for example) the span
crosses a boundary between filled and unfilled
(<markup>.nf</markup>/<markup>.fi</markup>) text.</para>

<para>The treatment of conditionals relies on the assumption that
conditional macros never generate structural or font-highlight markup
that differs between the if and else branches.  This appears to be
true of all the standard macro packages, but if you roll any of your
own macros you're on your own.</para>

<para>Macro definitions in a manual page NAME section are not
interpreted.</para>

<para>Uses of \c for line continuation sometimes are not translated,
leaving the \c in the output XML. The program will print a warning 
when this occurs.</para>

<para>It is not possible to unambiguously detect candidates for
wrapping in a DocBook <sgmltag>option</sgmltag> tag in running
text. If you care, you'll have to check for these and fix them by
hand.</para>

<para>The line numbers in <command>doclifter</command> error messages
are unreliable in the presence of <markup>.EQ/.EN</markup>, 
<markup>.PS/.PE</markup>, and quantum fluctuations.</para>
</refsect1>

<refsect1><title>Old macro sets</title> 
<para>There is a conflict between Berkeley ms's documented
<markup>.P1</markup> print-header-on-page request and an undocumented
Bell Labs use for displayed program and equation listings.  The
<emphasis remap='B'>ms</emphasis> translator uses the Bell Labs
interpretation when <markup>.P2</markup> is present in the document,
and otherwise ignores the request.</para>
</refsect1>

<refsect1><title>Return Values</title>
<para>On successful completion, the program returns status 0.  
It returns 1 if some file or standard input could not be translated.  
It returns 2 if one of the input sources was a <markup>.so</markup> inclusion.
It returns 3 if there is an error in reading or writing files.  
It returns 4 to indicate an internal error.
It returns 5 when aborted by a keyboard interrupt. </para>

<para>Note that a zero return does not guarantee that the output is
valid DocBook.  It will almost always (as in, more than 98% of cases)
be syntactically valid XML, but in some rare cases fixups by hand may be
necessary to meet the semantics of the DocBook DTD.  Validation
problems are most likely to occur with complicated list markup.</para>
</refsect1>

<refsect1><title>Requirements</title>
<para>The
<citerefentry><refentrytitle>pic2plot</refentrytitle><manvolnum>1</manvolnum></citerefentry>
utility must be installed in order to translate PIC diagrams to SVG.</para>
</refsect1>

<refsect1><title>See Also</title>
<para><citerefentry><refentrytitle>man</refentrytitle><manvolnum>7</manvolnum></citerefentry>,
<citerefentry><refentrytitle>mdoc</refentrytitle><manvolnum>7</manvolnum></citerefentry>,
<citerefentry><refentrytitle>ms</refentrytitle><manvolnum>7</manvolnum></citerefentry>,
<citerefentry><refentrytitle>me</refentrytitle><manvolnum>7</manvolnum></citerefentry>,
<citerefentry><refentrytitle>mm</refentrytitle><manvolnum>7</manvolnum></citerefentry>,
<citerefentry><refentrytitle>mwww</refentrytitle><manvolnum>7</manvolnum></citerefentry>,
<citerefentry><refentrytitle>troff</refentrytitle><manvolnum>1</manvolnum></citerefentry>.</para>
</refsect1>

<refsect1><title>Author</title>
<para>Eric S. Raymond <email>esr@thyrsus.com</email></para>