Commit 97f0e5a5 authored by Rick van Rein's avatar Rick van Rein
Browse files

Merge branch 'fixes' into 'master'

Fixes: Bugs, improvements

  * Upgraded Pypeline: Colour, Regex fork, Exit detect
  * use dynamic ports for fD client and server
  * haan uses it's own diasasl-extension-server-haan.conf
  * create sasldb2 using cmake
  * Solved a netly interaction of buffered/unbuffered
  * Set SCTP and ListenTo in freeDiameter
  * Corrected TCP:DIAMETER to SCTP:DIAMETER
  * Update of Pypeline, with SCTP improvement
  * Rename GS2-SXOVER-PLUS --> SXOVER-PLUS
  * Simplified freeDiameter test scripting

See merge request !44
parents 33ebd4c9 199118aa
Pipeline #249775431 passed with stage
in 4 minutes and 40 seconds
......@@ -11,6 +11,29 @@ program in a fixed order, and synchronises with the programs to
avoid race conditions. The programs need to be built for the
interaction, though.
[Pypeline project page](https://gitlab.com/arpa2/pypeline)
<!--
SPDX-License: CC-BY-SA-4.0
SPDX-Copyright: 2020 Rick van Rein <rick@openfortress.nl>
-->
## Suggested Use of Pypeline
We suggest that you make a local copy of `pypeline` and
`PYPELINE.MD` in the `test/` directory of your project. Use it
to setup any configuration of daemons that pleases you. Add
it to your repository.
You do not want to be disturbed by (incompatible) changes to
the Pypeline repository, and this way you will be shielded
from any of that. You can always copy new versions if you
like some new feature, of course.
Likewise, any changes you made since your last copy may
develop locally and when it seems generally useful it may
be proposed as a Merge Request. Only too glad to consider.
## Program Interaction: Initial, FreeRun, Results
......@@ -117,11 +140,11 @@ The following modifiers can be used as part of the Pypeline
commandline. They are usually placed before a command, though
versions that can produce text may also be placed on the line.
* `PYPE:FORK:<line>` sets up the `<line>` as the output to
match from the program before entering the free running
staged. The text is supposed to match the exact line as
it appears between newlines. Two special forms can be
used to be less literal; `PYPE:FORK:^` refers to the
* `PYPE:FORK:<regex>` sets up the `<regex>` as the output
match in a program line before entering the free running
stage. See Python module `re` for grammar; the default
is `--` as a separate line, so `^--$` as a pattern.
Two dedicated values exist; `PYPE:FORK:^` refers to the
start of the program, so full concurrency and
`PYPE:FORK:$` refers to the end of the program, so full
sequencing. Be careful to quote at least the last form,
......@@ -150,6 +173,17 @@ versions that can produce text may also be placed on the line.
be used after the command to insert the (previous)
value of `<progname>`.
The `<progname>` is also used to pick a colour for the
process. This colour would be the same across runs, so
it can be useful to consistently name processes over
many runs.
* `PYPE:COL:<colname>` requests the given colour name,
if it is not used elsewhere yet. This is preferred over
colour choice based on `PYPE:NAME:` settings. Colours
may not be available, and a simple incremental counter
is used as a fallback mechanism.
* `PYPE:ESC:<text>` can be used as a command or argument
to produce the literal `<text>` without it being interpreted
by Pypeline. You might for example use `PYPE:ESC:--` to
......
......@@ -20,7 +20,7 @@ Quick SASL interface does however set it in its general API call
QuickSASL may also choose other, software-related names.
* `InternetWide_Identity` is the default name for XoverSASL.
This offers the mechanism `GS2-SXOVER-PLUS` that offers an
This offers the mechanism `SXOVER-PLUS` that offers an
end-to-end encrypted/authenticated tunnel that wraps another
SASL mechanism. This setup is secure for realm crossover.
For the inner layer, so for the SASL exchange that passes
......
......@@ -21,7 +21,7 @@ KRB-FX-CF2 from RFC 6113 as its formal definition.
Clients need a pre-arranged secret for end-to-end encryption of
the tunnel used to protect Realm Crossover. This is the primary
value added by the SASL mechanism GS2-SXOVER-PLUS.
value added by the SASL mechanism SXOVER-PLUS.
The secret is not as good as the entire exchange however; it is
like in most TLS use cases, where the server is authenticated
......@@ -67,10 +67,10 @@ just a form of laziness in the initial coding.
With KIP Service using Realm Crossover for SASL, along with
Realm Crossover for SASL using KIP Service, there appears
to be a danger of inifinite nesting. There is not, in
practice, as every nesting depends on the GS2-SXOVER-PLUS
practice, as every nesting depends on the SXOVER-PLUS
mechanism, which is never going to be infinite. In fact,
the specification does not even allow GS2-SXOVER-PLUS to
pass GS2-SXOVER-PLUS as its own inner mechanism, because
the specification does not even allow SXOVER-PLUS to
pass SXOVER-PLUS as its own inner mechanism, because
extra indirections could interfere with traceability of
clients to administrators and would benefit abuse but not
proper use.
......
......@@ -10,13 +10,13 @@ may be used in SASL.
**Note:** This is presently just an idea!
## GS2-GENERATE
## GENERATE
To generate a new identity, either SASL or GSSAPI can be used.
The protocol is straightforward:
```
C: mech="GS2-GENERATE", c2s="unicorn.demo.arpa2.lab"
C: mech="GENERATE", c2s="unicorn.demo.arpa2.lab"
S: success,s2c="pub-lic-id sec-ret-part"
```
......@@ -25,7 +25,7 @@ secret. The public part and realm are communicated to an
application that may store it in a password maneger for day-to-day
use, or produce a PDF to print.
This protocol may even be run over GS2-SXOVER-PLUS, so that the
This protocol may even be run over SXOVER-PLUS, so that the
backend is known to be reliable and thus the identity supplied
comes from the targeted realm. In this case, the `c2s` token
must match that requested by the surrounding Diameter exchange.
......@@ -33,7 +33,7 @@ When returning, the username can be supplied to the foreign
server, which may continue to use it.
Is there a usecase for direct use of this connection? If so,
it might call for a GS2-GENERATE-PLUS mechanism and the
it might call for a GENERATE-PLUS mechanism and the
requirement to run it over a secure channel like TLS.
This does have the benefit of avoiding the browser, which is
quite attractive for PDF printing. Even in such cases, it
......
......@@ -128,7 +128,7 @@ void xsasl_close (XoverSASL *inout_xs);
/* Make a step as a SASL Client.
*
* This is a wrapper around the Quick SASL function that
* adds GS2-SXOVER-PLUS mechanism support.
* adds SXOVER-PLUS mechanism support.
*
* The return value is true for success passing through the
* programming steps, as an accumulated assertion of the entire
......@@ -152,7 +152,7 @@ bool xsasl_step_client (XoverSASL xs,
/* Make a step as a SASL Server.
*
* This is a wrapper around the Quick SASL function that
* adds GS2-SXOVER-PLUS mechanism support.
* adds SXOVER-PLUS mechanism support.
*
* The return value is true for success passing through the
* programming steps, as an accumulated assertion of the entire
......
......@@ -55,7 +55,7 @@ static bool split_commas (const uint8_t *token, uint32_t tokenlen, int commacoun
}
/* Parse the first C2S token in the GS2-SXOVER-PLUS mechanism.
/* Parse the first C2S token in the SXOVER-PLUS mechanism.
* Leave the output fields (realm/len and cbtyp/len) as they
* are when returning early.
*/
......@@ -100,7 +100,7 @@ static void do_sxover (const uint8_t *c2s0, uint32_t c2s0len,
* learn from the fit4plain flag.
*/
struct xoverpass xoverpassfun [] = {
{ "GS2-SXOVER-PLUS", do_sxover, true },
{ "SXOVER-PLUS", do_sxover, true },
{ NULL, NULL, false }
};
......
......@@ -33,7 +33,7 @@
#include "kip-dbg.h"
#define SXOVER_NAME "GS2-SXOVER-PLUS"
#define SXOVER_NAME "SXOVER-PLUS"
#define SXOVER_LEN (strlen (SXOVER_NAME))
......@@ -212,7 +212,7 @@ void xsasl_close (XoverSASL *inout_xs) {
}
/* Check if a dercursor contains the string "GS2-SXOVER-PLUS".
/* Check if a dercursor contains the string "SXOVER-PLUS".
*/
static bool mechlist_has_sxover (dercursor mechlist, size_t *where) {
int ofs = mechlist.derlen - SXOVER_LEN;
......@@ -475,7 +475,7 @@ static void setup_kipservice_clientid (XoverSASL xs) {
/* Make a step as a SASL Client.
*
* This is a wrapper around the Quick SASL function that
* adds GS2-SXOVER-PLUS mechanism support.
* adds SXOVER-PLUS mechanism support.
*
* The return value is true for success passing through the
* programming steps, as an accumulated assertion of the entire
......@@ -745,7 +745,7 @@ bool xsasl_step_client (XoverSASL xs,
/* Make a step as a SASL Server.
*
* This is a wrapper around the Quick SASL function that
* adds GS2-SXOVER-PLUS mechanism support.
* adds SXOVER-PLUS mechanism support.
*
* The return value is true for success passing through the
* programming steps, as an accumulated assertion of the entire
......
......@@ -86,7 +86,7 @@ The following protocol messages are defined:
has not reported a final positive or negative result.
* **Authn** is a SASL token interaction. The first includes a
mechanism choice, such as `GS2-SXOVER-PLUS` for Realm Crossover.
mechanism choice, such as `SXOVER-PLUS` for Realm Crossover.
Any may send a token from client to server, and a response token
may be replied. The reply may also be final, either reporting
an error or success. An error should never be accompanied by
......@@ -126,7 +126,7 @@ citizenship on the Internet:
_diapass_mapping = {
'GS2-SXOVER-PLUS': (True, re.compile (
'SXOVER-PLUS': (True, re.compile (
'^p=(?P<cbtyp>[^,]+),,(?P<realm>[^,@]+),')),
}
......@@ -491,7 +491,7 @@ class Session:
will be used by the backend node to determine the
support mechanishm list. The default is an empty
string, which may be configured to support no
mechanisms at all, or GS2-SXOVER-PLUS for full
mechanisms at all, or SXOVER-PLUS for full
Realm Crossover with SASL.
"""
assert isinstance (node, Node)
......
......@@ -288,7 +288,7 @@ static void diasasl_cli_message(dercursor s2c, struct session *sess)
/* Set the SASL-Mechanism AVP */
if (s2c.derptr == NULL)
{
char *sxover = "GS2-SXOVER-PLUS";
char *sxover = "SXOVER-PLUS";
int length = strlen(sxover);
CHECK_FCT_DO( fd_msg_avp_new ( diasasl_avp_sasl_mechanism, 0, &avp ), goto out );
......@@ -389,7 +389,7 @@ int main(int argc, char * argv[])
const char *_lineprompt;
dercursor mech_sxover = {
.derptr = "GS2-SXOVER-PLUS",
.derptr = "SXOVER-PLUS",
.derlen = 15
};
assert (xsasl_set_mech (sasl, mech_sxover));
......
......@@ -214,7 +214,9 @@ void diasasl_cb_open (struct diasasl_session *session,
sasl->mech_choice = opt_mech_choice;
sasl->send_mech_choice = true;
//
// Now rely on the event loop; it calls back relay_c2s_token()
// Do not rely on the event loop yet; it may not call back relay_c2s_token()
// if it is already in the buffer of the c2s_stream and then block
relay_c2s_token (session);
} else {
//
// Report any error
......
This diff is collapsed.
sasldb_path: @CMAKE_CURRENT_SOURCE_DIR@/bin/demo.sasldb2
sasldb_path: @CMAKE_CURRENT_BINARY_DIR@@sasldb@
sasldb_path: @CMAKE_CURRENT_SOURCE_DIR@/bin/demo.sasldb2
sasldb_path: @CMAKE_CURRENT_BINARY_DIR@@sasldb@
sasldb_path: @CMAKE_CURRENT_SOURCE_DIR@/bin/demo.sasldb2
sasldb_path: @CMAKE_CURRENT_BINARY_DIR@@sasldb@
## -------- Test configuration ---------
Identity = "fdcli.arpa2.net";
Realm = "arpa2.net";
Port = 0;
Realm = "@sasldb-realm@";
ListenOn = "#IP:DIAMETER#";
SecPort = 0;
Port = 0;
No_TCP;
TLS_Cred = "@CMAKE_BINARY_DIR@/test/fdcli.cert.pem",
"@CMAKE_BINARY_DIR@/test/fdcli.key.pem";
......@@ -11,5 +13,5 @@ TLS_CA = "@CMAKE_BINARY_DIR@/test/cacert.pem";
LoadExtension = "@CMAKE_BINARY_DIR@/src/diasasl.fdx" : "#FILE:DIACLIEXTCONF#";
ConnectPeer = "fdsrv.unicorn.demo.arpa2.org" { ConnectTo = "#IP:DIASASL#"; port = 3869; };
ConnectPeer = "fdsrv.unicorn.demo.arpa2.org" { ConnectTo = "#IP:DIAMETER#"; port = #SCTP:DIAMETER#; };
......@@ -2,8 +2,9 @@
Identity = "fdsrv.unicorn.demo.arpa2.org";
Realm = "unicorn.demo.arpa2.org";
Port = 3868;
SecPort = 3869;
ListenOn = "#IP:DIAMETER#";
SecPort = "#SCTP:DIAMETER#";
Port = 0;
TLS_Cred = "@CMAKE_BINARY_DIR@/test/fdsrv.cert.pem",
"@CMAKE_BINARY_DIR@/test/fdsrv.key.pem";
......
# -------- Test configuration ---------
## -------- Test configuration ---------
Identity = "fdsrv.unicorn.demo.arpa2.org";
Realm = "unicorn.demo.arpa2.org";
ListenOn = "#IP:DIAMETER#";
SecPort = #SCTP:DIAMETER#;
Port = 0;
SecPort = 3869;
No_TCP;
TLS_Cred = "@CMAKE_BINARY_DIR@/test/fdsrv.cert.pem",
"@CMAKE_BINARY_DIR@/test/fdsrv.key.pem";
TLS_CA = "@CMAKE_BINARY_DIR@/test/cacert.pem";
LoadExtension = "@CMAKE_BINARY_DIR@/src/diasasl.fdx" : "@CMAKE_BINARY_DIR@/test/bin/diasasl-extension-server.conf";
LoadExtension = "@CMAKE_BINARY_DIR@/src/diasasl.fdx" : "#FILE:DIASRVEXTCONF#";
LoadExtension = "acl_wl.fdx" : "@CMAKE_BINARY_DIR@/test/bin/acl_wl-extension.conf";
#! /bin/sh
#
# Usage:
#
# create-sasldb2.sh <filename> <user> <password> <realm>
#
# Does no error-checking at all.
echo "$3" | saslpasswd2 -f "$1" -u "$4" "$2"
......@@ -19,7 +19,8 @@
# a similar fashion, there can be "IP:<key>" to specify the local
# host as an IP address.
#
# From: Rick van Rein <rick@openfortress.nl>
# SPDX-License-Identifier: BSD-2-Clause
# SPDX-FileCopyrightText: 2020, Rick van Rein <rick@openfortress.nl>
import os
......@@ -27,6 +28,7 @@ import sys
import time
import re
import zlib
import socket
import signal
import threading
......@@ -76,10 +78,22 @@ except:
colours = [ 'BLASS' ]
colour_idx = 0
def pick_colour ():
global colours, colour_idx
colour = colours [colour_idx % len (colours)]
colour_idx += 1
colour_use = set ()
def pick_colour (request=None, hint=None):
global colours, colour_idx, colour_set
colour = None
if colour is None and request is not None and request not in colour_use and request in colours:
colour = request
if colour is None and hint is not None:
colour_try = zlib.crc32 (hint.encode ('utf-8')) % len (colours)
if colours [colour_try % 16] not in colour_use:
colour = colours [colour_try % 16]
elif colours [colour_try ] not in colour_use:
colour = colours [colour_try ]
if colour is None:
colour = colours [colour_idx % len (colours)]
colour_idx += 1
colour_use.add (colour)
return colour
......@@ -92,8 +106,9 @@ prox = [ [] ]
pidz = [ [] ]
zigz = [ [] ]
exok = [ [] ]
fork = [ '--' ]
fork = [ '^--$' ]
namz = [ None ]
colz = [ None ]
sigtypes = int (signal.Signals.SIGHUP)
......@@ -117,7 +132,7 @@ except:
soxtypes = {
'TCP:': (myfam, socket.SOCK_STREAM, 0),
'UDP:': (myfam, socket.SOCK_DGRAM, 0),
'SCTP:': (myfam, socket.SOCK_SEQPACKET, 0),
'SCTP:': (myfam, socket.SOCK_SEQPACKET, socket.IPPROTO_SCTP),
}
port_alloc = { }
......@@ -139,8 +154,9 @@ for argi in sys.argv [1:]:
pidz.insert ( 0,[] )
zigz.insert ( 0,[] )
exok.insert ( 0,[] )
fork.insert ( 0,'--' )
fork.insert ( 0,'^--$' )
namz.insert ( 0,None )
colz.insert ( 0,None )
elif proto == 'FILE:':
if tmpdir is None:
tmpdir = tempfile.mkdtemp ()
......@@ -211,6 +227,9 @@ for argi in sys.argv [1:]:
sys.stderr.write ('First use of PYPE:NAME: must be followed by a name\n')
sys.exit (1)
namz [0] = argo
elif argi [:9] == 'PYPE:COL:':
argo = argi [9:]
colz [0] = argo
elif argi [:9] == 'PYPE:ESC:':
argo = argi [9:]
else:
......@@ -241,6 +260,8 @@ for argi in sys.argv [1:]:
else:
prox [0].append (argi)
if argo is not None:
if argi not in argvcp_map or argvcp_map [argi] != argo:
sys.stdout.write ('pypeline: argvmap: %s --> %s\n' % (argi,argo))
argvcp_map [argi] = argo
if not precmd:
prox [0].append (argo)
......@@ -298,13 +319,13 @@ class ErrorTapper (threading.Thread):
class SyncedProcess (threading.Thread):
def __init__ (self, argv, name=None, signals=[], ok_exit=set ([0]), fork='--', env={}, pidfiles=[]):
def __init__ (self, argv, name=None, signals=[], ok_exit=set ([0]), fork='^--$', env={}, pidfiles=[], colour=None):
threading.Thread.__init__ (self, name=(name or argv [0]))
self.error = None
self.signals = signals
self.pidfiles = pidfiles
self.ok_exit = ok_exit
self.colour = pick_colour ()
self.colour = pick_colour (request=colour, hint=name)
self.proc = subprocess.Popen (
argv,
env=env,
......@@ -316,12 +337,10 @@ class SyncedProcess (threading.Thread):
self.sobad = ''
self.ertap = ErrorTapper (self.proc.stderr, self.errorline)
self.ertap.start ()
# '^' means fork now, '$' means "fork" at end, otherwise append '\n' and wait
# '^' means fork now, '$' means "fork" at end, otherwise wait for a regex
if fork != '^':
if fork == '$':
fork = ''
else:
fork = fork + '\n'
fork = '\A\Z'
self.copyloop (fork)
self.phase1locker = len (signals) == 0
if self.phase1locker:
......@@ -330,13 +349,14 @@ class SyncedProcess (threading.Thread):
phase1_mutex.release ()
def copyloop (self, terminator):
re_term = re.compile (terminator)
nextln = None
while nextln != terminator:
while nextln is None or not re_term.match (nextln):
nextln = self.proc.stdout.readline ()
nextln = binline_as_text (nextln)
if nextln != terminator:
if not re_term.match (nextln):
if nextln == '':
sys.stderr.write ('Program %s output ended before marker "%s"\n' % (self.name, terminator.strip (),))
sys.stderr.write ('Program %s output ended before pattern "%s"\n' % (self.name, terminator))
self.error = 1
return
attrs = attr ('dim') if 'DEBUG' in nextln else ''
......@@ -352,12 +372,13 @@ class SyncedProcess (threading.Thread):
self.sobad += nextln
def run (self):
self.copyloop ('')
self.copyloop ('\A\Z')
if self.phase1locker:
phase1_mutex.acquire ()
phase1_threads.remove (self)
phase1_mutex.notify ()
phase1_mutex.release ()
self.proc.wait ()
def could_signal (self):
return self.signals != []
......@@ -409,7 +430,7 @@ class SyncedProcess (threading.Thread):
strace = os.getenv ('STRACE')
valgrind = os.getenv ('VALGRIND')
syncprox = []
for (proc,pidl,zig,excitement,forkmark,name) in zip (prox,pidz,zigz,exok,fork,namz):
for (proc,pidl,zig,excitement,forkmark,name,colreq) in zip (prox,pidz,zigz,exok,fork,namz,colz):
if excitement == []:
excitement = [ 0 ]
if name is None:
......@@ -427,7 +448,7 @@ for (proc,pidl,zig,excitement,forkmark,name) in zip (prox,pidz,zigz,exok,fork,na
if valgrind is not None:
proc = ['valgrind', '--log-file=%s.valgrind' % name] + valgrind.split () + ['--'] + proc
sys.stdout.write ('pypeline: initial: %s == %r\n' % (name,proc) + attr (0))
newsyncer = SyncedProcess (proc, name=name, signals=zig, ok_exit=excitement, fork=forkmark, pidfiles=pidl, env=env)
newsyncer = SyncedProcess (proc, name=name, signals=zig, ok_exit=excitement, fork=forkmark, pidfiles=pidl, env=env, colour=colreq)
syncprox.insert (0, newsyncer)
sys.stdout.write ('pypeline: freerun: %s\n' % (name,) + attr (0))
newsyncer.start ()
......
#!/usr/bin/env python3
#
# Wrap a Diameter Daemon process -- for use with Pypeline
#
# This detects a point where the deamon is ready for action.
# It then sends the "--" separator that Pypeline is waiting for,
# so it can continue with the next process.
#
# The first argument is a signal number that will be listened
# to for stopping the server as soon as the client is done.
# It might be nice to also specify the TCP port to use, also
# because that provides a suitable IP from Pypeline, that also
# works on dreadful Docker.
#
# From: Rick van Rein <rick@openfortress.nl>
import sys
import signal
import subprocess
signum = int (sys.argv [1])
waitfor = sys.argv [2]
diacmd = sys.argv [3:]
# Create the subprocess
diadeem = subprocess.Popen (diacmd,
stdin=None,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
universal_newlines=True)
# Have a signal handler for the requested signal
def sighdl (sig,_st):
diadeem.send_signal (sig)
signal.signal (signum, signal.SIG_IGN)
signal.signal (signum, sighdl)
signal.signal (signal.SIGTERM, sighdl)
sys.stderr.write ('Waiting for "%s"\n' % waitfor)
# Clone header lines until the first "waitfor" line
ln = diadeem.stdout.readline ()
while ln != '':
sys.stdout.write (ln);
if waitfor is not None and waitfor in ln:
sys.stdout.write ('--\n')
sys.stdout.flush ()
waitfor = None
ln = diadeem.stdout.readline ()
# Check if we actually found the "waitfor" line
if waitfor is not None:
sys.stderr.write ('Oops, terminated while freeDiameterd is starting\n')
sys.exit (1)
# Done, return happily
retval = diadeem.wait ()
sys.exit (retval)
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment