WIP: Add CI tarball build

Finally found a stable yaml rule to use artifacts between stages (cache: doesn't work stable).

So here comes a build from tarball without the need of a fully configured development system. The CI image, an Alpine 3.8 minimal system, is already in build-images (MR is created when this code is finished).

The idea is to test if a build from tarball works for outside a development environment.

changed the description

Nice. A second stage looks like a good idea in general as we can bring also the build of applications using gnutls with a good test suite. I had tried that in the past with !477 (closed) but problems in gitlab prevented it from working. Maybe now we are close to that.

added 6 commits

• 20e39656...9e58b6b6 - 5 commits from branch gnutls:master
• 1566f15c - Add CI tarball build

Compare with previous version

This won't currently build because autogen is missing. Wasn't the latest std-arg related commit(s) tested without installed autogen ? @dueno WDYT ?

I also see in src/:

Package p11-kit-1 was not found in the pkg-config search path.
Perhaps you should add the directory containing `p11-kit-1.pc'
to the PKG_CONFIG_PATH environment variable
Package 'p11-kit-1', required by 'virtual:world', not found
cat: can't open '/p11-kit/pkcs11.h': No such file or directory

Do we need a if ENABLE_PKCS11 around p11tool ?

Wasn't the latest std-arg related commit(s) tested without installed autogen ?

Yes, it is. I can successfully compile the tarball from doc-dist.Fedora on alpine-3.5.2 build-ready VM after installing necessary packages (nettle-dev, libtasn1-dev, libunistring-dev) from the "edge" repository, though I see weird warnings as below:

./configure --disable-doc --disable-guile --without-p11-kit
...
  GEN      systemkey-args.stamp
/home/miles/gnutls-3.6.4/build-aux/missing: line 81: autogen: not found
WARNING: 'autogen' is missing on your system.
...
  CC       systemkey.o
  CC       systemkey-args.lo
systemkey-args.c:42: warning: macro "OPTION_CODE_COMPILE" is not used [-Wunused-macros]
 #define OPTION_CODE_COMPILE 1
 
  CCLD     libcmd-systemkey.la
copying selected object files to avoid basename conflicts...
ar: `u' modifier ignored since `D' is the default (see `U')
  CCLD     systemkey
...

These should be fixed once !808 (merged) is merged

added 16 commits

• 1566f15c...0f3dca28 - 12 commits from branch gnutls:master
• 22088a86 - Add CI tarball build
• ad9f8a90 - tests/scripts/common.sh: Make random port value work on busybox
• eb1398ab - tests/gnutls-cli-debug.sh: Remove bashisms
• 51cb1fb5 - tests/long-crl.sh: Increase portability (fix for busybox)

Compare with previous version

@dueno True, tarball compiles on the Alpine image despite those warnings :-)

There are still tests failing due to non portable assumptions (Alpine is using busybox + musl). I am slowly going through them...

added 7 commits

• 3c800b87 - cert-tests/certtool-crl-decoding: Increase portability (fix for busybox)
• 7da7946a - tests/cert-tests/pem-decoding: Increase portability (fix for busybox)
• 93e91b0b - tests/cert-tests/certtool-ecdsa: Increase portability (fix for busybox)
• b58dfabe - tests/cert-tests/pkcs12-corner-cases: Increase portability (fix for busybox)
• 651b47e6 - tests/cert-tests/pkcs12-utf8: Use /bin/sh instead of bash
• c7b1283a - tests/cert-tests/certtool: Remove bashism
• ef050242 - tests/cert-tests/certtool-eddsa: Increase portability (fix for busybox)

Compare with previous version

added 10 commits

• 02848b1f - tests/scripts/common.sh: Make random port value work on busybox
• 2d392c80 - tests/gnutls-cli-debug.sh: Remove bashisms
• 25bbe282 - tests/long-crl.sh: Increase portability (fix for busybox)
• 344eb678 - cert-tests/certtool-crl-decoding: Increase portability (fix for busybox)
• 320f5f8b - tests/cert-tests/pem-decoding: Increase portability (fix for busybox)
• ce48ab40 - tests/cert-tests/certtool-ecdsa: Increase portability (fix for busybox)
• c1110e75 - tests/cert-tests/pkcs12-corner-cases: Increase portability (fix for busybox)
• ce889c46 - tests/cert-tests/pkcs12-utf8: Use /bin/sh instead of bash
• 6058cb35 - tests/cert-tests/certtool: Remove bashism
• 652118ba - tests/cert-tests/certtool-eddsa: Increase portability (fix for busybox)

Compare with previous version

added 17 commits

• 652118ba...03bed2a0 - 6 commits from branch gnutls:master
• 6da511ca - Add CI tarball build
• 66dcd73b - tests/scripts/common.sh: Make random port value work on busybox
• 37a2335c - tests/gnutls-cli-debug.sh: Remove bashisms
• 0b8c56ad - tests/long-crl.sh: Increase portability (fix for busybox)
• 6ae8da50 - cert-tests/certtool-crl-decoding: Increase portability (fix for busybox)
• ef8e71ef - tests/cert-tests/pem-decoding: Increase portability (fix for busybox)
• 130b0c06 - tests/cert-tests/certtool-ecdsa: Increase portability (fix for busybox)
• 9ee91e2c - tests/cert-tests/pkcs12-corner-cases: Increase portability (fix for busybox)
• 34d1be8d - tests/cert-tests/pkcs12-utf8: Use /bin/sh instead of bash
• 0b5a86b2 - tests/cert-tests/certtool: Remove bashism
• ac48321d - tests/cert-tests/certtool-eddsa: Increase portability (fix for busybox)

Compare with previous version

added 8 commits

• 746051a8 - tests/long-crl.sh: Increase portability (fix for busybox)
• f5325ea9 - cert-tests/certtool-crl-decoding: Increase portability (fix for busybox)
• 1fb7b186 - tests/cert-tests/pem-decoding: Increase portability (fix for busybox)
• 9d719ce9 - tests/cert-tests/certtool-ecdsa: Increase portability (fix for busybox)
• 03e1fca9 - tests/cert-tests/pkcs12-corner-cases: Increase portability (fix for busybox)
• 5bd999d4 - tests/cert-tests/pkcs12-utf8: Use /bin/sh instead of bash
• b2c429b3 - tests/cert-tests/certtool: Remove bashism
• b0ca4c87 - tests/cert-tests/certtool-eddsa: Increase portability (fix for busybox)

Compare with previous version

added 6 commits

• 76cc96a3 - tests/cert-tests/pem-decoding: Increase portability (fix for busybox)
• daa87982 - tests/cert-tests/certtool-ecdsa: Increase portability (fix for busybox)
• 04f50f0e - tests/cert-tests/pkcs12-corner-cases: Increase portability (fix for busybox)
• 2ae34de1 - tests/cert-tests/pkcs12-utf8: Use /bin/sh instead of bash
• 71c89924 - tests/cert-tests/certtool: Remove bashism
• 864bd50f - tests/cert-tests/certtool-eddsa: Increase portability (fix for busybox)

Compare with previous version

added 2 commits

• 6e7d1865 - tests/cert-tests/certtool: Remove bashism
• 030bfddb - tests/cert-tests/certtool-eddsa: Increase portability (fix for busybox)

Compare with previous version

@nmav In the Alpine image, tests/slow/test-ciphers-api.sh fails with

trying aes128-gcm
trying aes256-gcm
trying aes128-cbc
trying aes256-cbc
trying 3des-cbc
Assertion failed: !(length % block_size) (cbc.c: nettle_cbc_encrypt: 53)
_check_wait_status:153: Child died with signal 11
default cipher tests failed
FAIL test-ciphers-api.sh (exit status: 1)

Any idea ?

added 2 commits

• 44e22aee - tests/cert-tests/certtool: Remove bashism
• 7bec5bb1 - tests/cert-tests/certtool-eddsa: Increase portability (fix for busybox)

Compare with previous version

Why does GNUTLS_PIN=${PASS} not work (for certtool) on the cross runners, but on all other runners ?

My issues are that

• --ask-pass is not portable (won't work on busybox, see my comments in tests/cert-tests/certtool)
• --password doesn't work on all runners
• GNUTLS_PIN doesn't work on the cross runners

So I wonder how to proceed here.

added 11 commits

• f0ce626e - Add CI tarball build
• 9e5bb4a9 - tests/scripts/common.sh: Make random port value work on busybox
• f9ce0d79 - tests/gnutls-cli-debug.sh: Remove bashisms
• b97a0e80 - tests/long-crl.sh: Increase portability (fix for busybox)
• c0b7b20c - cert-tests/certtool-crl-decoding: Increase portability (fix for busybox)
• d882af7b - tests/cert-tests/pem-decoding: Increase portability (fix for busybox)
• 74f77480 - tests/cert-tests/certtool-ecdsa: Increase portability (fix for busybox)
• c992f6f2 - tests/cert-tests/pkcs12-corner-cases: Increase portability (fix for busybox)
• e54d70c7 - tests/cert-tests/pkcs12-utf8: Use /bin/sh instead of bash
• 5039c9fc - tests/cert-tests/certtool: Remove bashism
• 6392e4f6 - tests/cert-tests/certtool-eddsa: Increase portability (fix for busybox)

Compare with previous version

added 11 commits

• 5b40ef49 - Add CI tarball build
• 04f9f671 - tests/scripts/common.sh: Make random port value work on busybox
• fbfa5698 - tests/gnutls-cli-debug.sh: Remove bashisms
• c59a89fe - tests/long-crl.sh: Increase portability (fix for busybox)
• 477503f8 - cert-tests/certtool-crl-decoding: Increase portability (fix for busybox)
• d29ff19b - tests/cert-tests/pem-decoding: Increase portability (fix for busybox)
• 21345839 - tests/cert-tests/certtool-ecdsa: Increase portability (fix for busybox)
• e101cab3 - tests/cert-tests/pkcs12-corner-cases: Increase portability (fix for busybox)
• c0bb3097 - tests/cert-tests/pkcs12-utf8: Use /bin/sh instead of bash
• 23af234a - tests/cert-tests/certtool: Remove bashism
• 5753e3fe - tests/cert-tests/certtool-eddsa: Increase portability (fix for busybox)

Compare with previous version

I haven't seen the comments, but I guess making the test suite work with busybox will be quite a challenge. Not sure how important is that to go for it. Is there a particular reason to run our testsuite on alpine? Why not use debian or centos?

Because Alpine is different with it's busybox + musl backend. It is a very clean/lean distribution widely used as base for docker / VM images. The fixed incompatibilities in the test suite may be blockers on other OSes as well, so fixing extends the possible GnuTLS distributions. And the above mentioned assertion may easily be a bug in libgnutls (didn't look at it closer , though).

Also we are pretty close, all is fixed except the error in tests/slow/test-ciphers-api.sh.

A question regarding \r (Carriage Return) handling: In some tests with diff, the CR is ignored (I guess it's for Windows compatibility) and in others not. Does anyone sometimes build+test GnuTLS on Windows ?