Rare failure in test-unit job
This ticket analysis the rare failure of this job. The test is locally restarted until failure on commit 20d9b5a5. A wireshark trace helps identifying the problem: the server answers an OpcUa_BadEncodingError
instead of the OpenSecureChannelResponse
.
The backtrace starts at csrc/secure_channels/sopc_chunks_mgr.c:3401
, which shows that the signature failed (SC_Chunks_EncodeSignature
). SOPC_CryptoProvider_AsymmetricSign
fails because CryptoProvider_AsymSign_RSASSA_PKCS1_v15_w_SHA256
fails because mbedtls_rsa_rsassa_pkcs1_v15_sign
fails with MBEDTLS_ERR_RSA_PRIVATE_FAILED
(its value is -0x4300 or 0xFFFFBD00).
The source for this function which fails is:
MBEDTLS_MPI_CHK( mbedtls_rsa_private( ctx, f_rng, p_rng, sig, sig_try ) );
MBEDTLS_MPI_CHK( mbedtls_rsa_public( ctx, sig_try, verif ) );
if( mbedtls_safer_memcmp( verif, sig, ctx->len ) != 0 )
{
ret = MBEDTLS_ERR_RSA_PRIVATE_FAILED;
goto cleanup;
}
The debugging show that the memcmp
fails, which says that the encrypt-decrypt of the sig vector fails.
(rr) # Content of sig
(rr) xxd 0x7faac4000cb0 256
0000000: 0001 ffff ffff ffff ffff ffff ffff ffff ................
0000010: ffff ffff ffff ffff ffff ffff ffff ffff ................
0000020: ffff ffff ffff ffff ffff ffff ffff ffff ................
0000030: ffff ffff ffff ffff ffff ffff ffff ffff ................
0000040: ffff ffff ffff ffff ffff ffff ffff ffff ................
0000050: ffff ffff ffff ffff ffff ffff ffff ffff ................
0000060: ffff ffff ffff ffff ffff ffff ffff ffff ................
0000070: ffff ffff ffff ffff ffff ffff ffff ffff ................
0000080: ffff ffff ffff ffff ffff ffff ffff ffff ................
0000090: ffff ffff ffff ffff ffff ffff ffff ffff ................
00000a0: ffff ffff ffff ffff ffff ffff ffff ffff ................
00000b0: ffff ffff ffff ffff ffff ffff ffff ffff ................
00000c0: ffff ffff ffff ffff ffff ffff 0030 3130 .............010
00000d0: 0d06 0960 8648 0165 0304 0201 0500 0420 ...`.H.e.......
00000e0: 1ebc c990 168a 2828 b7ff 300c 902c 1ea5 ......((..0..,..
00000f0: a92a 0a5b 813b 5a29 68c1 1186 7840 a569 .*.[.;Z)h...x@.i
(rr) # Content of sig_try
(rr) xxd 0x7faac4000dc0 256
0000000: 2200 f81e 5775 20a7 11c9 bf7f 14c8 2bea "...Wu .......+.
0000010: 13a8 cfe3 8e1d 1ff3 0d00 0296 ddc1 04b3 ................
0000020: 344f 2941 d531 4e29 2ecd daa5 2a19 e9ef 4O)A.1N)....*...
0000030: f284 a681 0f2d 7df9 2697 64ff 6e71 5b2b .....-}.&.d.nq[+
0000040: 88eb db16 223f 5c25 7630 8af8 6597 3c3d ...."?\%v0..e.<=
0000050: 2880 31f2 a2eb da1c 06ab bfd0 b811 3550 (.1...........5P
0000060: 04e9 49ad ec55 35a6 d84b 1256 e1d4 ce8a ..I..U5..K.V....
0000070: 5523 5692 af61 f0ce 8007 d7ef 6044 e6a3 U#V..a......`D..
0000080: 11af 4957 c65a c987 7205 7942 707d af61 ..IW.Z..r.yBp}.a
0000090: a20d 57ac cbc5 1dc4 eef9 9c7b 5838 f6d5 ..W........{X8..
00000a0: 1799 1eef 83a7 4fd9 0d31 dd78 f61a 7ea7 ......O..1.x..~.
00000b0: 84d3 39cd 1b65 7a4f a146 4be7 d386 0ca7 ..9..ezO.FK.....
00000c0: 9e32 0779 3091 fca9 89bf efbe d602 2614 .2.y0.........&.
00000d0: 0ab5 3165 efd9 b3b9 4f07 51be a370 69b9 ..1e....O.Q..pi.
00000e0: 9d7b 0eba c037 229a 287d c606 9a8b 73a9 .{...7".(}....s.
00000f0: 0d4e 41c6 bab6 fdf4 2fbb 7e71 3eae a9e3 .NA...../.~q>...
(rr) # Content of verif
(rr) xxd 0x7faac4000ed0 256
0000000: 7453 9aa6 62c7 fb50 21ea 7aac dade c0cd tS..b..P!.z.....
0000010: 43f7 31e9 c400 14da facc 3854 0195 ab02 C.1.......8T....
0000020: c230 152b f582 e103 4bf6 31cf d60b 45a7 .0.+....K.1...E.
0000030: 2b3f 6dd8 d032 8d35 f4d3 d0f4 bde3 cbb5 +?m..2.5........
0000040: 80db ea30 712a e810 e432 6cb9 4cd9 5947 ...0q*...2l.L.YG
0000050: 32cb 87f8 2264 d120 4a9d 8c4a f7a7 e23e 2..."d. J..J...>
0000060: 8493 1e4a 75f0 f971 460a 6db4 dcab cd03 ...Ju..qF.m.....
0000070: 5c3a ec5a 2eb4 25f2 2f1a 5647 f1fd 6b58 \:.Z..%./.VG..kX
0000080: bf61 81d9 9b9f e51f 66bc 539b 201a d43b .a......f.S. ..;
0000090: c490 3c4c e7d6 d7b4 af1e 9189 8a77 57da ..<L.........wW.
00000a0: 1f58 67c0 79b2 e295 de6f cdbe e185 0acf .Xg.y....o......
00000b0: 8f76 5e0b e7eb 4210 3090 37b4 9a77 755f .v^...B.0.7..wu_
00000c0: e89b db73 80a9 4223 c3c6 1749 b487 2590 ...s..B#...I..%.
00000d0: 872f 2998 9100 de06 e992 bb33 438e 273a ./)........3C.':
00000e0: f8e4 e536 74e6 69e1 3f64 51be 4d2a ace0 ...6t.i.?dQ.M*..
00000f0: 8bd2 4f18 4a5f acfa 7a46 15e0 e358 9146 ..O.J_..zF...X.F
This is the private operation that fails, as we can see:
$ diff <(cat verif.xxd) <(openssl rsautl -in sig_try -inkey bin/server_public/server_2k.der -keyform DER -certin -raw | xxd) && echo Success
Success
The is due to internal buffers that becomes invalid. It is possible to replay the same call to sign on the same hash, with the same in a different program. Without modifications of the internal buffers, the signature does not fail. When copying values of the internal buffers to the new program when paused with gdb, it is possible to reproduce the same output. The said buffers are:
/**
* \brief The RSA context structure.
* [...]
*/
typedef struct
{
/* [...] */
mbedtls_mpi Vi; /*!< The cached blinding value. */
mbedtls_mpi Vf; /*!< The cached un-blinding value. */
/* [...] */
}
mbedtls_rsa_context;
According to gdb/rr, it looks like that Vi
is modified in two threads at a time:
(rr) t a a bt
Thread 3 (Thread 2255.2258):
[...]
#3 0x00000000004b15a3 in mbedtls_rsa_private (ctx=0x1e9f4e0, f_rng=0x490afe <no_random_function>, p_rng=0x7f2ea4001bf8, input=0x7f2ea4000cb0 "", output=0x7f2ea4000dc0 "")
at /work/mbedtls-2.7.0/library/rsa.c:881
#4 0x00000000004b2e1c in mbedtls_rsa_rsassa_pkcs1_v15_sign (ctx=0x1e9f4e0, f_rng=0x490afe <no_random_function>, p_rng=0x7f2ea4001bf8, mode=1, md_alg=MBEDTLS_MD_SHA256, hashlen=32,
hash=0x7f2ea40034f0 "U\027\264\357\261\061\333-\202z\313\373\024f\177\271p.\006\312\370\251\313\346/\030\255\020\t\211g\325\060", sig=0x7f2ea4000cb0 "") at /work/mbedtls-2.7.0/library/rsa.c:1732
#5 0x0000000000490bfb in CryptoProvider_AsymSign_RSASSA_PKCS1_v15_w_SHA256 (pProvider=0x7f2ea4012c30, pInput=0x7f2ea4023580 "OPNF\222\006", lenInput=1342, pKey=0x1e9f4c0, pSignature=0x7f2ea4000cb0 "")
at csrc/crypto/mbedtls/crypto_functions_lib.c:508
#6 0x000000000043b57c in SOPC_CryptoProvider_AsymmetricSign (pProvider=0x7f2ea4012c30, pInput=0x7f2ea4023580 "OPNF\222\006", lenInput=1342, pKeyPrivateLocal=0x1e9f4c0, pSignature=0x7f2ea4000cb0 "",
lenSignature=256) at csrc/crypto/sopc_crypto_provider.c:1061
[...]
#9 0x0000000000481e8a in SOPC_ChunksMgr_Dispatcher (event=INT_SC_SND_OPN, eltId=4, params=0x7f2ea4001e70, auxParam=1)
at csrc/secure_channels/sopc_chunks_mgr.c:3641
#10 0x00000000004698a6 in SOPC_SecureChannelsEventMgr_Dispatcher (event=29, eltId=4, params=0x7f2ea4001e70, auxParam=1)
at csrc/secure_channels/sopc_secure_channels_api.c:108
[...]
Thread 2 (Thread 2255.2259):
#0 mbedtls_rsa_rsassa_pkcs1_v15_sign (ctx=0x1e9f4e0, f_rng=0x490afe <no_random_function>, p_rng=0x7f2eac001528, mode=1, md_alg=MBEDTLS_MD_SHA256, hashlen=32,
hash=0x7f2eac0017d0 "Iz\342\305\302\351v\261W\003_\272\354Q\301\rpzk\222", sig=0x7f2eac0016c0 "") at /work/mbedtls-2.7.0/library/rsa.c:1737
#1 0x0000000000490bfb in CryptoProvider_AsymSign_RSASSA_PKCS1_v15_w_SHA256 (pProvider=0x7f2eac001100,
pInput=0x7f2eac001e60 "0\202\004)0\202\003\021\240\003\002\001\002\002\001\005\060\r\006\t*\206H\206\367\r\001\001\v\005", lenInput=1101, pKey=0x1e9f4c0, pSignature=0x7f2eac0016c0 "")
at csrc/crypto/mbedtls/crypto_functions_lib.c:508
#2 0x000000000043b57c in SOPC_CryptoProvider_AsymmetricSign (pProvider=0x7f2eac001100,
pInput=0x7f2eac001e60 "0\202\004)0\202\003\021\240\003\002\001\002\002\001\005\060\r\006\t*\206H\206\367\r\001\001\v\005", lenInput=1101, pKeyPrivateLocal=0x1e9f4c0, pSignature=0x7f2eac0016c0 "",
lenSignature=256) at csrc/crypto/sopc_crypto_provider.c:1061
[...]
#8 0x00000000004729e8 in SOPC_ServicesEventDispatcher (scEvent=5, id=3, params=0x7f2ea40133c0, auxParam=2) at csrc/services/sopc_services_api.c:199
[...]
Both threads call SOPC_CryptoProvider_AsymmetricSign
with both the same pKeyPrivateLocal
. In fact, mbedtls_rsa_rsassa_pkcs1_v15_sign
modifies the key, hence the signature failure in one or both threads.
A first fix is to copy the pKeyPrivateLocal in CryptoProvider_AsymSign_RSASSA_PKCS1_v15_w_SHA256
before calling any underlying mbedtls_
function.
A more judicious fix would be to copy keys within modules once, to keep the thread safety without copying the keys for each signature or verification.