enable RDMA lab in CKI

This is a first draft of pipeline changes that would enable adding RDMA lab to CKI. This is not meant to be merged until WiP status resolved.

This enables RDMA patch testing only. Brew is not yet part of this, but can easily be made so. If this is to be changed, then it will be said explicitly in the commit message.

This patch has several pieces that do the necessary work:

1. kpet is run twice

• 1st time rdma test set is forcibly excluded, so we generate a beaker.xml file template without any RDMA tests

• 2nd time rdma test set if forcibly used, so we either generate a beaker_rdma.xml file template with RDMA tests only, or we don't generate any file at all

2. A special keytab is used for 2nd (RDMA) job

Keytab is okay and added to CI vars. File created as BEAKER_CONF_RDMA=${HOME}/.beaker_client/rdma_config should be OK, conf. was tested locally for now.

3. skt may be run twice

• If beaker_rdma.xml file was generated, then we use a slightly different logic and run skt twice insteadof once.
• A copy of RC_FILE is used, with job template set to beaker_rdma.xml.
• If SKT is run twice, then BEAKER_CLIENT_CONF is set to either BEAKER_CONF_RDMA or BEAKER_CONF_STD, see 2).
• Both skt instances have the same timeouts.
• Shell waits for both return codes and captures them.
• The rc state files are merged to RC_FILE; jobs from RC_FILE_SEC are appended, recipesetid_ and jobid_ from RC_FILE_SEC are reindex and added as new keys

4. The final return code is determined like so:

• Timeout in any job takes priority over infrastructure issue and anything else.
• Infrastructure issue in any job takes priority over other return codes (those being SKT_SUCCESS = 0, SKT_FAIL = 1, SKT_BOOT = 3)

V2:

• fixed KRB_PRINCIPAL V3:
• fixed a comment V4:
• remove adding dir and cert twice V5:
• use RDMA_KEYTAB_CONTENT that is already set
• set group rdma-users for Beaker job

Signed-off-by: Jakub Racek jracek@redhat.com