introduction.rst 5.35 KB
Newer Older
slobbeat's avatar
slobbeat committed
1 2 3 4 5 6 7
.. _sec:introduction:

*************************
Introduction
*************************

=========================
Árpád Rózsás's avatar
Árpád Rózsás committed
8
What is this?
slobbeat's avatar
slobbeat committed
9 10
=========================

11
Black-box Reliability Challenge 2019 is a challenge and framework to test the performance of structural reliability methods. It is the first challenge in structural reliability where the reliability problems are truly black boxes for participants, meaning that their performance functions are hidden from the participants by making them accessible via the internet in the form of submitted input arguments and received output. To our knowledge, it is also the first on-line challenge in structural reliability. It is inspired by black-box optimization competitions such as |BBComp|. After the challenge, all the details of the reliability problems will be made publicly available, moving them to the repository for the structural reliability benchmarking framework.
slobbeat's avatar
slobbeat committed
12

13
We intend the current challenge to be the first iteration which we wish to improve based on the gained experience and your feedback. We are indebted to those who have already contributed to the design of the challenge and collecting reliability problems. We hope that the challenge and repository will evolve into an open, community project!
Árpád Rózsás's avatar
Árpád Rózsás committed
14

15 16 17 18
.. |BBComp| raw:: html

	<a href="https://bbcomp.ini.rub.de/" target="_blank">BBComp</a>

slobbeat's avatar
slobbeat committed
19 20 21 22
=========================
Motivation
=========================

23
Over the last decades, a plethora of reliability analysis methods has been developed with various capabilities, accuracy, and efficiency. Yet, an objective comparison and evaluation of these methods is missing. The currently available comparative studies are limited in scope and do not allow impartial analysis as only the final results are disclosed and they allow unrealistic exploration of the reliability problem by the analyst.
slobbeat's avatar
slobbeat committed
24

25 26
Here we propose a testing framework which is designed to:

Árpád Rózsás's avatar
Árpád Rózsás committed
27 28
#. deliver the long-needed, systematic and structured comparison of the performance of reliability algorithms among simplified, academic problems and real-life engineering problems;
#. provide invaluable guidance for practical applications of reliability analysis in the domains of geotechnical engineering, hydraulic engineering, structural engineering, and earthquake engineering; and
29
#. aid setting future research directions.
slobbeat's avatar
slobbeat committed
30

Árpád Rózsás's avatar
Árpád Rózsás committed
31 32
.. _sec:timeline:

slobbeat's avatar
slobbeat committed
33 34 35 36
=========================
Timeline
=========================

Árpád Rózsás's avatar
Árpád Rózsás committed
37 38 39 40 41 42 43 44
.. altair-plot::
    :hide-code:

    import altair as alt
    import pandas as pd

    # Input - numbers are included because the ordering does not work with layers in vega-lite.
    # https://github.com/altair-viz/altair/issues/820
45 46 47
    event_start = ['2018-01-24', '2018-12-21', '2019-03-28', '2019-12-20', '2020-02-24']
    event_end = ['2018-01-24', '2018-12-21', '2019-12-04', '2019-12-20', '2020-02-25']
    event_name = ['1. Workshop 2018', '2. Announcement & test problems', '3. Challenge', '4. Announcement of results', '5. Workshop 2020']
Árpád Rózsás's avatar
Árpád Rózsás committed
48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68

    df = pd.DataFrame(
            {"Start" : event_start,
             "End" :   event_end,
             "Event" : event_name,
            })

    df = pd.melt(df, id_vars=("Event"), value_name="Date")
    df["Date"] = pd.to_datetime(df["Date"])
    df = df.sort_values("Event")

    base = alt.Chart(df).encode(
        x=alt.X('Date', type='temporal',
                scale=alt.Scale(padding=30),
                axis=alt.Axis(title=None)),
        y=alt.Y(field='Event', type='nominal',
                sort=list(reversed(event_name)),
                axis=alt.Axis(title=None)),
        color=alt.Color('Event', type='nominal',
                       legend=None)
    )
slobbeat's avatar
slobbeat committed
69

Árpád Rózsás's avatar
Árpád Rózsás committed
70 71 72 73 74
    points = base.mark_circle(size=50).encode(
        opacity=alt.value(1),
        tooltip=[alt.Tooltip('Date:T', format='%Y, %B %e'),
                 'Event']
    ).interactive()
slobbeat's avatar
slobbeat committed
75

Árpád Rózsás's avatar
Árpád Rózsás committed
76
    lines = base.mark_line().encode()
77

Árpád Rózsás's avatar
Árpád Rózsás committed
78 79 80 81 82 83 84
    alt.layer(points, lines).configure_legend(labelLimit=0).properties(width=500)


.. csv-table::
   :header: "Phase", "Start date", "End date"
   :widths: 20, 20, 20

85 86 87 88
    Workshop [Rozsas2018]_,                         2018-Jan-24, NA
    Announcement & test problems,                   2018-Dec-21, NA
    Challenge,                                      2019-March-28, 2019-Dec-04
    Announcement of results (:ref:`sec:results`),   2019-Dec-20, NA
89 90 91 92 93
    |Follow-up workshop|,                           2020-Feb-24, 2020-Feb-25

.. |Follow-up workshop| raw:: html

	<a href="https://reliabilityworkshop2020.com/" target="_blank">Follow-up workshop</a>
94

95 96 97 98
=========================
How to participate
=========================
 
Árpád Rózsás's avatar
Árpád Rózsás committed
99
Everyone is welcome to join the challenge (registration will open in early 2019). The participants use their computer and their selected method to solve reliability problems with performance functions located on the organizers servers. See the :ref:`sec:quick_start` for a walk through how to do this.
100

slobbeat's avatar
slobbeat committed
101 102 103 104
=========================
Who we are
=========================

Árpád Rózsás's avatar
Árpád Rózsás committed
105
The idea of the Black-box Reliability Challenge, the benchmarking framework, and the reliability problem types are the outcome of a joint discussion and follow-up sessions between the participants of a workshop held in 2018 in Delft [Rozsas2018]_.
106

Árpád Rózsás's avatar
Árpád Rózsás committed
107
The organization and implementation of the challenge are spearheaded by the Department of Structural Reliability, TNO. The members of the team have selected the reliability problems which are known exclusively to them until the announcement of the results.