A common requirement in science is to store and share large sets of simulation data in an efficient, nested, flexible and human-readable way. Modern high-level programming languages like Python and Perl have associated arrays, knowns as dictionaries or hashes, respectively, to fulfil this need. Low-level languages used more commonly for fast computational simulations such as C and FORTRAN lack this functionality. We present a `libcdict`, a C dictionary library, to mostly solve this problem for C users. `Libcdict` provides C and FORTRAN application programming interfaces (APIs) to dictionaries, called `cdict` structures (`cdict`s), and functions for `cdict` to JSON, and vice versa, conversion for easy interpretation in other software.
A common requirement in science is to store and share large sets of simulation data in an efficient, nested, flexible and human-readable way. Such data contain number counts and distributions, i.e.~histograms and maps, of arbitrary dimension and variable type, e.g.~floating-point number, integer or character string. Modern high-level programming languages like Python and Perl have associated arrays, knowns as dictionaries or hashes, respectively, to fulfil this storage need. Low-level languages used more commonly for fast computational simulations, such as C and FORTRAN, lack this functionality. We present a `libcdict`, a C dictionary library, to mostly solve this problem for C users. `Libcdict` provides C and FORTRAN application programming interfaces (APIs) to dictionaries, called `cdict` structures (`cdict`s), and functions for `cdict` to JSON, and vice versa, conversion for easy interpretation in other software.
# Statement of need
Users of high-level languages such as Python or Perl have access to associated-array data structures through dictionaries and hashes, respectively. These allow arbitrary data types to be stored in array-like structures. These are in turn accessed through key-value pairs which allow the value to be a further, nested associated array, allowing arbitrary nesting of data. Compiled languages more suited to high-speed and repeated calculations typical in science, like C and FORTRAN, lack native associated-array functionality. `libcdict` provides an API for such functionality which allows `cdict`s to be nested in `cdict`s, hence arbitrarily-nested variables in C just as in Python or Perl.
Users of high-level languages such as Python or Perl have access to associated-array data structures through dictionaries and hashes, respectively. These allow arbitrary data types to be stored in array-like structures. These are in turn accessed through key-value pairs which allow the value to be a further, nested associated array, allowing arbitrary nesting of data. Compiled low-level languages, like C and FORTRAN, are more suited to high-speed and repeated calculations typical in science. These languages lack native associated-array functionality. `libcdict` provides an API for such functionality which allows `cdict`s to be nested in `cdict`s, hence arbitrarily-nested variables in C just as in Python or Perl.
`libcdict` is written in C and provides an API through a set of C macros. `libcdict` has been used for the last year in the `binary_c` single- and binary-star population nucleosynthesis framework [@izzard:2004; @izzard:2006; @izzard:2009; @izzard:2018] which computes the evolution of millions of single- and binary-stellar systems in only a few hours using its `binary_c-python` Python frontend. We provide `libcdict` as open-source code on gitlab subject to the GPL3. `libcdict` also has a comprehensive test suite run through its configuration program `cdict-config`.
# Using libcdict
`libcdict` is flexible but pragmatic. Keys to `cdict`s can be any C scalar or pointer. Values can be scalars, pointers, arrays or other `cdict`s, but arrays must be of a single C type. Values can also store metadata of arbitrary type(s). Pointer values are optionally garbage collected when a `cdict` is freed. A set of API macros provides simple nesting facilities so that placing a value in a nested location given a list of keys is a simple task for the C programmer. Issues such as C variable typing are automatically handled for the user.
`libcdict` is flexible but pragmatic. Keys to `cdict`s can be any C scalar or pointer. Values can be scalars, pointers, arrays or other `cdict`s, but arrays must be of a single C type. Values can store metadata of arbitrary type. Pointer values are optionally garbage collected when a `cdict` is freed. A set of API macros provides simple nesting facilities so that placing a value in a nested location given a list of keys is a simple task for the C programmer. Issues such as C variable typing are automatically handled for the user.
1
Variables are internally hashed using `uthash` [@uthash]. `libcdict` provides a custom JSON output function and inputs JSON using `jsmn` [@jsmn]. Floating-point input and output uses `fast double parser` [@fastdoubleparser; @lemire:2021] and `ryu` [@ryu; @adams:2018], respectively, both of which are considerably faster than equivalent native-C functions. `libdict` allows customizable floating-point accuracy for output and when comparing floating-point numbers, e.g.~during sorting of key or variable lists.
Variables are internally hashed using `uthash` [@uthash]. `libcdict` provides a custom JSON output function and inputs JSON using `jsmn` [@jsmn]. Floating-point input and output uses `fast double parser` [@fastdoubleparser; @lemire:2021] and `ryu` [@ryu; @adams:2018], both of which are considerably faster than equivalent native-C functions. `libdict` allows customizable floating-point accuracy both for output and when comparing floating-point numbers. Installation uses `meson` [@meson] and `ninja` [@ninja]. `libcdict` has been tested with the GCC (10.3.0) and Clang (12.0.0) compilers.
Installation uses `meson` [@meson] and `ninja` [@ninja]. `libcdict` has been tested with the GCC (10.3.0) and Clang (12.0.0) compilers.
# `libcdict` in stellar-population statistics calculations
`libcdict` was developed to solve the problem of storing statistics in stellar-population calculations in `binary_c`. When evolving a population of millions, sometimes billions, of stars, each for thousands of timesteps, an enormous amount of data is computed. It is impractical to output this data every timestep as this is at least $\sim 10^{6} \times 10^{4} = 10^{10}$ lines of data, each of which can easily be $\sim 1\,\mathrm{KB}$ long. The data from each star could be sent to a Python or Perl front-end which merges them into a dictionary or hash of statistics. This communication between programming languages involves significant overhead which compares similarly to the runtime of the stellar code itself thus greatly increases runtime and cost.
`libcdict` was developed to solve the problem of storing statistics in stellar-population calculations in `binary_c`. When evolving a population of millions, sometimes billions, of stars, each for thousands of timesteps, enormous amounts of data are computed. It is impractical to output these data every timestep as these are typically $\sim 10^{6} \times 10^{4} = 10^{10}$ lines, each of which can easily be $\sim 1\,\mathrm{KB}$ long. The data from each star could be sent to a Python or Perl front-end which merges them into a dictionary or hash of statistics. This communication between programming languages involves significant overhead which compares similarly to the runtime of the stellar code itself thus greatly increases runtime and cost.
To overcome this problem, `binary_c` generates the associative-array `cdict` in native C internally. This `cdict`, and the statistics it contains, is filled in the `binary_c` simulation. Generation of the stellar-population dataset in the `cdict` is thus now both simple, quick and all in C. The `cdict`'s data is output \emph{only once}, as human-readable JSON easily understood by Python, at the end of the simulation. Large simulations are often split across clusters of machines using `binary_c-python`. The data from each run are stored as JSON then merged in Python when the final run completes with little overhead.
To overcome this problem, `binary_c`internally generates an associative-array `cdict` in native C. This `cdict`, and the stellar statistics it contains, is filled inside the `binary_c` simulation as each star is simulated. Generation of the stellar-population data in the `cdict` is efficient because it is only in C and communication with the frontend (Python) code is kept to a minimum. The `cdict`'s dataset is output \emph{only once}, as human-readable JSON easily understood by Python or Perl, at the end of the simulation. Large simulations are often split across clusters of machines using `binary_c-python`. The data from each run are stored as JSON chunks then merged in Python when the final run completes. The overhead involved in this joining is small compared to the effort of simulating the stars: the goal of `libcdict` has thus been achieved.
