New feature: IRR
INTER-RATER RELIABILITY (KAPPA)
Coded utterances only Principle rater (PR) designates (relevant context for) narrative segments that are of particular import in various sources (e.g. controversial or tricky coding, densely or sparsely coded sections); these are presented to Rater 1 (R1) and Rater 2 (R2); kappa is calculated by comparing: a) randomly selected utterances from those segments b) all utterances in those segments
Coded and non-coded utterances Principle rater (PR) designates (relevant context for) narrative segments, but randomly selected utterances/segments are also added by ROCK. These utterances/segments can be: a) coded (with same or various codes) b) non-coded
Utterance-based Random selection of coded narrative segments from all sources are presented to R1 and R2, and compared to PR, with the following selection options: a) any code (randomized selection from all coded utterances) b) any code + non-coded utterances (randomized selection from all coded and non-coded utterances) ((can manually determine proportion here?)) c) designated code(s) (randomized selection from all utterances coded with certain code/s) d) designated code(s) + non-coded utterances
Window-based Same description and options as above, except you can manually determine a “window size” for each utterance to create relevant context for the randomly selected utterances. You can choose from various sizes, e.g.: 1-2, 10-15, 20-25.
Why do IRR? -to make sure raters in the same project are on the same page concerning code definitions (this is especially important if multiple raters are using the same codes to code different sources) -to test whether a team that is reproducing a study is able to reproduce the coding itself (sharing a code book is not enough, the operationalization of each code can get tricky)
INTRA-RATER RELIABILITY Same options as above (both manual and automatic, the latter is probably preferred). The PR is the same as R1, i.e. you are checking if your understanding of your codes are the same at the beginning of the project as at the end.
Why would you want to do this? Even if you are an expert in your own codes (whether adopted or self-developed), your understanding and operationalization of each code may change with time (as you identify various examples for codes in narrative corpuses, particularities may inconspicuously alter your definition of the code itself). This is a good way to check whether your coding has stayed consistent throughout the sources that you have coded.