inconsistent annotations
In several data points the annotations are inconsistent with the scheme provided.
(e.g. train 516 has yes
for q1, followed by nan
in q2)
The evaluation code also seems not assert the validity of nan
values (ie. checking if the answer to q1 is no
).
Can we get a fast update on this? Are the gold labels for q2-5 going to be assigned based on q1, or the current labels?