From table to comparison: User Story
Step by step
All of this outside the discussions is just an observation. Opinions and Discussions are tried to be contained inside the discussion, to be - well - discussed.
The structure
To get from this list:
to a comparison, the following steps are required:
- create 1 template for the adding each row: Recommendation System
- This template has 1 target class, also called Recommendation System
- Within that template, there are 3 properties for the 3 main columns:
- each of these 3 properties (columns) take exactly one value (with nested subvalues for subcolumns), respectively of the 1 class they each have:
- each of these 3 classes have 1 template:
- within those templates, there are 11 new and 6 reused properties for the 17 subcolumns
- relevancy estimation
- new: based on human rating
- new: based on dataset
- new: based on paper
- target value
- new: clicked
- new: read
- new: cited
- new: liked
- new: relevancy
- new: other user
- new: other automatic
- evaluation measures
- relevancy estimation
Discussion points:
Refered to as "1.1." etc., cannot format otherwise here
- Should properties be uppercase or lowercase? - see Precision and recall
- Should abreviations be used? - see MRR (Mean Reciprocal Rank) and mean Average Precision (see 1. here as well)
- Templates can require a field to be of a certain type - but not a list of certain types. e.g. I might want "stated in" to be "paper", but also allow "Review" and "List" - right now, I can only hard-force one single thing. I can also not set paper to "should be", which would be a recommendation, but the user could still add something else, maybe through a "I understand" checkbox or similar. -> orkg-frontend#1666
.
- "Cardinality" does not seem to persist - I can edit it to "Exactly 1", save, refresh, and it's gone. -> orkg-frontend#1667 (closed)
The list
Given these 1(+3 nested inside) templates, one can use the doi and titles from the References to make a list of the papers:
Here choices like merging entries or not found entities can be annotated.
Discussion points:
Nothing. Lists are wonderful as they are right now. Maybe one thing:
Refered to as "2.1." etc., cannot format otherwise here
- If I add a paper to a list, it gets added with all it's contributions. If I then add it to a comparison, I need to click into the paper to select which contribution is ment. I was able to do that, but this feature might be helpful in the list / comparison view. -> orkg-frontend#1668
- All people that did anything to any of the papers inside the list are called contributor. I'd seperate the two kinds of contribution, counting a) List contribution (anything done to the list, like text and adding links to papers), and b) (List) content contribution (anything done to papers added to the list). This would be much, much less confusing. The "..." for smaller percentage contributors is great already, but does not solve this issue. Related:
- orkg-frontend#1198 discusses this issue existis on other artefacts as well. Maybe we need to come up with universal terms for the a) "Object right in front of me" and b) "content inside that object" contributions. I guess anyone who created one of the properties mentioned above is now a contributor to my list, which just feels wrong on many levels.
- orkg-frontend#852 notes differences between published and unpublished. maybe that part is resolved, the underlying issue is the same.
The comparison
Given this list, one can select all of the resources inside to create a comparison:
Discussion points:
Refered to as "3.1." etc., cannot format otherwise here
- Settings like Actions - Select Properties -> does not persist when pressing F5. Given there is not really a way to just update the comparison, I'd appreciate either the settings to persist (best case) or a different way to refresh the comparison. -> orkg-frontend#1674
- Adding a single row (~10 clicks in Excel) takes ~100 seconds, and (I did not count yet) (I did now: 48 clicks, 1x typing the top level template name, 1x pressing F5, + the 6x2=12 clicks for every true value clicks: 60 clicks and some button presses to add 6x "true", 11x "false" to a table.)
- Using the comparison contribution editor is (absolutely!) great in theory, but has no way (that I found) to use templates. If I could just say "use the template here like everywhere else and just click may way through the comparison editor, I would absolutely love that.
- CSV import, as far as I know, also does not support templates. If I could, say, export a template as CSV, that would naturally be lovely as well: Export my template, do everything in Excel, import it, bam - solved. Naturally, we should be able to support fluent working inside the ORKG as well, but as seen here, some things are just faster and more dynamic in excel, so why not allow some more flexibility. -> orkg-frontend#1670
- While I can hide the property relevancy estimation, I can't change the fact that the other properties are called relevancy estimation/relevancy estimation/x ,where the last is actually not the class relevancy estimation, but an individual resource having the same name as the class it's instance to. I understand why all of that is like this, but I'd appreciate the toggle option to hide this, even if just for a screenshot. This becomes especially dramatic if you transpose the table, making a row that was called "liked" now "evaluation measures/evaluation measures/liked".
- The comparison, as far as I can tell, does not support re-arranging the order of properties. This means I cannot a) recreate the table as it was originally (Human Rating, Dataset, Papers, ...), but have to live with whatever Order the comaprison thinks is best. Neither can I recreate the pattern used by my template, where I was able to sort properties - this means the data is in different order when I add it to the contribution vs. when I see it in the comparison. Fixes would be a) the comparison listening to the template order, or b) even better, allow me to sort them as I see fit, like I can do with the contributions (I think I can do that, right?) -> might be this issue: orkg-frontend#1472
- Contributions from the same paper should be grouped / groupable under the same title like two merged columns. The user might still wnat to change this, e.g. P1_c1, P2_c1, P1_c2, P2_c2, so this should just happen for neighboring contributions of the same paper. This is a bit more complicated since the label might not be in the view for 20 contributions from the same paper, and making it sticky would be complicated. -> orkg-frontend#1669
Related issues here:
- Comparisons should benefit from collapsable titles (and other wider content) to get a big picture: orkg-frontend#1661 (closed)
- Nested labels look nowhere near as good as the artefact inside: orkg-frontend#1662
Inside the comparison: Paper Contributions
Shortly adding to the math we begun with, we now add 53 contributions with 3 resources each, being instances of:
Discussion points
- Adding a value means -> clicking +, then having it autoset to false (great!), then requiring a "create" (not great). The "create" part feels unintuitive and not in line with the rest of the orkg, making everything "permanent" as sonon as I do it. I don't confirm adding the propety, but have to for the value. On an edit-level, I see the issue of "not wanting to wrongfully overwrite". This is a different, more complex issue. Creating should not be this complicated. -> orkg-frontend#1673
- When leaving a page with these "uncreated" resources, I get no warning - through F5 or through any other navigation. This exact issue is mentioned here: orkg-frontend#1408
- If a template is used and has "exactly 1", e.g. boolean, it could be argued that this boolean should already be waiting there for me when I select the template
That means we have 4 templates, 4 classes, with 17 properties, 1 list, 1 comparison with 53 paper resources, 53x1=53 contributions, 53x3=159 resources.
of these, 53 resources + 1 class + 1 property are called relevancy estimation, same for target value and evaluation measures.