Pasa Features March / April

Serena did some user feedback within Pasa with those interacting with farmers for the Soil Health Benchmarking Study, and identified some improvements specifically to support Reducing errors in filling out the yearly management form.

Update with Serena on June 11th 2024

Key Dates

@manuel-ch Note talked with Serena and updated the timeline -->

1. Fully scope (pre-technical and technical scoping) by June 1, "MVP", "Want to have", "Nice to have"
2. Implementation period: June/July/August 2024
3. Total effort: $24000 total = 30 person days total = 3 months x 10 person days per month x $800 per day.
4. Final deployment August 31 2024.

Time estimate

@serena.x, below I included the Person Days (PDs) for every piece (for estimation, ballpark 1 PD ~ $800). For transparency - we also include code review and refactoring in any quotes, even on small work. As a small company that mostly works through feature requests, that's how we ensure that the code base is kept up to date and well tested and ultimately save everyone time and effort!

Availability: We'll be splitting OT and Pasa work through June, July and August to progress on both fronts. We'll have devs available part time through that period for this work.

MVP

1. Improving visibility of fields in the selector UIs
   • 1 PD for fields but especially for plantings -- here are some proposed changes (https://miro.com/app/board/uXjVMD9-x1g=/?moveToWidget=3458764584282864994&cot=14, combined proposal is out of scope of this task, also image below) --> !417 (merged) @manuel-ch

🛑 Desired MVP but still in active review

1. Treat Matrix like Groups, allow relevance and initialize in new Matrix structure
   • Described here #324.
   • Allows them to auto-populate the Planting name.
   • Improves usability of the matrix question by using relevance to show only the needed items.
   • Leads into (3) below "General Validation..."
   • (from Serena July 1) could we also make the default behavior in Spreadsheet view for 'more info' to show this on hover (desktop) or on click (mobile) rather than just showing the full column name. Could also be a modal.
2. General Validation or (at a minimum) Matrix Validation
   • Validation Script for Matrix Questions and Groups
     • For Matrix - 15 PD When a user click's the 'add row' button, a validation checker shows an error (red), warning (yellow), or success (green, optional) result in the matrix. They know the validation applied to the row just added, which makes it clear what they need to fix. You can validate the row, it's contents (values), and the entire table.
     • For Pages - 3 PD (optional - priority is application to Matrix question) the same option could be applied to a page question type, and applied when the user clicks on the the NEXT button. If error appears, then prevent from continuing.
     • Technical implementation Create a new js script type called 'validate', similar to relevance. It is run only on click of 'add row' as compared to being run during any change. This validate script returns a highly structured object { error: { header_name: text }, warning: { header_name: text }, info: { header_name: text } }. This object is returned to the UI to display in a regulated way below the matrix question.
3. Feedback from Serena
   • Will think about how much to prioritize (hours)
   • To realistically use it for this year, it would need to be done by Thanksgiving at the latest
   • First round of training for their partners in about a month (August)
   • (greg) Ask openteam - are they interested in as well? Could we apply some funding for this?
   • (greg) Ask Manuel - given 50% time on Pasa, could we finish with thorough testing by Nov 25? This would also require me to be able to update the surveys (so... probably more like Nov 1)

Want to have

1. Prevent leakage of sensitive data via apiCompose
   • 6 PD this is a data security issue: #298
2. Select Starting Date
   • (🛑 delay) 3 PD Add an option to Date questions in the builder which allows the Survey Builder to specify the starting date which appears to the user when they click on the Date button when filling out the survey. This addresses a problem that users aren't paying attention to the year, and Pasa frequently has users filling out 2024 information for year 2023 (for example), which becomes a data quality problem down the line. This should default to 'today' (as is the case now). !426 (merged) @desousab
     • This option should appear below the type field in the Data question.
     • User should select this option by using the date picker.
     • 🛑 do not start until point 3 validation is scoped, maybe validation solves this more elegant
3. Easier to manage spreadsheet questions @manuel-ch
   • 1 PD improve table header visibility !422 (merged)
     • Allow 2 rows before adding the ellipsis in the header.

Nice to have

1. Modal - Add (?) to allow users to learn more without being overwhelmed by information, generates modal with close option or (x).
   • 3 PD add checkbox option under more_info to change more info to display as modal from (?). Modal will content from markdown... again, not necessarily a bad thing because it gives survey creators lots of options for what to put in there.
   • this prevents an additional field by sharing use of more_info.
   • see above - we could also show a modal on click for matrix view.
2. Easier to manage spreadsheet questions
   • 1 PD quick sort (up or down) in matrix row headers @manuel-ch
     • should work for date, number, or alphabetical.
     • !423 (merged)

Discussion

@manuel-ch or @desousab could you scope these and provide any comments?

1. Tooltips - Add (?) to allow users to learn more without being overwhelmed by information.
   • make this markdown by default for bold, italics, etc.
   • appear at the end of the label or hint fields (proposed logic: if (label) --> label (?), if (hint) --> hint (?), if (!label && !hint) --> label (?) (in short, force the label to contain the (?) if neither is present)).
   • ❓ @gbathree We already have 3 properties to explain a question (label, hint, more info), feels like a lot already. could we also just add a "tooltip" option to the hint or more info property which makes that appear as a tooltip instead of being initially rendered? Or maybe the "overwhelming" effect mostly takes place on mobile, so we could automatically switch to tooltip button instead of fully render hint/moreinfo? ( @manuel-ch )
   • 📋 @DanTerAvest read @manuel-ch 's point above, I'd like your opinion. My long desire for all this is to move our surveys into JSON Schema, and populate the normal keywords $description, $examples, and $comments from what we're adding today. I know that's down the road but just stating it here :) Near term... maybe some options
     • checkbox option so users can specify if more_info should be displayed below question, or added to (?). This may be nice, but bad if they want both. This feels like not a bad solution IMO. But it means (?) should be a popup because it will generate from markdown... again, not necessarily a bad thing because it gives survey creators lots of options for what to put in there.
     • IMO (separately not related, but FYI) I do think we eventually need $comments which is information to pass from the survey creator to another survey creator. It's the 'what were you thinking about when you created this question, and where is it going in the future' kind of thing. It would be very helpful especially for Question Sets where those questions are often asked by those looking at the question set itself.
2. Select Starting Date - Add an option to Date questions in the builder which allows the Survey Builder to specify the starting date which appears to the user when they click on the Date button when filling out the survey. This addresses a problem that users aren't paying attention to the year, and Pasa frequently has users filling out 2024 information for year 2023 (for example), which becomes a data quality problem down the line. This should default to 'today' (as is the case now).
   • This option should appear below the type field in the Data question.
   • User should select this option by using the date picker.
3. Easier to manage spreadsheet questions
   • drag matrix rows up or down the page to reorganize
     • We'd need to do this without consuming a lot of visual space (like, a permanent element to indicate a 'grab' with your mouse I think takes too much space)
     • ❓ @gbathree I don't see much alternatives to a (slim) grab indicator at the beginning. But for compensation, we could move the current two buttons (delete and clone) into a single three-dot menu. ( @manuel-ch )
     • 📋 mmm... IMO delete and close are super helpful and needed always + hesitant to take up more space... is it possible to grab + drag on the card itself (guessing not easily or without lots of potential for error)
       • let's ask Serena esp... do you think sorting is enough to address the goals here? Should we try that first and see if the problem continues to rank high on the list?
   • sort matrix rows by date or field
     • Personally I like this a lot. Add some functionality to our custom table that's intuitive and easy. Need to review UI options, again, users are often viewing this in mobile or situations where space matters.
   • improve table header visibility
     • We try to give the user settings to set table header size, but it often doesn't work out. I wonder if we could review the table header visibility and see if we could improve it. For example, maybe we should go 2 rows and then use elipsis... this may handle 98% of cases well automatically, without causing crazy extended table header rows... ideas?
     • ❓ @gbathree matrix headers already do ellipsis if required and also a tooltip shows up to see the full header text. Also the "scale minimum width" setting per col is pretty effective in my test case. Could you provide an example/screenshot where it does not lead to the desired result? ( @manuel-ch )
       • I agree generally - I think the option to change table heading width wasn't super obvious, so they didn't use it. However, I do think even with that, it seems to often be the case that one line is just not quite enough, and extending the table heading with to ensure the header text fits causes the overall table to be quite large unnecessarily. Serena and others have reported that wide tables (lots of scrolling) is one of a few repeated complaints to this data entry system - so small gains there do help! Examples: In the two examples below...
         • image 1: area covered (%) has a numerical option which is very short. It's important for the user to understand it's a (%) but that is at the end of the header. So we really want a thin row but for the whole header to be visible
         • image 2: used for termination is a longer header and important for the user to see... however, the answer is yes which is short. Ideally, we could shrink the row size but keep the header fully visible.
       • @manuel-ch in both cases, if we simply allowed 2 rows before elipsis instead of 1, the problem IMO would be solved. Also @manuel-ch it may allow us to default our columns slightly thinner (as we now have double the default space for the header) which would cause the problem to arise less often for users in the first place. Serena and @DanTerAvest would you agree here?

4. General Validation or (at a minimum) Matrix Validation
   • Errors are made at many levels - errors like adding "565 tons of manure" versus "565 lbs", errors like having a planting date later than a harvest date, duplicating a row and not realizing it, (from Serena) "Start date 3/1/24 is after end date 9/1/23", "Planted area of 100 acre(s) is greater than the total field size of 1 acre(s)", "Planted area of 2 acre(s) is greater than the total field size of 1.9 acre(s)"
   • Some errors in matrix occur at the level of the individual data, the row, and the table itself.
   • Data validation is also desirable in other applications and questions, and could be a generalized need.
   • Ideally, the requirements would be:
     • check each value
     • check each row (row by row, so errors are caught quickly making fixing them easier and more obvious)
     • check the table
   • Proposal 1: Validation Script for Matrix Questions and Groups
     • 👍 check each value
     • 👍 check each row
     • 👍 check each row row-by-row
     • 👍 check the table
     • Experience: When a user click's the 'add row' button, a validation checker shows an error (red), warning (yellow), or success (green, optional) result in the matrix. They know the validation applied to the row just added, which makes it clear what they need to fix. You can validate the row, it's contents (values), and the entire table.
     • Technical implementation: Create a new js script type called 'validate', similar to relevance. It is run only on click of 'add row' as compared to being run during any change. This validate script returns a highly structured object { error: { header_name: text }, warning: { header_name: text }, info: { header_name: text } }. This object is returned to the UI to display in a regulated way below the matrix question.
       • Note, the same option could be applied to a Group question type, and applied when the user clicks on the the NEXT button. If error appears, then prevent from continuing.
   • Proposal 2: Make the UI for Scripts cleaner so scripts can be added to pages which require validation. Users click 'check my work' and it reviews the page and displays a custom output.
     • 👍 check each value
     • 👍 check each row
     • 👎 check each row row-by-row
     • 👍 check the table
     • Experience: When a user completes a table, they see an additional button (for a script, below the table) which says 'check my work'. When they click on it, it generates errors which occurred anywhere in the table.
     • Technical implementation: Requires no changes, except allowing the user to re-run a script easily, ensuring the script button is closer to the top of the screen (more accessible) rather than at the bottom, and designing the script itself.
   • 💡 @gbathree My favorite is proposal 1 as it follows the current pattern of specific small expressions scripts per question. As opposed, using the Script feature seems to be the wrong pattern here as Scripts belong to Script questions and it would probably require more changes than expected to make Scripts work as part of other questions types. ( @manuel-ch )
     • 👍 agreed!

Additional potential items

1. ❗ Prevent leakage of sensitive data via apiCompose #298
2. Adding equipment references and animals: #294
3. Improving visibility of fields in the selector UIs (for fields but especially for plantings) -- here are some proposed changes (miro - https://miro.com/app/board/uXjVMD9-x1g=/?moveToWidget=3458764584282864994&cot=14, also image below) --> !417 (merged)