Drafts Phase 1 - Sync Drafts to Server
Given some of the changes in course from the original work started here: #248 and here: !272 (closed), this work stream is being broken into phases, the first of which is described in this issue.
Phase 1
- store drafts on the server
- sync drafts from user devices to the server automatically
- ability to delete drafts
- only creator can see their drafts
Subsequent phases will include:
- my submissions / drafts UI updates
- make drafts visible to group admins, read-only
Cross Cutting Changes / Conceptual Overview
Submission State Representation
- READY_TO_SUBMIT
- represented by an object with
type
READY_TO_SUBMIT
within thesubmission.meta.status
array. (same as today) - [client] will be added on drafts in IDB when a user attempts to submit the draft.
- [api] will be used ephemerally on the server to decide whether to turn drafts into submissions when drafts are received from the client.
- [api] will not need to be stored in the database, and will be removed from existing submissions via migration.
- represented by an object with
- READY_TO_DELETE
- represented by an object with
type
READY_TO_DELETE
within thesubmission.meta.status
array. (this will be a new status type) - [client] will be added on drafts in IDB when a user attempts to delete the draft.
- [api] will be used ephemerally on the server to decide whether to delete drafts when drafts are received from the client.
- [api] will not need to be stored in the database.
- represented by an object with
- FAILED_TO_SUBMIT
- represented by an object with
type
FAILED_TO_SUBMIT
within thesubmission.meta.status
array. (this will be a new status type) - [api] this will be added to a draft that was
READY_TO_SUBMIT
if submission fails due tohandleApiCompose
failing
- represented by an object with
- UNAUTHORIZED_TO_SUBMIT
- represented by an object with
type
UNAUTHORIZED_TO_SUBMIT
within thesubmission.meta.status
array. (this will be a new status type) - [api] this will be added to a draft that was
READY_TO_SUBMIT
if the submission fails due to lack of authorization according tohasSubmissionRights
- represented by an object with
- DELETED DRAFT
- represented by a boolean property:
submission.meta.isDeletedDraft
- [api] set to false by default
- [api] set to true when a draft is deleted
- We soft delete drafts in this way because we need to preserve their
dateModified
timestamp in order to reconcile them against other drafts. (This comes up in edge cases where a user is interacting with a draft on multiple offline devices. If they delete the draft on one device, but later edit it on another, we need to be able to persist their latest changes)
- represented by a boolean property:
- DRAFT
- represented by a boolean property:
submission.meta.isDraft
- [client] set it to true when drafts are created in IDB
- [api] store it in the database.
- [api] set it to false when (1) posting a submitted submission and (2) when turning a READY_TO_SUBMIT draft into a submitted submission
- [api] will use a database migration to add
isDraft: false
to all existing submissions in the database
- represented by a boolean property:
Proxy Submissions
-
THIS WILL NOT BE PART OF PHASE 1:
- Implementing proxy drafts requires additional work not relevant to the rest of phase 1, and so is a good piece to drop to reduce the scope of phase 1.
- Phase 1 will give the same treatment to proxy drafts as drafts of resubmissions (they will not be persisted, and the user will be prompted to complete them or discard them if they leave the draft)
- There are currently bugs with proxy drafts that result in the proxy info being lost on submission, so we aren't losing functionality, and we're fixing a bug by taking this approach during phase 1.
- When syncing drafts with the server, we will make a single request with all drafts that need syncing. That request may include proxy drafts (which may each have different delegated users), and non-proxy drafts. Proxy submissions currently utilize a
x-delegate-to
header. This header won't work with the newsync-drafts
endpoint, because we may have a mix of proxy and non-proxy drafts to sync in a single request. We need to know which specific drafts are the proxy drafts, and which user to delegate them to.- To support the
sync-drafts
endpoint we will have new logic that looks tosubmission.meta.submitAsUser
instead of thex-delegate-to
header. The POST/submissions
endpoint will be updated to utilize this same logic. - The GET
/farmos/farms
and GET/farmos/assets
endpoints that rely on thex-delegate-to
header could remain as is, or could be updated to take auser
query parameter if the request is for a user other than the requesting user's resources, instead of looking to thex-delegate-to
header. Given time constraints, the farmos endpoints will probably remain as is for now. - In the process, we will fix some bugs where draft proxy submissions would not be submitted correctly.
- To support the
- When getting drafts, the true author (not the delegated user) should be used to determine the drafts that belong to a user. Another way to say this is that proxy drafts belong to the user that started them as long they are drafts. The user a proxy draft is delegated to will not have visiblity to the draft until it is submitted.
Syncing Drafts Between Client and Server
- see
/submissions/sync-draft
below.
Survey cleanup
- Questions:
- How many different types of cleanup tools are there?
- Answer: just survey cleanup
- Are we still using them or are they supplanted by requests only asking for latest versions?
- Answer: They are still in use. But maybe they don't need to be anymore?
- If we're still using them, do they have the ability to cleanup too much and break stuff?
- Answer: Yes. The survey cleanup tool could delete versions of a survey that are relied upon by drafts in idb that have not been synced to the server yet.
- How many different types of cleanup tools are there?
- Info:
cleanupSurvey
: I think it just removes version from thesurvey.revisions
array. It doesn't delete that survey version I don't think. - Potential Solution: So, if we were to encounter a submission that references a survey version that is not in the revisions array, we could add back that version to the revisions array I think?
- Potential Solution: Are cleanup tools still necessary? If we could remove them, we wouldn't need to maintain them.
Server Changes / Requirements
/submissions
, GET /submissions/page
GET - do not include submissions where
isDraft
is true unless the requesting user is thecreator
- if
isDraft
andsubmission.meta.proxyUserId
is present, useproxyUserId
in place ofcreator
. - We will need a query parameter to allow requests to choose which submission states they care about. Requests should be able to ask for only drafts, only submitted, or all.
- The default should probably be to exclude drafts
/submissions/csv
GET - Update to allow for the inclusion of drafts via query parameter (the same one for GET /submissions and GET /submissions/page). There are no immediate plans to use this for this endpoint, but we should maintain consistency between
/submissions
,/submissions/page
and/submission/csv
. - This endpoint is only used on the results page, which will not be including drafts.
/submissions/:id/archive
, POST /submissions/bulk-archive
POST - cannot archive if
isDraft
/submissions/:id/reassign
, POST /submissions/bulk-reassign
POST - if
isDraft
, requesting user must be thecreator
/submissions/:id/send-email
, GET /submissions/:id/pdf
, POST submissions/pdf
POST - OPEN QUESTION: is it desired to be able to get pdfs of drafts?
- if
isDraft
, requesting user must be thecreator
- we don't currently assert users are authorized for this endpoint, but need to
/submissions/:id
GET - if
isDraft
, requesting user must be thecreator
- if
isDraft
andsubmission.meta.proxyUserId
is present, userproxyUserId
in place ofcreator
.
/submissions
POST - this endpoints job remains the same: to accept the initial submitted copy of a submission and save it to the db. It will not handle the posting of draft submissions (that will be handled by the
sync
endpoint). The changes below allow this endpoint to distinguish those two situations. - if
isDraft
is true, submission must beREADY_TO_SUBMIT
. If it's not, reject the request. - set
isDraft
to false before inserting into DB. - if id already exists in db, return 409 (conflict)
- stop using
handleDelegates
and instead look tosubmission.meta.submitAsUser
/submissions/:id
PUT - don't allow updating of drafts with this endpoint. Only support that via
sync
endpoint. You can turn a draft into a submitted submission with this endpoint. - if
isDraft
, submission must beREADY_TO_SUBMIT
. If it's not, reject the request. - if
isDraft
, ensure the copy in the db is not already submitted, return 409 (conflict) if it is. - set
isDraft
to false before inserting into DB.
/submissions/:id
, POST /submissions/bulk-delete
DELETE - you cannot delete drafts with these endpoints. Deleting drafts involves soft delete logic, ignoring deletes if a draft is already submitted, etc. We don't want to complicate this endpoint with that logic. To delete drafts, we will utilize the
sync-drafts
endpoint. - lookup submission by id in db. if
isDraft
, reject request.
/submissions/sync-draft
New Endpoint: POST - request body contains a draft from the client that needs syncing (this request will be made for each draft in IDB).
- high level: the server will reconcile conflicts, save the results to the database and return a success response once finished. The client can then get the latest drafts/submissions by making a GET
/submissions/*
request. If the request fails in a way where retrying is likely to succeed, the server will return 500. If the request fails due to lack of authorization, failed api compose, or any other reason that would continue to fail if retried, the draft will be updated with a status ofFAILED_TO_SUBMIT
orUNAUTHORIZED_TO_SUBMIT
and persisted to the server, and the response will have status 200. - for each draft in the request, the server will see if there is a submission with the same id in the db.
- if there is no submission in the db, the draft from the request is put into the db.
- if there is a submission in the db
- if the db copy is submitted, the request copy is discarded. Even when the req copy is
READY_TO_SUBMIT
. - if the db copy is a draft, the latest last modified timestamps will be compared between the db copy and the request copy.
- these timestamps can appear on: (1)
meta.status
forREADY_TO_SUBMIT
(2)meta.status
forREADY_TO_DELETE
(3)meta.dateModified
- if the request copy is the latest
- if
meta.status
isREADY_TO_DELETE
, set the submission in the database to be soft deleted (viasubmission.meta.isDeletedDraft
. It is soft deleted because we must preserve the timestamp from theREADY_TO_DELETE
status, to reconcile against if we see the draft with this id gain at the sync endpoint. - if
meta.status
isREADY_TO_SUBMIT
, save it to the db, removing theREADY_TO_SUBMIT
status, and settingisDraft
to false.- submit to QSLs if applicable
- run handleApiCompose
- check authorization in the same way we do for
createSubmission
- Note: It is possible for
READY_TO_SUBMIT
to be present without it being the latest timestamp (in the case of a user going back to edit a draft that was previously put intoREADY_TO_SUBMIT
status). Make sure drafts in this state are still submitted.
- if neither of the above
meta.status
is present, save it to the db.
- if
- if the db copy is the latest (and is a draft), or if timestamps are equal between DB and req
- if req copy is READY_TO_SUBMIT, don't care. Don't update the db.
- if req copy is READY_TO_DELETE, don't care. Don't update the db.
- else if req copy is edited, don't care. Don't update the db.
- these timestamps can appear on: (1)
- if the db copy is submitted, the request copy is discarded. Even when the req copy is
- server confirms requesting user is the creator of the drafts, returns 401 if not
- server validates the submission in the request is a draft, returns 400 if not
- for proxy drafts look at
submission.meta.submitAsUser
(not thex-delegate-to
header), perform authorization, and assigncreator
andproxyUserId
as appropriate for proxy submissions ----- this stuff is covered in the requirements above:- this will be part of a later iteration of development
-
READY_TO_SUBMIT
: once submitted, ignore further edits, there is no draft anymore. -
READY_TO_DELETE
: when reconciling conflicts between deleted drafts and edited drafts, last modified wins (even if it means undeleting a draft, or deleting a draft).
- high level: the server will reconcile conflicts, save the results to the database and return a success response once finished. The client can then get the latest drafts/submissions by making a GET
Migrations
- remove
READY_TO_SUBMIT
status from existing submissions- need to add code to remove
READY_TO_SUBMIT
when processing submissions with POST /submissions and maybe PUT /submissions also, so that we don't ever putREADY_TO_SUBMIT
in the db again
- need to add code to remove
- set
isDraft
false on existing submissions - set
isDeletedDraft
false on existing submissions - make sure to bump submission specVersion where submissions are created in client, or wherever it's initially set
Client Changes / Requirements
- For pinned surveys, make sure drafts for that survey are available offline with NetworkFirst strategy.
- When you try to edit a draft while offline, it fails if the survey for that draft is not cached. It shows an infinite loading spinner instead of an error message.
- OPEN QUESTION: Do we want to ensure you can edit your drafts while offline (by making sure the survey is cached)? Or do we just want to show an error message that says "There was an error fetching the survey. Please try again. Pin surveys to make them available offline."
- Where are all the places we render submission in the UI? Where we do, ensure submissions with
isDraft
true are presented correctly, or decide to filter them out if they don't make sense in that place.- survey results
- We'll filter out drafts to start. The logic for when to show archive, delete, reassign, restore, etc. doesn't work if drafts are included in the list. Users don't see drafts there today, so this should be fine for now.
- my submissions
- others?
- survey results
- READY_TO_SUBMIT
- remove buttons to push READY_TO_SUBMIT drafts to client, as it will happen automatically instead
- button next to each READY_TO_SUBMIT drafts in drafts list
- button above drafts list to push all READY_TO_SUBMIT drafts
- remove chip showing count of READY_TO_SUBMIT in nav.
- review places that reference
'READY_TO_SUBMIT'
, and remove any unused bits - indicate on the my submissions / draft list item that a draft is READY_TO_SUBMIT maybe. Maybe say something like "This draft will be submitted when you are back online." or "Awaiting sync..."
- remove buttons to push READY_TO_SUBMIT drafts to client, as it will happen automatically instead
- Sync when:
- leaving a draft without submitting or discarding
- opening the app (if any drafts in idb)
- returning to the tab (if any drafts in idb)
- leaving the app (if any drafts in idb)
- will we have a chance to receive the response and clear drafts from idb or only to send the request? If the latter, may not be worth doing.
- before fetching a list of submissions (if any drafts in idb)
- my submissions page
- survey results page (if drafts for that survey in idb)
- where else?
- ensure failing to sync is handled gracefully (due to being offline, or 500, or timeout)
- we should continue on failure, not blocking the user from using the app. So, how are unsynced drafts reconciled?
- a later successful sync
- they can be deleted
- if the server returns a response indicating the draft was already deleted or submitted, delete the copy from idb.
- if they are updated but not submitted, the server will accept them because it always accepts the latest action from the client, even if the client was missing some intermediate data. Upon successful update, the copy in idb is removed.
- if they are completed and submitted
- if they are not already submitted, the server will accept them because it always accepts the latest action from the client, even if the client was missing some intermediate data. Upon successful submission, the client deletes the copy in idb.
- if we see a more recent copy in the course of making GET
/submissions/*
requests, the idb copy should be deleted. (do nothing if we see the same id but the idb copy is more recent).
- we should continue on failure, not blocking the user from using the app. So, how are unsynced drafts reconciled?
- prevent drafts of resubmissions
- when leaving a draft resubmission, notify user it will be deleted, then delete it when they confirm leaving
- when a user is working on a resubmission, pass
persist=false
toDraftComponent.vue
, to instruct the app not to persist the submission in progress in idb.- ensure that a user can leave/minimize the app and return without losing their progress with this approach.
- set READY_TO_SUBMIT before attempting to submit (current behavior)
- editing of READY_TO_SUBMIT drafts...
- current behavior allows editing of READY_TO_SUBMIT drafts, but does not update READY_TO_SUBMIT timestamp
- we will keep this behavior. We need to ensure that
meta.dateModified
is updated when changes are made (this should be current behavior), and that thesync-drafts
endpoint submitsREADY_TO_SUBMIT
drafts even if themeta.dateModified
timestamp is later than theREADY_TO_SUBMIT
timestamp.
- how to represent states on the client
- READY_TO_SUBMIT and READY_TO_DELETE statuses
- be explicit about isDraft even though anything in IDB is necessarily a draft. So, add
isDraft: true
when instantiating a draft submission
- ensure
meta.dateModified
is set when it should be, whenever an answer is updated (this currently happens in Control.vue::methods::setProperty) - When resuming editing of a draft that has been saved to the server, we need to manage the working copy of the draft, and the latest copy from the server.
- a working copy of the draft should be put into idb when editing of the draft is started (this should be current behavior)
- the vuex drafts store will store
- an array of drafts from idb (current behavior)
- an array of drafts from the server (new behavior)
- for a given draft id, vuex getters for drafts will return the idb copy if present, otherwise the server copy.
- this will ensure the latest copy known to the device will be returned from the getters each time, even if the service worker has cached a copy of that draft. It also keeps the array of drafts from the server in vuex being just that. We don't get into logic where we'd try to clear a draft from the drafts from server array when we start editing that draft, or discard part of the response from the service worker if we see a draft with the same id in idb.
- ability to delete/discard draft from list of drafts on my submissions page
- mark as READY_TO_DELETE
- remove READY_TO_SUBMIT if present
- remove from IDB on successful delete response from server
- do not display READY_TO_DELETE submissions in UI, they will be deleted on the next sync. As far as the user is concern, they are deleted as soon as they hit delete, regardless of successfully communicating that to the server.
- ConfirmLeaveDialog
- for resubmissions, notify progress will be lost
- for submissions, options to (1) discard, (2) save & leave, (3) stay
- ConfirmSubmissionDialog
- any new behavior here?
- ensure READY_TO_SUBMIT is applied
- ensure draft is cleared from idb on successful submit
- ensure draft is preserved in idb on submit failure
- toasts / indications of syncing in progress, success and failure
- add
isDraft
true to draft submissions when they are initialized - ensure we sync before making requests to
/submissions/*
if there are any drafts in IDB. - a user may have drafts for another user in idb. (they create a draft while offline, logout while offline, then login as another user when back online)
- so when syncing and displaying drafts, filter out drafts if creator is not current user.
- delete a draft from idb completely (not just mark as READY_TO_DELETE) on:
- successful sync
- successful deletion (which will happen via sync endpoint)
- successful post or put
- failed post due to already being on the server in deleted, draft, or submitted state (409) (should be impossible)
- failed put due to already being submitted (409)
Edited by Brian Kockritz