Merge branch 'jira-8687' into 'master'

See merge request floss/liza!313

Shelly Shaver authored Nov 18, 2020 and Shelly Shaver committed Nov 25, 2020

agentID should continue to save in mongo as a string;
eventually this field will not be needed and we will begin
to use meta data to verify ownership and visibility of a
quote

Shelly Shaver authored Nov 17, 2020 and Shelly Shaver committed Nov 25, 2020

Additional changes to enforce that agentId is numeric; What is in
Session may not be a number so enforce this in the getter.

Add unit tests

This includes some fixes for UI-related issues.  UI changes are also behind
the kickback flag now.

This feature was introduced with kickback and needn't take effect if
kickbacks are not occurring.

This should have been behind a flag to begin with; the change was rushed.

This restores the DataValidator back to its previous behavior, because it
was causing unintended side-effects with displays.  Really, though, it
probably would have caused side-effects with fields as well by clearing them
in certain scenarios when a sibling index was modified.

This was a rushed change initially due to time constrains, as is this
change, so I want to be a bit more conservative and isolate the clearing as
much as possible.  Implementing this _properly_ would take far too long.

The comment in `emptyBucket` explains what this change does.  Essentially,
`undefined` should only be in the bucket on empty if the validator failed
for existing bucket data, so we should clear the field.  Clearing the field
ensures that default values on DOM elements are forcefully cleared so that
the state of the bucket (bogus) is accurately reflected, and the user is
able to select a value even if that value is the default.

These were temporarily reverted because our deployment process isn't in a
good place atm and something else needed to go out.  Considering the
refactoring involved here, I did not want to rush things.

This reverts commit 172eaf49.

See merge request floss/liza!315

A new method was added to GroupUi to call the GroupContext
removeIndex method as the first step of the GroupUi removeIndex
method, for all groups that support multi-index.

We have a somewhat urgent release that needs to be made, and there are some
outsanding issues with these changes, so they'll be reintroduced shortly.

Revert "[DEV-8616] Only clear invalid data on retrieve"

This reverts commit d7348e4b, reversing
changes made to 365db4ab.

Revert "Kickback on program version change"

This reverts commit 365db4ab, reversing
changes made to 87d40fa9.

See merge request floss/liza!314

The idea is that any old documents will have been migrated by triggering
them to generate their field state, e.g. via a POST using skey.  Then we can
simply fail when it's not available, allowing us to directly address
failures rather than risk it operating in ways that are unexpected and cause
problems.

Since saving prior state happens even when the feature flag is disabled,
this allows us to deploy the change to start the process of gathering those
data, without actually affecting documents in any meaningful way.

This was something I mentioned early on in planning this little project, but
forgot to address, especially after the direction changed with invalid field
state.

This is a step toward implementing states for bucket field values (the
intent will be something along the lines of
NA|Pending v|Invalid v|Valid v).

Since I do not have time to fundamentally rework the Bucket and how it's
accessed at the moment, this does the best we can with what we have by
introducing the concept of an invalid state using an ASCII NACK character as
a string prefix.  The validators will consider this to be invalid in a
guaranteed way, and calculated values have been modified to take that into
consideration as well, since the validators may not have run by
then (e.g. on init).

eslint: Ignore loader, since it fails to parse

  0:0  error  Parsing error: Cannot read property 'map' of undefined

Will check again in future eslint versions.

This is only one of the missing ones, but it's better than nothing...!

The intent of this is to catch validations caused from server-side changes,
since we previously assumed (incorrectly) that, once things are in the
bucket, they are in a valid state, and forever will be.

This is not ideal, but is an incremental step toward a new way of
considering bucket data, which will require a far more fundamental
refactoring.

See extensive detail in the unit tests and comments.  This is the core of
this task and the feature that's been worked toward all along.

Also note that this forces the user to the current step when loading a quote
now, which will break hyperlinks to specific steps.  I don't think that's
currently done, and in practice we redirect users to policies after
documents (quotes) are locked, so this shouldn't affect us; ideally, though,
this would use e.g. the kickback action, which does not currently work
because the Quote object is not yet available to the action on the client
during init.

This task is also not all that great in terms of FP, but as was mentioned in
a previous commit, I intend to refactor further later on.  We also need time
to get the team acquainted before diving too deep into a sea of monads.

Migration may cause a kickback, so this will ensure that it is immediately
persisted.

This also logs information about whether a kickback has occurred (just a
little bit).  In the future I'd like to also log what fields caused it.

Tests are absent for this.  This has been long delayed and certain things
have to give; I have more important things I have to move on to.

After the classifier is run, a subset of its state will be persisted so that
it can be compared on Program version change.

fieldState will only be saved if it's present and non-empty, so as not to
overwrite previous state if the classifier has not been run.

We no longer need to provide Promise support to tests; that was for really
old environments.

The index.js files must be regenerated for many tests to work properly.

These are dynamically generated (and then cached), and will only be used by
the server.  I wanted to avoid generating them as part of Program.js (from
liza-proguic) because that'd bloat the JS sent to the client even more when
the information is not only redundant, but will also (at present) not be
used by the client.

The note about ease.js here represents a bug that must have existed since
getters were first introduced nearly a decade ago.  I don't have the time or
will to fix it, since we're actively deprecating ease.js.

It's not very good, as I continue to learn the particular implementation
details of fp-ts, but this will allow us to continue adding to the migration
in a clear, linear way.  Finally.

At present, this is only on program mismatch, but we can further it as time
goes on.

Note that this is a near-verbatim copy, with minor modifications to get it
to compile with TS.  Further improvements will be made at some point.

There's no use in having all of these similar concepts in different
places.  The convention with OOP was to have each class in its own separate
file, but functional programs don't have quite the same distinction.

It'll be easier to reason about the system if all of this is the in the same
place.

This gets rid of the nasty hook-based system and instead injects the
dependency.  While this does add to the dependencies of Server, which is
bad, the overall effect is much cleaner, more understandable code.  The
dependency will be removed in the future as things are moved into the
loader.

I decided to leave `clean` async since, as it becomes more intensive, we may
wish to take advantage of that, at least as long as it's on the same
thread.  Further, other parts of the loader are async, so the types will fit
together more nicely.

This will aid in a transition to the loader.

Further, it appears that it never actually returns any errors...and there's
a bunch of complex code around whether cleaning fails.  This can be cleaned
up a ton, which would be great, since version migration is some of the most
confusing (and therefore dangerous) code.

Saving still occurs within `Server`, but the loader is now responsible for
preparing all related data.

The functional code is a mess; I'll refactor further in a separate commit
so it's more clear what's being changed, though I don't know if I'll have
time any time soon.  In particular, perhaps we ought to use a
ReaderTaskEither.

There is no point in having this separate anymore.  This is essentially a
copy, so the logic is still imperative for now.

This is essentially like exporting a private method, but we'll sort that out
in the future.

This replaces the previous incremental mess with fp-ts.  The chaining is a
little bit awkward with the type conversions; I'll sort that out in a bit; I
wanted a literal translation to show how this evolves.

This extracts the logic into `loader`.  This is once again a transitionary
commit; see previous commits for more information.

This change is notable in that the `#initQuote` method is now fully glue
code for functions in the loader, and so itself can now be extracted in its
entirety.

Also noteworthy is that I intend to merge ProgramInit itself into `loader`,
since it does not make sense to have that logic stand on its own.

This is another incremental stepping stone; it's not just to make the code
more verbose. ;)  See previous commits.

0 is never a valid document id; it's used to request that a new document be
created.

The body of this function now consists of two (tightly-coupled) calls to
functions defined in the new loader module.  See previous commit for more
information, including on the coupling.

This begins a transition into a functional module, which I'll commit
incrementally to show how it evolves.  It is tightly coupled at the moment,
and probably will remain that way until it's extracted out of Server
entirely.  This is partially due to my availability.

This is the first use of fp-ts here.

This makes this method a pure function of its inputs, which simplifies
extraction.

This timestamp should only be set when a quote is created, anyway.

This will be used by future commits.

I've wanted to introduce a functional library into Liza for quite some time,
and TS has enabled the creation of a fairly robust implementation.  Others
on the team did the research on this (Joe and Anthony, primarily).

While it does have some shortcoming and awkward limitations due to TS, and
it is heavy on terminology, it does seem like a good library so far.