Generate coverage for all journals/members
Current implementation of coverage in ES-based REST API does not use scroll: https://gitlab.com/crossref/rest_api/blob/develop/src/cayenne/tasks/coverage.clj As a result, cayenne.tasks.coverage/check-members
and cayenne.tasks.coverage/check-journals
functions generate coverage for up to 10,000 journals/members only. Indeed, after running check-journals
on staging some journals still do not have coverage (https://api.staging.crossref.org/journals/2053-1583), while others do (https://api.staging.crossref.org/journals/0022-247X).
Scroll needs to be used to cover all journals/members.
Definition of done
-
Available via a staging URL -
Code reviewed -
Unit tests identified, implemented, and passing -
Public documentation reviewed and updated -
Knowledge base reviewed and updated -
Acceptance criteria met -
offset+sort is used to cover all journals/members -
make sure all journals and members are covered by coverage generation -
make sure journal coverage statistics are not broken down by content type -
make sure only the current coverage records are kept in coverage
index, and not historical data (historical data will be covered in #118 (closed)) -
speed up coverage calculations by calculating coverage for relevant content types only
-
Edited by Patrick Polischuk