Ingestion of DOIs of type database_title throws exception in ES REST API
Background
DOI 10.14293/s2199-1006.1.sor-matsci.cl3wluw.v1 is of type database_title, as can be seen in it's XML representation:
<doi type="database_title">10.14293/S2199-1006.1.SOR-MATSCI.CL3WLUW.v1</doi>
REST API officially does not support this type (https://crossref.gitlab.io/knowledge_base/docs/topics/content-types/). Nevertheless, DOIs of this type are being successfully ingested in production, resulting in type = null in JSON (http://api.crossref.org/works/10.14293/s2199-1006.1.sor-matsci.cl3wluw.v1):
"type": null
In ES REST API, the attempt to ingest this DOI results in an exception (pasted below).
Even though what happens in production is not ideal, I suggest reimplementing the same behaviour in ES REST API, and filing another issue to clean this up after the migration.
ES REST API exception:
expected: nil
actual: nu.xom.ParsingException: null
at nu.xom.Builder.build (:-1)
nu.xom.Builder.build (:-1)
cayenne.xml$process_xml.invokeStatic (xml.clj:74)
cayenne.xml$process_xml.invoke (xml.clj:48)
cayenne.api.v1.feed$eval17380$fn__17381.invoke (feed.clj:132)
clojure.lang.MultiFn.invoke (MultiFn.java:233)
user$index_work_files$fn__24076.invoke (user.clj:116)
user$index_work_files.invokeStatic (user.clj:115)
user$index_work_files.invoke (user.clj:109)
user$index_work_files.invokeStatic (user.clj:112)
user$index_work_files.invoke (user.clj:109)
cayenne.works.retrieval_test$index_doc.invokeStatic (retrieval_test.clj:10)
cayenne.works.retrieval_test$index_doc.invoke (retrieval_test.clj:8)
cayenne.works.retrieval_test$fn__32476$fn__32477.invoke (retrieval_test.clj:51)
cayenne.works.retrieval_test$fn__32476.invokeStatic (retrieval_test.clj:15)
cayenne.works.retrieval_test/fn (retrieval_test.clj:13)
clojure.test$test_var$fn__7983.invoke (test.clj:716)
clojure.test$test_var.invokeStatic (test.clj:716)
clojure.test$test_var.invoke (test.clj:707)
clojure.test$test_vars$fn__8005$fn__8010.invoke (test.clj:734)
clojure.test$default_fixture.invokeStatic (test.clj:686)
clojure.test$default_fixture.invoke (test.clj:682)
clojure.test$test_vars$fn__8005.invoke (test.clj:734)
clojure.test$default_fixture.invokeStatic (test.clj:686)
clojure.test$default_fixture.invoke (test.clj:682)
clojure.test$test_vars.invokeStatic (test.clj:730)
clojure.test$test_all_vars.invokeStatic (test.clj:736)
clojure.test$test_ns.invokeStatic (test.clj:757)
clojure.test$test_ns.invoke (test.clj:742)
user$eval24220$fn__24331.invoke (form-init11988606910142817184.clj:1)
clojure.lang.AFn.applyToHelper (AFn.java:156)
clojure.lang.AFn.applyTo (AFn.java:144)
clojure.core$apply.invokeStatic (core.clj:648)
clojure.core$apply.invoke (core.clj:641)
leiningen.core.injected$compose_hooks$fn__24154.doInvoke (form-init11988606910142817184.clj:1)
clojure.lang.RestFn.applyTo (RestFn.java:137)
clojure.core$apply.invokeStatic (core.clj:646)
clojure.core$apply.invoke (core.clj:641)
leiningen.core.injected$run_hooks.invokeStatic (form-init11988606910142817184.clj:1)
leiningen.core.injected$run_hooks.invoke (form-init11988606910142817184.clj:1)
leiningen.core.injected$prepare_for_hooks$fn__24159$fn__24160.doInvoke (form-init11988606910142817184.clj:1)
clojure.lang.RestFn.applyTo (RestFn.java:137)
clojure.lang.AFunction$1.doInvoke (AFunction.java:29)
clojure.lang.RestFn.invoke (RestFn.java:408)
clojure.core$map$fn__4785.invoke (core.clj:2646)
clojure.lang.LazySeq.sval (LazySeq.java:40)
clojure.lang.LazySeq.seq (LazySeq.java:49)
clojure.lang.Cons.next (Cons.java:39)
clojure.lang.RT.boundedLength (RT.java:1749)
clojure.lang.RestFn.applyTo (RestFn.java:130)
clojure.core$apply.invokeStatic (core.clj:648)
clojure.test$run_tests.invokeStatic (test.clj:767)
clojure.test$run_tests.doInvoke (test.clj:767)
clojure.lang.RestFn.applyTo (RestFn.java:137)
clojure.core$apply.invokeStatic (core.clj:646)
clojure.core$apply.invoke (core.clj:641)
user$eval24220$fn__24343$fn__24394.invoke (form-init11988606910142817184.clj:1)
user$eval24220$fn__24343$fn__24344.invoke (form-init11988606910142817184.clj:1)
user$eval24220$fn__24343.invoke (form-init11988606910142817184.clj:1)
user$eval24220.invokeStatic (form-init11988606910142817184.clj:1)
user$eval24220.invoke (form-init11988606910142817184.clj:1)
clojure.lang.Compiler.eval (Compiler.java:6927)
clojure.lang.Compiler.eval (Compiler.java:6917)
clojure.lang.Compiler.load (Compiler.java:7379)
clojure.lang.Compiler.loadFile (Compiler.java:7317)
clojure.main$load_script.invokeStatic (main.clj:275)
clojure.main$init_opt.invokeStatic (main.clj:277)
clojure.main$init_opt.invoke (main.clj:277)
clojure.main$initialize.invokeStatic (main.clj:308)
clojure.main$null_opt.invokeStatic (main.clj:342)
clojure.main$null_opt.invoke (main.clj:339)
clojure.main$main.invokeStatic (main.clj:421)
clojure.main$main.doInvoke (main.clj:384)
clojure.lang.RestFn.invoke (RestFn.java:421)
clojure.lang.Var.invoke (Var.java:383)
clojure.lang.AFn.applyToHelper (AFn.java:156)
clojure.lang.Var.applyTo (Var.java:700)
clojure.main.main (main.java:37)
Caused by: java.lang.NullPointerException: null
at clojure.core$name.invokeStatic (core.clj:1546)
clojure.core$name.invoke (core.clj:1540)
cayenne.elastic.convert$item_type.invokeStatic (convert.clj:82)
cayenne.elastic.convert$item_type.invoke (convert.clj:81)
cayenne.elastic.convert$item__GT_es_doc.invokeStatic (convert.clj:416)
cayenne.elastic.convert$item__GT_es_doc.invoke (convert.clj:408)
cayenne.elastic.index$index_command.invokeStatic (index.clj:8)
cayenne.elastic.index$index_command.invoke (index.clj:7)
cayenne.elastic.index$index_item.invokeStatic (index.clj:15)
cayenne.elastic.index$index_item.invoke (index.clj:11)
cayenne.api.v1.feed$eval17380$fn__17381$f__17382.invoke (feed.clj:130)
cayenne.xml$process_xml$fn__9356.invoke (xml.clj:66)
cayenne.xml.proxy$nu.xom.NodeFactory$ff19274a.finishMakingElement (:-1)
nu.xom.XOMHandler.endElement (:-1)
org.apache.xerces.parsers.AbstractSAXParser.endElement (:-1)
org.apache.xerces.impl.XMLNamespaceBinder.handleEndElement (:-1)
org.apache.xerces.impl.XMLNamespaceBinder.endElement (:-1)
org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanEndElement (:-1)
org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch (:-1)
org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument (:-1)
org.apache.xerces.parsers.DTDConfiguration.parse (:-1)
org.apache.xerces.parsers.DTDConfiguration.parse (:-1)
org.apache.xerces.parsers.XMLParser.parse (:-1)
org.apache.xerces.parsers.AbstractSAXParser.parse (:-1)
nu.xom.Builder.build (:-1)
nu.xom.Builder.build (:-1)
cayenne.xml$process_xml.invokeStatic (xml.clj:74)
cayenne.xml$process_xml.invoke (xml.clj:48)
cayenne.api.v1.feed$eval17380$fn__17381.invoke (feed.clj:132)
clojure.lang.MultiFn.invoke (MultiFn.java:233)
user$index_work_files$fn__24076.invoke (user.clj:116)
user$index_work_files.invokeStatic (user.clj:115)
user$index_work_files.invoke (user.clj:109)
user$index_work_files.invokeStatic (user.clj:112)
user$index_work_files.invoke (user.clj:109)
cayenne.works.retrieval_test$index_doc.invokeStatic (retrieval_test.clj:10)
cayenne.works.retrieval_test$index_doc.invoke (retrieval_test.clj:8)
cayenne.works.retrieval_test$fn__32476$fn__32477.invoke (retrieval_test.clj:51)
cayenne.works.retrieval_test$fn__32476.invokeStatic (retrieval_test.clj:15)
cayenne.works.retrieval_test/fn (retrieval_test.clj:13)
clojure.test$test_var$fn__7983.invoke (test.clj:716)
clojure.test$test_var.invokeStatic (test.clj:716)
clojure.test$test_var.invoke (test.clj:707)
clojure.test$test_vars$fn__8005$fn__8010.invoke (test.clj:734)
clojure.test$default_fixture.invokeStatic (test.clj:686)
clojure.test$default_fixture.invoke (test.clj:682)
clojure.test$test_vars$fn__8005.invoke (test.clj:734)
clojure.test$default_fixture.invokeStatic (test.clj:686)
clojure.test$default_fixture.invoke (test.clj:682)
clojure.test$test_vars.invokeStatic (test.clj:730)
clojure.test$test_all_vars.invokeStatic (test.clj:736)
clojure.test$test_ns.invokeStatic (test.clj:757)
clojure.test$test_ns.invoke (test.clj:742)
user$eval24220$fn__24331.invoke (form-init11988606910142817184.clj:1)
clojure.lang.AFn.applyToHelper (AFn.java:156)
clojure.lang.AFn.applyTo (AFn.java:144)
clojure.core$apply.invokeStatic (core.clj:648)
clojure.core$apply.invoke (core.clj:641)
leiningen.core.injected$compose_hooks$fn__24154.doInvoke (form-init11988606910142817184.clj:1)
clojure.lang.RestFn.applyTo (RestFn.java:137)
clojure.core$apply.invokeStatic (core.clj:646)
clojure.core$apply.invoke (core.clj:641)
leiningen.core.injected$run_hooks.invokeStatic (form-init11988606910142817184.clj:1)
leiningen.core.injected$run_hooks.invoke (form-init11988606910142817184.clj:1)
leiningen.core.injected$prepare_for_hooks$fn__24159$fn__24160.doInvoke (form-init11988606910142817184.clj:1)
clojure.lang.RestFn.applyTo (RestFn.java:137)
clojure.lang.AFunction$1.doInvoke (AFunction.java:29)
clojure.lang.RestFn.invoke (RestFn.java:408)
clojure.core$map$fn__4785.invoke (core.clj:2646)
clojure.lang.LazySeq.sval (LazySeq.java:40)
clojure.lang.LazySeq.seq (LazySeq.java:49)
clojure.lang.Cons.next (Cons.java:39)
clojure.lang.RT.boundedLength (RT.java:1749)
clojure.lang.RestFn.applyTo (RestFn.java:130)
clojure.core$apply.invokeStatic (core.clj:648)
clojure.test$run_tests.invokeStatic (test.clj:767)
clojure.test$run_tests.doInvoke (test.clj:767)
clojure.lang.RestFn.applyTo (RestFn.java:137)
clojure.core$apply.invokeStatic (core.clj:646)
clojure.core$apply.invoke (core.clj:641)
user$eval24220$fn__24343$fn__24394.invoke (form-init11988606910142817184.clj:1)
user$eval24220$fn__24343$fn__24344.invoke (form-init11988606910142817184.clj:1)
user$eval24220$fn__24343.invoke (form-init11988606910142817184.clj:1)
user$eval24220.invokeStatic (form-init11988606910142817184.clj:1)
user$eval24220.invoke (form-init11988606910142817184.clj:1)
clojure.lang.Compiler.eval (Compiler.java:6927)
clojure.lang.Compiler.eval (Compiler.java:6917)
clojure.lang.Compiler.load (Compiler.java:7379)
clojure.lang.Compiler.loadFile (Compiler.java:7317)
clojure.main$load_script.invokeStatic (main.clj:275)
clojure.main$init_opt.invokeStatic (main.clj:277)
clojure.main$init_opt.invoke (main.clj:277)
clojure.main$initialize.invokeStatic (main.clj:308)
clojure.main$null_opt.invokeStatic (main.clj:342)
clojure.main$null_opt.invoke (main.clj:339)
clojure.main$main.invokeStatic (main.clj:421)
clojure.main$main.doInvoke (main.clj:384)
clojure.lang.RestFn.invoke (RestFn.java:421)
clojure.lang.Var.invoke (Var.java:383)
clojure.lang.AFn.applyToHelper (AFn.java:156)
clojure.lang.Var.applyTo (Var.java:700)
clojure.main.main (main.java:37)
Definition of ready
-
Product owner: @ppolischuk1 -
Tech lead: @dtkaczyk -
Service:: label applied -
Definition of done updated -
Weight applied
Definition of done
-
Unit tests identified, implemented, and passing -
Code reviewed -
Available via a staging URL -
Knowledge base reviewed and updated -
Public documentation reviewed and updated -
Consider any impacts to current or future architecture/infrastructure, and update specifications and documentation as needed -
Acceptance criteria met -
DOIs of type database_title
do not throw exceptions in ES REST API.type
may be returned as null, matching production behavior. -
Follow-up issue is filed into the Service::REST API backlog for implementing properly post-migration.
-
Notes
Edited by Patrick Polischuk