Intermittent timeouts using the REST API
A member has contacted us regarding some intermittent timeouts (cursor expirations) that are happening when using the REST API. Decreasing the read timeout and even reducing the rows to 500 to try and prevent this has not helped yet.
Here is the error shown in the logs:
at
sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:565)
at
sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:609)
at
sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:696)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at
sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3393)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
It only happened once in 38 API calls and the largest results return was only 13k.
They are a plus member also using mailto and they are using the java code snippet below:
System.out.println("calling " + urlQueryString);
String json = null;
StringBuilder sb = new StringBuilder();
try {
URL url = new URL(urlQueryString);
HttpsURLConnection connection = (HttpsURLConnection)
url.openConnection();
connection.setDoOutput(true);
connection.setInstanceFollowRedirects(false);
connection.setRequestMethod("GET");
connection.setRequestProperty("Content-Type",
"application/json");
connection.setRequestProperty("charset", "utf-8");
connection.setRequestProperty("User-Agent", "Java Client;
mailto:literature@ebi.ac.uk");
connection.setRequestProperty("Authorization", AUTH_TOKEN);
connection.setRequestProperty("Crossref-Plus-API-Token", API_TOKEN);
connection.setConnectTimeout(90000);
connection.setReadTimeout(900000);
connection.connect();
Map<String, List<String>> map = connection.getHeaderFields();
System.out.println(map);
if (map.keySet().contains("x-rate-limit-limit")) {
REQS_LIMIT =
Integer.parseInt(map.get("x-rate-limit-limit").get(0));
}
if (map.keySet().contains("x-rate-limit-interval")) {
String intervalStr =
map.get("x-rate-limit-interval").get(0);
REQS_INTERVAL =
1000*Integer.parseInt(intervalStr.substring(0,intervalStr.indexOf("s")));
}
BufferedReader in = new BufferedReader(new
InputStreamReader(url.openStream(), "UTF-8"));
String readLine;
while ((readLine = in.readLine()) != null) {
sb.append(readLine);
}
json = sb.toString();
} catch (Exception ex) {
ex.printStackTrace();
System.out.println("Error during calling " + urlQueryString);
return null;
}
return json;
}
Note that the urlQueryString variable can be something like this:
Zendesk ticket #394371
How urgent
Definition of ready
-
Product owner: -
Tech lead: -
Service:: label applied -
Definition of done updated -
Acceptance testing plan: -
Weight applied
Definition of done
-
Unit tests identified, implemented, and passing -
Code reviewed -
Available for acceptance testing via a staging URL, or otherwise -
Consider any impacts to current or future architecture/infrastructure, and update specifications and documentation as needed -
Knowledge base reviewed and updated -
Public documentation reviewed and updated -
Acceptance criteria met -
AC 1 -
AC 2
-
-
Acceptance testing passed -
Deployed to production
Notes
Edited by Paul Davis