apscheduler.executors.default - ERROR - Job "get_jobs [...]" raised an exception
While looking at my logs for something else, I spotted the following exception. Now I've extracted the relevant part to raise this issue it looks like it's failed to connect to network.satnogs.org
so perhaps relatively innocuous, although it did happen again at 15:03:27. I assume it worked successfully every hour or so in-between but didn't log success? Although maybe it would have missed any scheds in the hour immediately after the failure?
Apr 20 09:10:27 pinog satnogs-client[330]: apscheduler.executors.default - ERROR - Job "get_jobs (trigger: interval[0:01:00], next run at: 2020-04-20 09:10:42 UTC)" raised an exception
Apr 20 09:10:27 pinog satnogs-client[330]: Traceback (most recent call last):
Apr 20 09:10:27 pinog satnogs-client[330]: File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 384, in _make_request
Apr 20 09:10:27 pinog satnogs-client[330]: six.raise_from(e, None)
Apr 20 09:10:27 pinog satnogs-client[330]: File "<string>", line 3, in raise_from
Apr 20 09:10:27 pinog satnogs-client[330]: File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 380, in _make_request
Apr 20 09:10:27 pinog satnogs-client[330]: httplib_response = conn.getresponse()
Apr 20 09:10:27 pinog satnogs-client[330]: File "/usr/lib/python3.7/http/client.py", line 1336, in getresponse
Apr 20 09:10:27 pinog satnogs-client[330]: response.begin()
Apr 20 09:10:27 pinog satnogs-client[330]: File "/usr/lib/python3.7/http/client.py", line 306, in begin
Apr 20 09:10:27 pinog satnogs-client[330]: version, status, reason = self._read_status()
Apr 20 09:10:27 pinog satnogs-client[330]: File "/usr/lib/python3.7/http/client.py", line 267, in _read_status
Apr 20 09:10:27 pinog satnogs-client[330]: line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
Apr 20 09:10:27 pinog satnogs-client[330]: File "/usr/lib/python3.7/socket.py", line 589, in readinto
Apr 20 09:10:27 pinog satnogs-client[330]: return self._sock.recv_into(b)
Apr 20 09:10:27 pinog satnogs-client[330]: File "/usr/lib/python3.7/ssl.py", line 1052, in recv_into
Apr 20 09:10:27 pinog satnogs-client[330]: return self.read(nbytes, buffer)
Apr 20 09:10:27 pinog satnogs-client[330]: File "/usr/lib/python3.7/ssl.py", line 911, in read
Apr 20 09:10:27 pinog satnogs-client[330]: return self._sslobj.read(len, buffer)
Apr 20 09:10:27 pinog satnogs-client[330]: socket.timeout: The read operation timed out
Apr 20 09:10:27 pinog satnogs-client[330]: During handling of the above exception, another exception occurred:
Apr 20 09:10:27 pinog satnogs-client[330]: Traceback (most recent call last):
Apr 20 09:10:27 pinog satnogs-client[330]: File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send
Apr 20 09:10:27 pinog satnogs-client[330]: timeout=timeout
Apr 20 09:10:27 pinog satnogs-client[330]: File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
Apr 20 09:10:27 pinog satnogs-client[330]: _stacktrace=sys.exc_info()[2])
Apr 20 09:10:27 pinog satnogs-client[330]: File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 367, in increment
Apr 20 09:10:27 pinog satnogs-client[330]: raise six.reraise(type(error), error, _stacktrace)
Apr 20 09:10:27 pinog satnogs-client[330]: File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
Apr 20 09:10:27 pinog satnogs-client[330]: raise value
Apr 20 09:10:27 pinog satnogs-client[330]: File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
Apr 20 09:10:27 pinog satnogs-client[330]: chunked=chunked)
Apr 20 09:10:27 pinog satnogs-client[330]: File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 386, in _make_request
Apr 20 09:10:27 pinog satnogs-client[330]: self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
Apr 20 09:10:27 pinog satnogs-client[330]: File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 306, in _raise_timeout
Apr 20 09:10:27 pinog satnogs-client[330]: raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
Apr 20 09:10:27 pinog satnogs-client[330]: urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='network.satnogs.org', port=443): Read timed out. (read timeout=45)
Apr 20 09:10:27 pinog satnogs-client[330]: During handling of the above exception, another exception occurred:
Apr 20 09:10:27 pinog satnogs-client[330]: Traceback (most recent call last):
Apr 20 09:10:27 pinog satnogs-client[330]: File "/usr/lib/python3/dist-packages/apscheduler/executors/base.py", line 125, in run_job
Apr 20 09:10:27 pinog satnogs-client[330]: retval = job.func(*job.args, **job.kwargs)
Apr 20 09:10:27 pinog satnogs-client[330]: File "/var/lib/satnogs/lib/python3.7/site-packages/satnogsclient/scheduler/tasks.py", line 187, in get_jobs
Apr 20 09:10:27 pinog satnogs-client[330]: timeout=45)
Apr 20 09:10:27 pinog satnogs-client[330]: File "/usr/lib/python3/dist-packages/requests/api.py", line 75, in get
Apr 20 09:10:27 pinog satnogs-client[330]: return request('get', url, params=params, **kwargs)
Apr 20 09:10:27 pinog satnogs-client[330]: File "/usr/lib/python3/dist-packages/requests/api.py", line 60, in request
Apr 20 09:10:27 pinog satnogs-client[330]: return session.request(method=method, url=url, **kwargs)
Apr 20 09:10:27 pinog satnogs-client[330]: File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
Apr 20 09:10:27 pinog satnogs-client[330]: resp = self.send(prep, **send_kwargs)
Apr 20 09:10:27 pinog satnogs-client[330]: File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
Apr 20 09:10:27 pinog satnogs-client[330]: r = adapter.send(request, **kwargs)
Apr 20 09:10:27 pinog satnogs-client[330]: File "/usr/lib/python3/dist-packages/requests/adapters.py", line 529, in send
Apr 20 09:10:27 pinog satnogs-client[330]: raise ReadTimeout(e, request=request)
Apr 20 09:10:27 pinog satnogs-client[330]: requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='network.satnogs.org', port=443): Read timed out. (read timeout=45)
Could this error be handled better than through the exceptions and associated verbiage? Possibly with a retry as whatever was causing it appears to be transient.
Complete logs from today attached: satnogs-client.service.log. I don't have any earlier logs so can't say if this is just today or repetitive. (I'm assuming they're only stored in RAM and I rebooted the station this morning to switch cables to charge my phone!)