INSERT exception while polling feed 99
trace after turning on verbosity and disabling exception handling:
Record(id=99, language='en', url='https://correctiv.org/en/feed/', incomplete=False, salt_url=False, exclude=None, asy=True, cookies=None, tor=False)
2019-09-12T11:46:46Z - polling feed 99
to retrieve: https://correctiv.org/en/latest-stories/grand-theft-europe-en/2019/08/14/grand-theft-europe-the-news-game/
to retrieve: https://correctiv.org/en/latest-stories/grand-theft-europe-en/2019/05/13/how-the-uk-is-combating-vat-fraud/
to retrieve: https://correctiv.org/en/top-stories-en/2019/05/07/grand-theft-europe/
to retrieve: https://correctiv.org/en/latest-stories/2019/04/05/correctiv-publishes-graphic-novel-on-congo-massacres/
to retrieve: https://correctiv.org/en/top-stories-en/2019/08/15/frontex-watching-the-watchers/
to retrieve: https://correctiv.org/en/top-stories-en/2018/12/06/black-sites/
to retrieve: https://correctiv.org/en/top-stories-en/2018/12/11/journalism-is-not-a-crime/
to retrieve: https://correctiv.org/en/latest-stories/2018/10/21/cumex-files/
to retrieve: https://correctiv.org/en/top-stories-en/2018/10/18/the-cumex-files/
to retrieve: https://correctiv.org/en/videos-dokus-en/2018/10/18/the-cumex-files-trailer/
to retrieve: https://correctiv.org/en/top-stories-en/2018/05/03/in-the-hands-of-the-mafia/
to retrieve: https://correctiv.org/en/top-stories-en/2018/05/02/the-dark-side-of-moroccos-booming-agricultural-exports/
about to insert {'content': '...', 'url': 'https://correctiv.org/en/top-stories-en/2018/10/18/the-cumex-files/', 'title': 'The CumEx Files', 'stamp': '2018-10-18T04:07:37Z', 'author': 'Benjamin Schubert', 'feed_id': 99, 'language': 'en'}
about to insert {'content': '...', 'url': 'https://correctiv.org/en/top-stories-en/2018/05/03/in-the-hands-of-the-mafia/', 'title': 'In the hands of the mafia', 'stamp': '2018-05-03T08:22:00Z', 'author': 'Pascale Müller', 'feed_id': 99, 'language': 'en'}
about to insert {'content': '...', 'url': 'https://correctiv.org/en/top-stories-en/2018/12/06/black-sites/', 'title': 'Black Sites Turkey', 'stamp': '2018-12-06T15:51:54Z', 'author': 'Benjamin Schubert', 'feed_id': 99, 'language': 'en'}
about to insert {'content': '...', 'url': 'https://correctiv.org/en/top-stories-en/2018/12/11/journalism-is-not-a-crime/', 'title': 'Journalism is not a Crime', 'stamp': '2018-12-11T09:19:40Z', 'author': 'Benjamin Schubert', 'feed_id': 99, 'language': 'en'}
about to insert {'content': '...', 'url': 'https://correctiv.org/en/latest-stories/2019/04/05/correctiv-publishes-graphic-novel-on-congo-massacres/', 'title': 'CORRECTIV publishes graphic novel on Congo massacres', 'stamp': '2019-04-05T06:45:44Z', 'author': 'Frederik Richter', 'feed_id': 99, 'language': 'en'}
about to insert {'content': '...', 'url': 'https://correctiv.org/en/top-stories-en/2019/05/07/grand-theft-europe/', 'title': 'Grand Theft Europe', 'stamp': '2019-05-07T00:00:48Z', 'author': 'Benjamin Schubert', 'feed_id': 99, 'language': 'en'}
about to insert {'content': '...', 'url': 'https://correctiv.org/en/top-stories-en/2018/05/02/the-dark-side-of-moroccos-booming-agricultural-exports/', 'title': 'The dark side of Morocco’s booming agricultural exports', 'stamp': '2018-05-02T08:30:00Z', 'author': 'Pascale Müller', 'feed_id': 99, 'language': 'en'}
about to insert {'content': '...', 'url': 'https://correctiv.org/en/top-stories-en/2019/08/15/frontex-watching-the-watchers/', 'title': 'Frontex: watching the watchers', 'stamp': '2019-08-15T15:05:30Z', 'author': 'marius.wolf@correctiv.org', 'feed_id': 99, 'language': 'en'}
about to insert {'content': '...', 'url': 'https://correctiv.org/en/latest-stories/grand-theft-europe-en/2019/05/13/how-the-uk-is-combating-vat-fraud/', 'title': 'How the UK is combating VAT fraud', 'stamp': '2019-05-13T14:25:36Z', 'author': 'Benjamin Schubert', 'feed_id': 99, 'language': 'en'}
about to insert {'content': '...', 'url': 'https://correctiv.org/en/videos-dokus-en/2018/10/18/the-cumex-files-trailer/', 'title': 'Trailer #CumExFiles', 'stamp': '2018-10-18T03:58:36Z', 'author': 'Benjamin Schubert', 'feed_id': 99, 'language': 'en'}
about to insert {'content': '...', 'url': 'https://correctiv.org/en/latest-stories/grand-theft-europe-en/2019/08/14/grand-theft-europe-the-news-game/', 'title': 'Grand Theft Europe – The News Game', 'stamp': '2019-08-14T15:23:08Z', 'author': 'Benjamin Schubert', 'feed_id': 99, 'language': 'en'}
about to insert {'content': '...', 'url': 'https://correctiv.org/en/latest-stories/2018/10/21/cumex-files/', 'title': 'The CumEx Files', 'stamp': '2018-10-21T21:00:26Z', 'author': 'Benjamin Schubert', 'feed_id': 99, 'language': 'en'}
Traceback (most recent call last):
File "py/poll.py", line 33, in <module>
p.poll()
File "/srv/calo.news/py/poller.py", line 255, in poll
loop.run_until_complete(main(loop, entries, self.args, self.retrieved, self.failed, self.verbose))
File "/usr/lib/python3.5/asyncio/base_events.py", line 466, in run_until_complete
return future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
result = coro.throw(exc)
File "/srv/calo.news/py/poller.py", line 194, in main
*[retrieve_asy(client, e, args, retrieved, failed, verbose) for e in entries]
File "/usr/lib/python3.5/asyncio/futures.py", line 380, in __iter__
yield self # This tells Task to wait for completion.
File "/usr/lib/python3.5/asyncio/tasks.py", line 304, in _wakeup
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
result = coro.send(None)
File "/srv/calo.news/py/poller.py", line 179, in retrieve_asy
id = store_article(author=e.get('author', ''), title=e['title'], url=url, content=content, feed_id=args.id, language=args.language, stamp=e['stamp'], salt_url=args.salt_url)
File "/srv/calo.news/py/poller.py", line 90, in store_article
RETURNING id""".format(title_field, content_field), data)
psycopg2.IntegrityError: duplicate key value violates unique constraint "url_unique"
DETAIL: Key (url)=(https://correctiv.org/en/top-stories-en/2018/12/06/black-sites/) already exists.