Skip to content

Username mentions don't work in Docker development environment

The generator is here: https://gitlab.com/tildes-community/tildes-cf/-/blob/master/tildes/consumers/comment_user_mentions_generator.py

A systemd service is generated using consumer.service.jinja2 template and placed at /etc/systemd/system/consumer-comment_user_mentions_generator.service

From $ vagrant ssh session:

$ systemctl status consumer-comment_user_mentions_generator.service
consumer-comment_user_mentions_generator.service - Comment User Mentions Generator (Queue Consumer)
    Loaded: loaded (/etc/systemd/system/consumer-comment_user_mentions_generator.service, enabled)
    Active: inactive (dead)

Got redis.exceptions.ConnectionError: Connection closed by server.

Expand for full log
$ cat '/var/log/journal/consumer-comment_user_mentions_generator.service.log'
Traceback (most recent call last):
  File "/opt/tildes/consumers/comment_user_mentions_generator.py", line 47, in <module>
    CommentUserMentionGenerator(
  File "/opt/tildes/tildes/lib/event_stream.py", line 137, in consume_streams
    messages = self._get_messages(pending=False)
  File "/opt/tildes/tildes/lib/event_stream.py", line 226, in _get_messages
    response = self.redis.xreadgroup(
  File "/opt/venvs/tildes/lib/python3.9/site-packages/redis/client.py", line 2607, in xreadgroup
    return self.execute_command('XREADGROUP', *pieces)
  File "/opt/venvs/tildes/lib/python3.9/site-packages/redis/client.py", line 901, in execute_command
    return self.parse_response(conn, command_name, **options)
  File "/opt/venvs/tildes/lib/python3.9/site-packages/redis/client.py", line 915, in parse_response
    response = connection.read_response()
  File "/opt/venvs/tildes/lib/python3.9/site-packages/redis/connection.py", line 739, in read_response
    response = self._parser.read_response()
  File "/opt/venvs/tildes/lib/python3.9/site-packages/redis/connection.py", line 324, in read_response
    raw = self._buffer.readline()
  File "/opt/venvs/tildes/lib/python3.9/site-packages/redis/connection.py", line 256, in readline
    self._read_from_socket()
  File "/opt/venvs/tildes/lib/python3.9/site-packages/redis/connection.py", line 201, in _read_from_socket
    raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server.

Appended to the log after restarting the Docker container:

psycopg2.OperationalError: connection to server on socket "/var/run/postgresql/.s.PGSQL.6432" failed: Connection refused
        Is the server running locally and accepting connections on that socket?

The above exception was the direct cause of the following exception:

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) connection to server on socket "/var/run/postgresql/.s.PGSQL.6432" failed: Connection refused
        Is the server running locally and accepting connections on that socket?

(Background on this error at: http://sqlalche.me/e/13/e3q8)
Expand for full log
Traceback (most recent call last):
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
    return fn()
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 364, in connect
    return _ConnectionFairy._checkout(self)
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
    rec = pool._do_get()
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 241, in _do_get
    return self._create_connection()
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
    return _ConnectionRecord(self)
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 440, in __init__
    self.__connect(first_connect_check=True)
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
    pool.logger.debug("Error on connect(): %s", e)
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.raise_(
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
    connection = pool._invoke_creator(self)
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    return dialect.connect(*cargs, **cparams)
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 508, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/opt/venvs/tildes/lib/python3.9/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server on socket "/var/run/postgresql/.s.PGSQL.6432" failed: Connection refused
        Is the server running locally and accepting connections on that socket?


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/tildes/consumers/comment_user_mentions_generator.py", line 47, in <module>
    CommentUserMentionGenerator(
  File "/opt/tildes/tildes/lib/event_stream.py", line 140, in consume_streams
    self.process_message(message)
  File "/opt/tildes/consumers/comment_user_mentions_generator.py", line 18, in process_message
    self.db_session.query(Comment)
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3490, in one
    ret = self.one_or_none()
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3459, in one_or_none
    ret = list(self)
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3535, in __iter__
    return self._execute_and_instances(context)
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3556, in _execute_and_instances
    conn = self._get_bind_args(
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3571, in _get_bind_args
    return fn(
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3550, in _connection_from_session
    conn = self.session.connection(**kw)
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1142, in connection
    return self._connection_for_bind(
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1150, in _connection_for_bind
    return self.transaction._connection_for_bind(
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 433, in _connection_for_bind
    conn = bind._contextual_connect()
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2302, in _contextual_connect
    self._wrap_pool_connect(self.pool.connect, None),
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2339, in _wrap_pool_connect
    Connection._handle_dbapi_exception_noconnection(
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1583, in _handle_dbapi_exception_noconnection
    util.raise_(
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
    return fn()
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 364, in connect
    return _ConnectionFairy._checkout(self)
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
    rec = pool._do_get()
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 241, in _do_get
    return self._create_connection()
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
    return _ConnectionRecord(self)
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 440, in __init__
    self.__connect(first_connect_check=True)
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
    pool.logger.debug("Error on connect(): %s", e)
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.raise_(
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
    connection = pool._invoke_creator(self)
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    return dialect.connect(*cargs, **cparams)
  File "/opt/venvs/tildes/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 508, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/opt/venvs/tildes/lib/python3.9/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) connection to server on socket "/var/run/postgresql/.s.PGSQL.6432" failed: Connection refused
        Is the server running locally and accepting connections on that socket?

(Background on this error at: http://sqlalche.me/e/13/e3q8)

PostgreSQL is running though:

$ systemctl status postgresql@15-main
postgresql@15-main.service - PostgreSQL Cluster 15-main
    Loaded: loaded (/usr/lib/systemd/system/postgresql@.service, enabled)
    Active: active (running)

Tried starting the systemd service (using systemctl3.py):

$ sudo systemctl start consumer-comment_user_mentions_generator.service
$ systemctl status consumer-comment_user_mentions_generator.service
consumer-comment_user_mentions_generator.service - Comment User Mentions Generator (Queue Consumer)
    Loaded: loaded (/etc/systemd/system/consumer-comment_user_mentions_generator.service, enabled)
    Active: failed (failed)

Tried manually running the script:

$ cd /opt/tildes/consumers
$ INI_FILE=/opt/tildes/development.ini /opt/venvs/tildes/bin/python comment_user_mentions_generator.py
Traceback (most recent call last):
  File "/opt/tildes/consumers/comment_user_mentions_generator.py", line 47, in <module>
    CommentUserMentionGenerator(
  File "/opt/tildes/tildes/lib/event_stream.py", line 94, in __init__
    self._init_metrics()
  File "/opt/tildes/tildes/lib/event_stream.py", line 122, in _init_metrics
    start_http_server(self.METRICS_PORT, registry=self.metrics_registry)
  File "/opt/venvs/tildes/lib/python3.9/site-packages/prometheus_client/exposition.py", line 149, in start_wsgi_server
    httpd = make_server(addr, port, app, ThreadingWSGIServer, handler_class=_SilentHandler)
  File "/usr/local/lib/python3.9/wsgiref/simple_server.py", line 154, in make_server
    server = server_class((host, port), handler_class)
  File "/usr/local/lib/python3.9/socketserver.py", line 452, in __init__
    self.server_bind()
  File "/usr/local/lib/python3.9/wsgiref/simple_server.py", line 50, in server_bind
    HTTPServer.server_bind(self)
  File "/usr/local/lib/python3.9/http/server.py", line 137, in server_bind
    socketserver.TCPServer.server_bind(self)
  File "/usr/local/lib/python3.9/socketserver.py", line 466, in server_bind
    self.socket.bind(self.server_address)
OSError: [Errno 98] Address already in use
Edited by talklittle