Problems with license db extraction
We're unable to read from license_db
2019-12-04 08:10:54,086] INFO - [2019-12-04 08:10:54,086] INFO - b'psycopg2.OperationalError: could not connect to server: Connection timed out\n'
[2019-12-04 08:10:54,086] INFO - [2019-12-04 08:10:54,086] INFO - b'\tIs the server running on host "10.138.16.11" and accepting\n'
[2019-12-04 08:10:54,087] INFO - [2019-12-04 08:10:54,087] INFO - b'\tTCP/IP connections on port 5432?\n'
Full log:
*** Reading local file: /usr/local/airflow/logs/license_db_extract/license-db-incremental/2019-12-04T00:00:00+00:00/2.log
[2019-12-04 08:06:17,093] INFO - Dependencies all met for <TaskInstance: license_db_extract.license-db-incremental 2019-12-04T00:00:00+00:00 [queued]>
[2019-12-04 08:06:17,102] INFO - Dependencies all met for <TaskInstance: license_db_extract.license-db-incremental 2019-12-04T00:00:00+00:00 [queued]>
[2019-12-04 08:06:17,102] INFO -
--------------------------------------------------------------------------------
[2019-12-04 08:06:17,102] INFO - Starting attempt 2 of 2
[2019-12-04 08:06:17,103] INFO -
--------------------------------------------------------------------------------
[2019-12-04 08:06:17,129] INFO - Executing <Task(KubernetesPodOperator): license-db-incremental> on 2019-12-04T00:00:00+00:00
[2019-12-04 08:06:17,130] INFO - Running: ['airflow', 'run', 'license_db_extract', 'license-db-incremental', '2019-12-04T00:00:00+00:00', '--job_id', '22469', '--raw', '-sd', 'DAGS_FOLDER/extract/gitlab_dbs.py', '--cfg_path', '/tmp/tmpy57gwtvp']
[2019-12-04 08:06:17,774] INFO - Job 22469: Subtask license-db-incremental [2019-12-04 08:06:17,774] INFO - settings.configure_orm(): Using pool settings. pool_size=6, pool_recycle=1800, pid=1892444
[2019-12-04 08:06:17,932] INFO - Job 22469: Subtask license-db-incremental [2019-12-04 08:06:17,932] INFO - Using executor LocalExecutor
[2019-12-04 08:06:18,227] INFO - Job 22469: Subtask license-db-incremental [2019-12-04 08:06:18,226] INFO - Filling up the DagBag from /usr/local/airflow/analytics/dags/extract/gitlab_dbs.py
[2019-12-04 08:06:18,618] INFO - Job 22469: Subtask license-db-incremental [2019-12-04 08:06:18,618] INFO - Running <TaskInstance: license_db_extract.license-db-incremental 2019-12-04T00:00:00+00:00 [running]> on host airflow-deployment-b666dd78d-dzz4r
[2019-12-04 08:06:18,890] INFO - [2019-12-04 08:06:18,890] INFO - Event: license-db-incremental-df1efe5e had an event of type Pending
[2019-12-04 08:06:19,897] INFO - [2019-12-04 08:06:19,897] INFO - Event: license-db-incremental-df1efe5e had an event of type Pending
[2019-12-04 08:06:20,904] INFO - [2019-12-04 08:06:20,904] INFO - Event: license-db-incremental-df1efe5e had an event of type Pending
[2019-12-04 08:06:21,912] INFO - [2019-12-04 08:06:21,912] INFO - Event: license-db-incremental-df1efe5e had an event of type Pending
[2019-12-04 08:06:22,920] INFO - [2019-12-04 08:06:22,920] INFO - Event: license-db-incremental-df1efe5e had an event of type Pending
[2019-12-04 08:06:23,927] INFO - [2019-12-04 08:06:23,927] INFO - Event: license-db-incremental-df1efe5e had an event of type Pending
[2019-12-04 08:06:24,933] INFO - [2019-12-04 08:06:24,933] INFO - Event: license-db-incremental-df1efe5e had an event of type Pending
[2019-12-04 08:06:25,940] INFO - [2019-12-04 08:06:25,940] INFO - Event: license-db-incremental-df1efe5e had an event of type Pending
[2019-12-04 08:06:26,948] INFO - [2019-12-04 08:06:26,948] INFO - Event: license-db-incremental-df1efe5e had an event of type Pending
[2019-12-04 08:06:27,957] INFO - [2019-12-04 08:06:27,956] INFO - Event: license-db-incremental-df1efe5e had an event of type Pending
[2019-12-04 08:06:28,964] INFO - [2019-12-04 08:06:28,964] INFO - Event: license-db-incremental-df1efe5e had an event of type Running
[2019-12-04 08:06:29,009] INFO - [2019-12-04 08:06:29,009] INFO - b"Cloning into 'analytics'...\n"
[2019-12-04 08:06:30,579] INFO - [2019-12-04 08:06:30,579] INFO - b'INFO:root:Reading manifest at location: ../manifests/license_db_manifest.yaml\n'
[2019-12-04 08:06:30,589] INFO - [2019-12-04 08:06:30,589] INFO - b'INFO:root:Creating database engines...\n'
[2019-12-04 08:06:30,616] INFO - [2019-12-04 08:06:30,616] INFO - b'INFO:root:Engine(postgresql://postgres:***@10.138.16.11:5432/license_gitlab_com_production)\n'
[2019-12-04 08:06:30,718] INFO - [2019-12-04 08:06:30,718] INFO - b'INFO:root:Engine(snowflake://airflow:***@gitlab/RAW/tap_postgres?role=LOADER&warehouse=LOADING)\n'
[2019-12-04 08:06:30,718] INFO - [2019-12-04 08:06:30,718] INFO - b'INFO:root:Processing Table: add_ons\n'
[2019-12-04 08:06:30,723] INFO - [2019-12-04 08:06:30,723] INFO - b'INFO:botocore.vendored.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): gitlab.snowflakecomputing.com\n'
[2019-12-04 08:06:30,911] INFO - [2019-12-04 08:06:30,911] INFO - b'INFO:botocore.vendored.requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): ocsp.snowflakecomputing.com\n'
[2019-12-04 08:10:54,065] INFO - [2019-12-04 08:10:54,065] INFO - b'Traceback (most recent call last):\n'
[2019-12-04 08:10:54,066] INFO - [2019-12-04 08:10:54,066] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect\n'
[2019-12-04 08:10:54,067] INFO - [2019-12-04 08:10:54,067] INFO - b' return fn()\n'
[2019-12-04 08:10:54,068] INFO - [2019-12-04 08:10:54,067] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool.py", line 400, in connect\n'
[2019-12-04 08:10:54,068] INFO - [2019-12-04 08:10:54,068] INFO - b' return _ConnectionFairy._checkout(self)\n'
[2019-12-04 08:10:54,069] INFO - [2019-12-04 08:10:54,069] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool.py", line 788, in _checkout\n'
[2019-12-04 08:10:54,070] INFO - [2019-12-04 08:10:54,070] INFO - b' fairy = _ConnectionRecord.checkout(pool)\n'
[2019-12-04 08:10:54,071] INFO - [2019-12-04 08:10:54,071] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool.py", line 529, in checkout\n'
[2019-12-04 08:10:54,071] INFO - [2019-12-04 08:10:54,071] INFO - b' rec = pool._do_get()\n'
[2019-12-04 08:10:54,072] INFO - [2019-12-04 08:10:54,072] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool.py", line 1193, in _do_get\n'
[2019-12-04 08:10:54,072] INFO - [2019-12-04 08:10:54,072] INFO - b' self._dec_overflow()\n'
[2019-12-04 08:10:54,073] INFO - [2019-12-04 08:10:54,073] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__\n'
[2019-12-04 08:10:54,074] INFO - [2019-12-04 08:10:54,074] INFO - b' compat.reraise(exc_type, exc_value, exc_tb)\n'
[2019-12-04 08:10:54,075] INFO - [2019-12-04 08:10:54,075] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 249, in reraise\n'
[2019-12-04 08:10:54,075] INFO - [2019-12-04 08:10:54,075] INFO - b' raise value\n'
[2019-12-04 08:10:54,076] INFO - [2019-12-04 08:10:54,076] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool.py", line 1190, in _do_get\n'
[2019-12-04 08:10:54,076] INFO - [2019-12-04 08:10:54,076] INFO - b' return self._create_connection()\n'
[2019-12-04 08:10:54,077] INFO - [2019-12-04 08:10:54,077] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool.py", line 347, in _create_connection\n'
[2019-12-04 08:10:54,078] INFO - [2019-12-04 08:10:54,078] INFO - b' return _ConnectionRecord(self)\n'
[2019-12-04 08:10:54,079] INFO - [2019-12-04 08:10:54,079] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool.py", line 474, in __init__\n'
[2019-12-04 08:10:54,079] INFO - [2019-12-04 08:10:54,079] INFO - b' self.__connect(first_connect_check=True)\n'
[2019-12-04 08:10:54,080] INFO - [2019-12-04 08:10:54,080] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool.py", line 671, in __connect\n'
[2019-12-04 08:10:54,080] INFO - [2019-12-04 08:10:54,080] INFO - b' connection = pool._invoke_creator(self)\n'
[2019-12-04 08:10:54,081] INFO - [2019-12-04 08:10:54,081] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/strategies.py", line 106, in connect\n'
[2019-12-04 08:10:54,082] INFO - [2019-12-04 08:10:54,082] INFO - b' return dialect.connect(*cargs, **cparams)\n'
[2019-12-04 08:10:54,083] INFO - [2019-12-04 08:10:54,083] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 412, in connect\n'
[2019-12-04 08:10:54,083] INFO - [2019-12-04 08:10:54,083] INFO - b' return self.dbapi.connect(*cargs, **cparams)\n'
[2019-12-04 08:10:54,084] INFO - [2019-12-04 08:10:54,084] INFO - b' File "/usr/local/lib/python3.7/site-packages/psycopg2/__init__.py", line 130, in connect\n'
[2019-12-04 08:10:54,085] INFO - [2019-12-04 08:10:54,085] INFO - b' conn = _connect(dsn, connection_factory=connection_factory, **kwasync)\n'
[2019-12-04 08:10:54,086] INFO - [2019-12-04 08:10:54,086] INFO - b'psycopg2.OperationalError: could not connect to server: Connection timed out\n'
[2019-12-04 08:10:54,086] INFO - [2019-12-04 08:10:54,086] INFO - b'\tIs the server running on host "10.138.16.11" and accepting\n'
[2019-12-04 08:10:54,087] INFO - [2019-12-04 08:10:54,087] INFO - b'\tTCP/IP connections on port 5432?\n'
[2019-12-04 08:10:54,087] INFO - [2019-12-04 08:10:54,087] INFO - b'\n'
[2019-12-04 08:10:54,087] INFO - [2019-12-04 08:10:54,087] INFO - b'\n'
[2019-12-04 08:10:54,088] INFO - [2019-12-04 08:10:54,088] INFO - b'The above exception was the direct cause of the following exception:\n'
[2019-12-04 08:10:54,088] INFO - [2019-12-04 08:10:54,088] INFO - b'\n'
[2019-12-04 08:10:54,088] INFO - [2019-12-04 08:10:54,088] INFO - b'Traceback (most recent call last):\n'
[2019-12-04 08:10:54,089] INFO - [2019-12-04 08:10:54,089] INFO - b' File "main.py", line 341, in <module>\n'
[2019-12-04 08:10:54,089] INFO - [2019-12-04 08:10:54,089] INFO - b' Fire({"tap": main})\n'
[2019-12-04 08:10:54,090] INFO - [2019-12-04 08:10:54,090] INFO - b' File "/usr/local/lib/python3.7/site-packages/fire/core.py", line 127, in Fire\n'
[2019-12-04 08:10:54,091] INFO - [2019-12-04 08:10:54,091] INFO - b' component_trace = _Fire(component, args, context, name)\n'
[2019-12-04 08:10:54,092] INFO - [2019-12-04 08:10:54,092] INFO - b' File "/usr/local/lib/python3.7/site-packages/fire/core.py", line 366, in _Fire\n'
[2019-12-04 08:10:54,092] INFO - [2019-12-04 08:10:54,092] INFO - b' component, remaining_args)\n'
[2019-12-04 08:10:54,093] INFO - [2019-12-04 08:10:54,093] INFO - b' File "/usr/local/lib/python3.7/site-packages/fire/core.py", line 542, in _CallCallable\n'
[2019-12-04 08:10:54,093] INFO - [2019-12-04 08:10:54,093] INFO - b' result = fn(*varargs, **kwargs)\n'
[2019-12-04 08:10:54,094] INFO - [2019-12-04 08:10:54,094] INFO - b' File "main.py", line 319, in main\n'
[2019-12-04 08:10:54,094] INFO - [2019-12-04 08:10:54,094] INFO - b' table_name,\n'
[2019-12-04 08:10:54,095] INFO - [2019-12-04 08:10:54,095] INFO - b' File "/analytics/extract/postgres_pipeline/postgres_pipeline/utils.py", line 260, in check_if_schema_changed\n'
[2019-12-04 08:10:54,096] INFO - [2019-12-04 08:10:54,096] INFO - b' con=source_engine,\n'
[2019-12-04 08:10:54,097] INFO - [2019-12-04 08:10:54,097] INFO - b' File "/usr/local/lib/python3.7/site-packages/pandas/io/sql.py", line 397, in read_sql\n'
[2019-12-04 08:10:54,097] INFO - [2019-12-04 08:10:54,097] INFO - b' chunksize=chunksize)\n'
[2019-12-04 08:10:54,098] INFO - [2019-12-04 08:10:54,098] INFO - b' File "/usr/local/lib/python3.7/site-packages/pandas/io/sql.py", line 1099, in read_query\n'
[2019-12-04 08:10:54,099] INFO - [2019-12-04 08:10:54,099] INFO - b' result = self.execute(*args)\n'
[2019-12-04 08:10:54,099] INFO - [2019-12-04 08:10:54,099] INFO - b' File "/usr/local/lib/python3.7/site-packages/pandas/io/sql.py", line 990, in execute\n'
[2019-12-04 08:10:54,100] INFO - [2019-12-04 08:10:54,100] INFO - b' return self.connectable.execute(*args, **kwargs)\n'
[2019-12-04 08:10:54,101] INFO - [2019-12-04 08:10:54,101] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2074, in execute\n'
[2019-12-04 08:10:54,102] INFO - [2019-12-04 08:10:54,102] INFO - b' connection = self.contextual_connect(close_with_result=True)\n'
[2019-12-04 08:10:54,103] INFO - [2019-12-04 08:10:54,103] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect\n'
[2019-12-04 08:10:54,103] INFO - [2019-12-04 08:10:54,103] INFO - b' self._wrap_pool_connect(self.pool.connect, None),\n'
[2019-12-04 08:10:54,104] INFO - [2019-12-04 08:10:54,104] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect\n'
[2019-12-04 08:10:54,105] INFO - [2019-12-04 08:10:54,105] INFO - b' e, dialect, self)\n'
[2019-12-04 08:10:54,106] INFO - [2019-12-04 08:10:54,106] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception_noconnection\n'
[2019-12-04 08:10:54,106] INFO - [2019-12-04 08:10:54,106] INFO - b' exc_info\n'
[2019-12-04 08:10:54,107] INFO - [2019-12-04 08:10:54,107] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause\n'
[2019-12-04 08:10:54,108] INFO - [2019-12-04 08:10:54,108] INFO - b' reraise(type(exception), exception, tb=exc_tb, cause=cause)\n'
[2019-12-04 08:10:54,109] INFO - [2019-12-04 08:10:54,109] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 248, in reraise\n'
[2019-12-04 08:10:54,109] INFO - [2019-12-04 08:10:54,109] INFO - b' raise value.with_traceback(tb)\n'
[2019-12-04 08:10:54,111] INFO - [2019-12-04 08:10:54,111] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect\n'
[2019-12-04 08:10:54,111] INFO - [2019-12-04 08:10:54,111] INFO - b' return fn()\n'
[2019-12-04 08:10:54,113] INFO - [2019-12-04 08:10:54,112] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool.py", line 400, in connect\n'
[2019-12-04 08:10:54,113] INFO - [2019-12-04 08:10:54,113] INFO - b' return _ConnectionFairy._checkout(self)\n'
[2019-12-04 08:10:54,114] INFO - [2019-12-04 08:10:54,114] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool.py", line 788, in _checkout\n'
[2019-12-04 08:10:54,114] INFO - [2019-12-04 08:10:54,114] INFO - b' fairy = _ConnectionRecord.checkout(pool)\n'
[2019-12-04 08:10:54,115] INFO - [2019-12-04 08:10:54,115] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool.py", line 529, in checkout\n'
[2019-12-04 08:10:54,116] INFO - [2019-12-04 08:10:54,116] INFO - b' rec = pool._do_get()\n'
[2019-12-04 08:10:54,117] INFO - [2019-12-04 08:10:54,117] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool.py", line 1193, in _do_get\n'
[2019-12-04 08:10:54,117] INFO - [2019-12-04 08:10:54,117] INFO - b' self._dec_overflow()\n'
[2019-12-04 08:10:54,118] INFO - [2019-12-04 08:10:54,118] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__\n'
[2019-12-04 08:10:54,119] INFO - [2019-12-04 08:10:54,119] INFO - b' compat.reraise(exc_type, exc_value, exc_tb)\n'
[2019-12-04 08:10:54,120] INFO - [2019-12-04 08:10:54,120] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 249, in reraise\n'
[2019-12-04 08:10:54,120] INFO - [2019-12-04 08:10:54,120] INFO - b' raise value\n'
[2019-12-04 08:10:54,121] INFO - [2019-12-04 08:10:54,121] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool.py", line 1190, in _do_get\n'
[2019-12-04 08:10:54,122] INFO - [2019-12-04 08:10:54,122] INFO - b' return self._create_connection()\n'
[2019-12-04 08:10:54,123] INFO - [2019-12-04 08:10:54,123] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool.py", line 347, in _create_connection\n'
[2019-12-04 08:10:54,123] INFO - [2019-12-04 08:10:54,123] INFO - b' return _ConnectionRecord(self)\n'
[2019-12-04 08:10:54,124] INFO - [2019-12-04 08:10:54,124] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool.py", line 474, in __init__\n'
[2019-12-04 08:10:54,125] INFO - [2019-12-04 08:10:54,125] INFO - b' self.__connect(first_connect_check=True)\n'
[2019-12-04 08:10:54,126] INFO - [2019-12-04 08:10:54,126] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool.py", line 671, in __connect\n'
[2019-12-04 08:10:54,126] INFO - [2019-12-04 08:10:54,126] INFO - b' connection = pool._invoke_creator(self)\n'
[2019-12-04 08:10:54,127] INFO - [2019-12-04 08:10:54,127] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/strategies.py", line 106, in connect\n'
[2019-12-04 08:10:54,128] INFO - [2019-12-04 08:10:54,128] INFO - b' return dialect.connect(*cargs, **cparams)\n'
[2019-12-04 08:10:54,129] INFO - [2019-12-04 08:10:54,129] INFO - b' File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 412, in connect\n'
[2019-12-04 08:10:54,130] INFO - [2019-12-04 08:10:54,129] INFO - b' return self.dbapi.connect(*cargs, **cparams)\n'
[2019-12-04 08:10:54,131] INFO - [2019-12-04 08:10:54,131] INFO - b' File "/usr/local/lib/python3.7/site-packages/psycopg2/__init__.py", line 130, in connect\n'
[2019-12-04 08:10:54,132] INFO - [2019-12-04 08:10:54,132] INFO - b' conn = _connect(dsn, connection_factory=connection_factory, **kwasync)\n'
[2019-12-04 08:10:54,133] INFO - [2019-12-04 08:10:54,133] INFO - b'sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not connect to server: Connection timed out\n'
[2019-12-04 08:10:54,133] INFO - [2019-12-04 08:10:54,133] INFO - b'\tIs the server running on host "10.138.16.11" and accepting\n'
[2019-12-04 08:10:54,134] INFO - [2019-12-04 08:10:54,134] INFO - b'\tTCP/IP connections on port 5432?\n'
[2019-12-04 08:10:54,134] INFO - [2019-12-04 08:10:54,134] INFO - b' (Background on this error at: http://sqlalche.me/e/e3q8)\n'
[2019-12-04 08:10:59,152] INFO - [2019-12-04 08:10:59,152] INFO - b'ERROR:snowflake.connector.network:could not find io module state (interpreter shutdown?)\n'
[2019-12-04 08:10:59,152] INFO - [2019-12-04 08:10:59,152] INFO - b'Traceback (most recent call last):\n'
[2019-12-04 08:10:59,153] INFO - [2019-12-04 08:10:59,153] INFO - b' File "/usr/local/lib/python3.7/site-packages/snowflake/connector/network.py", line 781, in _request_exec\n'
[2019-12-04 08:10:59,154] INFO - [2019-12-04 08:10:59,154] INFO - b' auth=SnowflakeAuth(token),\n'
[2019-12-04 08:10:59,155] INFO - [2019-12-04 08:10:59,155] INFO - b' File "/usr/local/lib/python3.7/site-packages/botocore/vendored/requests/sessions.py", line 451, in request\n'
[2019-12-04 08:10:59,155] INFO - [2019-12-04 08:10:59,155] INFO - b' prep = self.prepare_request(req)\n'
[2019-12-04 08:10:59,156] INFO - [2019-12-04 08:10:59,156] INFO - b' File "/usr/local/lib/python3.7/site-packages/botocore/vendored/requests/sessions.py", line 382, in prepare_request\n'
[2019-12-04 08:10:59,157] INFO - [2019-12-04 08:10:59,157] INFO - b' hooks=merge_hooks(request.hooks, self.hooks),\n'
[2019-12-04 08:10:59,158] INFO - [2019-12-04 08:10:59,158] INFO - b' File "/usr/local/lib/python3.7/site-packages/botocore/vendored/requests/models.py", line 307, in prepare\n'
[2019-12-04 08:10:59,158] INFO - [2019-12-04 08:10:59,158] INFO - b' self.prepare_body(data, files, json)\n'
[2019-12-04 08:10:59,160] INFO - [2019-12-04 08:10:59,159] INFO - b' File "/usr/local/lib/python3.7/site-packages/botocore/vendored/requests/models.py", line 436, in prepare_body\n'
[2019-12-04 08:10:59,160] INFO - [2019-12-04 08:10:59,160] INFO - b' length = super_len(data)\n'
[2019-12-04 08:10:59,161] INFO - [2019-12-04 08:10:59,161] INFO - b' File "/usr/local/lib/python3.7/site-packages/botocore/vendored/requests/utils.py", line 59, in super_len\n'
[2019-12-04 08:10:59,161] INFO - [2019-12-04 08:10:59,161] INFO - b' fileno = o.fileno()\n'
[2019-12-04 08:10:59,162] INFO - [2019-12-04 08:10:59,162] INFO - b'RuntimeError: could not find io module state (interpreter shutdown?)\n'
[2019-12-04 08:10:59,162] INFO - [2019-12-04 08:10:59,162] INFO - b'\n'
[2019-12-04 08:10:59,163] INFO - [2019-12-04 08:10:59,163] INFO - b'During handling of the above exception, another exception occurred:\n'
[2019-12-04 08:10:59,163] INFO - [2019-12-04 08:10:59,163] INFO - b'\n'
[2019-12-04 08:10:59,163] INFO - [2019-12-04 08:10:59,163] INFO - b'Traceback (most recent call last):\n'
[2019-12-04 08:10:59,164] INFO - [2019-12-04 08:10:59,164] INFO - b' File "/usr/local/lib/python3.7/site-packages/snowflake/connector/network.py", line 648, in _request_exec_wrapper\n'
[2019-12-04 08:10:59,164] INFO - [2019-12-04 08:10:59,164] INFO - b' **kwargs)\n'
[2019-12-04 08:10:59,165] INFO - [2019-12-04 08:10:59,165] INFO - b' File "/usr/local/lib/python3.7/site-packages/snowflake/connector/network.py", line 871, in _request_exec\n'
[2019-12-04 08:10:59,166] INFO - [2019-12-04 08:10:59,166] INFO - b' raise RetryRequest(err)\n'
[2019-12-04 08:10:59,167] INFO - [2019-12-04 08:10:59,167] INFO - b'snowflake.connector.network.RetryRequest: could not find io module state (interpreter shutdown?)\n'
[2019-12-04 08:11:04,486] INFO - [2019-12-04 08:11:04,486] INFO - Event: license-db-incremental-df1efe5e had an event of type Failed
[2019-12-04 08:11:04,487] INFO - [2019-12-04 08:11:04,487] INFO - Event with job id license-db-incremental-df1efe5e Failed
[2019-12-04 08:11:04,493] INFO - [2019-12-04 08:11:04,493] INFO - Event: license-db-incremental-df1efe5e had an event of type Failed
[2019-12-04 08:11:04,493] INFO - [2019-12-04 08:11:04,493] INFO - Event with job id license-db-incremental-df1efe5e Failed
[2019-12-04 08:11:04,564] ERROR - Pod Launching failed: Pod returned a failure: failed
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 142, in execute
'Pod returned a failure: {state}'.format(state=final_state)
airflow.exceptions.AirflowException: Pod returned a failure: failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/airflow/models/__init__.py", line 1441, in _run_raw_task
result = task_copy.execute(context=context)
File "/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 147, in execute
raise AirflowException('Pod Launching failed: {error}'.format(error=ex))
airflow.exceptions.AirflowException: Pod Launching failed: Pod returned a failure: failed
[2019-12-04 08:11:04,565] INFO - All retries failed; marking task as FAILED
[2019-12-04 08:11:04,894] INFO - Job 22469: Subtask license-db-incremental Traceback (most recent call last):
[2019-12-04 08:11:04,894] INFO - Job 22469: Subtask license-db-incremental File "/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 142, in execute
[2019-12-04 08:11:04,894] INFO - Job 22469: Subtask license-db-incremental 'Pod returned a failure: {state}'.format(state=final_state)
[2019-12-04 08:11:04,894] INFO - Job 22469: Subtask license-db-incremental airflow.exceptions.AirflowException: Pod returned a failure: failed
[2019-12-04 08:11:04,894] INFO - Job 22469: Subtask license-db-incremental
[2019-12-04 08:11:04,894] INFO - Job 22469: Subtask license-db-incremental During handling of the above exception, another exception occurred:
[2019-12-04 08:11:04,894] INFO - Job 22469: Subtask license-db-incremental
[2019-12-04 08:11:04,894] INFO - Job 22469: Subtask license-db-incremental Traceback (most recent call last):
[2019-12-04 08:11:04,894] INFO - Job 22469: Subtask license-db-incremental File "/usr/local/bin/airflow", line 32, in <module>
[2019-12-04 08:11:04,894] INFO - Job 22469: Subtask license-db-incremental args.func(args)
[2019-12-04 08:11:04,894] INFO - Job 22469: Subtask license-db-incremental File "/usr/local/lib/python3.6/site-packages/airflow/utils/cli.py", line 74, in wrapper
[2019-12-04 08:11:04,894] INFO - Job 22469: Subtask license-db-incremental return f(*args, **kwargs)
[2019-12-04 08:11:04,894] INFO - Job 22469: Subtask license-db-incremental File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 523, in run
[2019-12-04 08:11:04,894] INFO - Job 22469: Subtask license-db-incremental _run(args, dag, ti)
[2019-12-04 08:11:04,894] INFO - Job 22469: Subtask license-db-incremental File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 442, in _run
[2019-12-04 08:11:04,894] INFO - Job 22469: Subtask license-db-incremental pool=args.pool,
[2019-12-04 08:11:04,894] INFO - Job 22469: Subtask license-db-incremental File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 73, in wrapper
[2019-12-04 08:11:04,895] INFO - Job 22469: Subtask license-db-incremental return func(*args, **kwargs)
[2019-12-04 08:11:04,895] INFO - Job 22469: Subtask license-db-incremental File "/usr/local/lib/python3.6/site-packages/airflow/models/__init__.py", line 1441, in _run_raw_task
[2019-12-04 08:11:04,895] INFO - Job 22469: Subtask license-db-incremental result = task_copy.execute(context=context)
[2019-12-04 08:11:04,895] INFO - Job 22469: Subtask license-db-incremental File "/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 147, in execute
[2019-12-04 08:11:04,895] INFO - Job 22469: Subtask license-db-incremental raise AirflowException('Pod Launching failed: {error}'.format(error=ex))
[2019-12-04 08:11:04,895] INFO - Job 22469: Subtask license-db-incremental airflow.exceptions.AirflowException: Pod Launching failed: Pod returned a failure: failed
[2019-12-04 08:11:07,868] INFO - [2019-12-04 08:11:07,867] INFO - Task exited with return code 1