Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The current approach crashes the worker processes because the sqlalchemy connection isn't initiated properly.
The sympton is that the first job was processed beautifully but subsequent jobs failed with the following stack trace:
Exception during reset or similar
Traceback (most recent call last):
File "app/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 739, in _r
ollback_impl
self.engine.dialect.do_rollback(self.connection)
File "app/lib/python3.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 22
41, in do_rollback
dbapi_connection.rollback()
MySQLdb._exceptions.OperationalError: (2006, 'MySQL server has gone away')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "app/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 680, in _fin
alize_fairy
fairy._reset(pool)
File "app/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 865, in _res
et
self._reset_agent.rollback()
File "app/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1705, in r
ollback
self._do_rollback()
File "app/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1743, in _
do_rollback
self.connection._rollback_impl()
File "app/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 741, in _r
ollback_impl
self._handle_dbapi_exception(e, None, None, None, None)
File "app/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1466, in _
handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "app/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 399, in ra
ise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "app/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 153, in re
raise
raise value.with_traceback(tb)
File "app/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 739, in _r
ollback_impl
self.engine.dialect.do_rollback(self.connection)
File "app/lib/python3.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 22
41, in do_rollback
dbapi_connection.rollback()
sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (2006, 'MySQL server has gone a
way')
(Background on this error at: http://sqlalche.me/e/e3q8)
The
FlaskJob.load_app
has the good intention to reuse thecurrent_app
but IMHO it's making too many assumptions of how the Flask app is structured.Since RQ is mainly used in backend processes where performance is not the highest priority, I'd like to suggest we call
create_app
every time a job is performed. This provides the maximum guarantee on the correctness of the appcontext.