CachePartion.create_file contextmanager doesn't close storage file properly
The context manager "create_file" in "CachePartiton" class doesn't close/flush de file properly. This cause data lost when use file-like object storage (like S3Boto3Storage).
In order to reproduce:
On a fresh/new docker install, you have to configure an S3 like Storage for cache: (or for common storage, produces the same issue in other location)
MAYAN_DOCUMENTS_CACHE_STORAGE_BACKEND=storages.backends.s3boto3.S3Boto3Storage MAYAN_DOCUMENTS_CACHE_STORAGE_BACKEND_ARGUMENTS={'access_key':'','secret_key':'','bucket_name':'','default_acl':'private', 'location':'cache'}
In mayan frontend: upload a file and preview them. Then, on celery workers you get some errors and no preview is shown.
[2020-08-28 13:30:46,629: ERROR/ForkPoolWorker-1] Task mayan.apps.documents.tasks.task_generate_document_page_image[1410cdc4-2282-4fbe-91cb-2932480d394c] raised unexpected: AttributeError("'NoneType' object has no attribute 'size'",)
Traceback (most recent call last):
File "~/.virtualenvs/mayan/lib/python3.6/site-packages/celery/app/trace.py", line 385, in trace_task
R = retval = fun(*args, **kwargs)
File "~/.virtualenvs/mayan/lib/python3.6/site-packages/celery/app/trace.py", line 648, in __protected_call__
return self.run(*args, **kwargs)
File "~/projectes/mayan-edms/mayan/apps/documents/tasks.py", line 78, in task_generate_document_page_image
return document_page.generate_image(user=user, **kwargs)
File "~/projectes/mayan-edms/mayan/apps/documents/models/document_page_models.py", line 99, in generate_image
image = self.get_image(transformations=transformation_list)
File "~/projectes/mayan-edms/mayan/apps/documents/models/document_page_models.py", line 212, in get_image
converter.transform(transformation=transformation)
File "~/projectes/mayan-edms/mayan/apps/converter/classes.py", line 232, in transform
self.image = transformation.execute_on(image=self.image)
File "~/projectes/mayan-edms/mayan/apps/converter/transformations.py", line 422, in execute_on
super(TransformationResize, self).execute_on(*args, **kwargs)
File "~/projectes/mayan-edms/mayan/apps/converter/transformations.py", line 93, in execute_on
self.aspect = 1.0 * image.size[0] / image.size[1]
AttributeError: 'NoneType' object has no attribute 'size'
In the cache storage, you can find a 0 bytes file.
This is due to that the context manager doesn't close(and flush) the file:
class CachePartition(models.Model):
@contextmanager
def create_file(self, filename):
lock_id = 'cache_partition-create_file-{}-{}'.format(self.pk, filename)
try:
logger.debug('trying to acquire lock: %s', lock_id)
lock = locking_backend.acquire_lock(lock_id)
logger.debug('acquired lock: %s', lock_id)
try:
self.cache.prune()
# Since open "wb+" doesn't create files force the creation of an
# empty file.
self.cache.storage.delete(
name=self.get_full_filename(filename=filename)
)
self.cache.storage.save(
name=self.get_full_filename(filename=filename),
content=ContentFile(content='')
)
try:
with transaction.atomic():
partition_file = self.files.create(filename=filename)
yield partition_file.open(mode='wb')
partition_file.update_size()
except Exception as exception:
logger.error(
'Unexpected exception while trying to save new '
'cache file; %s', exception
)
self.cache.storage.delete(
name=self.get_full_filename(filename=filename)
)
raise
finally:
lock.release()
except LockError:
logger.debug('unable to obtain lock: %s' % lock_id)
raise
When use a custom context manager, this is responsible of releasing the captured resources, for example: https://docs.python.org/3/library/contextlib.html
from contextlib import contextmanager
@contextmanager
def managed_resource(*args, **kwds):
# Code to acquire resource, e.g.:
resource = acquire_resource(*args, **kwds)
try:
yield resource
finally:
# Code to release resource, e.g.:
release_resource(resource)