[4.0.7] Whoosh backend: searching for metadata values gives wrong/erratic results

I switched to the Whoosh backend because I was affected by issue #983 (closed). Even though I can now search for file contents again, searching for metadata seems to be broken.

Steps to reproduce, as an example:

1. Create metadata type "amount" and attach it to a document type.
2. Create a document with "amount" set to "10.00".
3. Go to advanced search.
4. Enter "10.00" in the Metadata value field.

Expected result: the document should be found. Actual result: no result.

I already tried to rebuild the search index without success.

However, I'm not completely sure how reproducible this issue is. Searching for metadata values gives me very erratic results. I also have another metadata type "date", which contains dates in the form YYYY-MM-DD. Searching for Matadata value "2021" produces many results correct results where 2021 is part of the metadata value. Searching for Metadata value "2021-06-03" gives me several documents with date "2021-03-06", even though other documents with the date 2021-06-03 exist as well. Searching for Metadata value "2021-06-02" returns document with date "2021-06-02" as well as "2021-02-06". Searching for 2021-02-10 does not return any results, even though it should.

In the logs I found:

[2021-06-27 12:33:11,291: ERROR/ForkPoolWorker-7888] Task mayan.apps.dynamic_search.tasks.task_deindex_instance[0d5ce54a-b459-4451-9f37-149d0d144b32] raised unexpected: DoesNotExist('IndexInstanceNode matching query does not exist.')                                        
Traceback (most recent call last):                                                                                                                                                                                                                                               
  File "/opt/mayan-edms/lib/python3.7/site-packages/celery/app/trace.py", line 412, in trace_task                                                                                                                                                                                
    R = retval = fun(*args, **kwargs)                                                                                                                                                                                                                                            
  File "/opt/mayan-edms/lib/python3.7/site-packages/celery/app/trace.py", line 704, in __protected_call__                                                                                                                                                                        
    return self.run(*args, **kwargs)                                                                                                                                                                                                                                             
  File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/dynamic_search/tasks.py", line 22, in task_deindex_instance                                                                                                                                                       
    instance = Model._meta.default_manager.get(pk=object_id)                                                                                                                                                                                                                     
  File "/opt/mayan-edms/lib/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method                                                                                                                                                                     
    return getattr(self.get_queryset(), name)(*args, **kwargs)                                                                                                                                                                                                                   
  File "/opt/mayan-edms/lib/python3.7/site-packages/django/db/models/query.py", line 408, in get                                                                                                                                                                                 
    self.model._meta.object_name                                                                                                                                                                                                                                                 
mayan.apps.document_indexing.models.IndexInstanceNode.DoesNotExist: IndexInstanceNode matching query does not exist.

but I'm not sure whether this is related.