## Context --- While working with Bots NG after upgrading system from the previous versions. We receive an x12 files that contains ~ as the field separator and | as a record separator. NOTE: I'm being a bit over descriptive in system symptoms here because it was a difficult issue to pinpoint and if others have similar issues I figured it may help. This file processed under the old version of Bots we were running correctly. After updating to the Bots NG odd behavior started. When this file was received no translation step occurred. Bots engine logging file naming indicates a huge jump in processing. {width=494 height=95} Engine Error log indicates ``` 2026.04.11 04:44:57 ERROR [bots.engine] Traceback (most recent call last): File "E:\Data\bots_customer_prod\configuration\project\.venv\Lib\site-packages\bots\preprocess.py", line 58, in preprocess function(ta_from=ta_from, endstatus=status, routedict=routedict, **kwargs) ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Data\bots_customer_prod\configuration\project\.venv\Lib\site-packages\bots\preprocess.py", line 384, in mailbag ta_to.update( ~~~~~~~~~~~~^ statust=OK, ^^^^^^^^^^^ ...<3 lines>... filesize=filesize, ^^^^^^^^^^^^^^^^^^ ) ^ File "E:\Data\bots_customer_prod\configuration\project\.venv\Lib\site-packages\bots\botslib.py", line 119, in update changeq( ~~~~~~~^ """UPDATE ta ^^^^^^^^^^^^ ...<2 lines>... ta_info, ^^^^^^^^ ) ^ File "E:\Data\bots_customer_prod\configuration\project\.venv\Lib\site-packages\bots\botslib.py", line 328, in changeq cursor.execute(querystring, *args) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "E:\Data\bots_customer_prod\configuration\project\.venv\Lib\site-packages\bots\botssqlite.py", line 56, in execute sqlite3.Cursor.execute(self, query, parameters) ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^ sqlite3.OperationalError: unable to open database file ``` This occurred over and over until a full lock occurred due to available disk space being 0 bytes with the $MFT showing as having the remaining disk space. {width=900 height=143} botsys/data folder at this point had 15.5 million new of 0 byte files after the first occurrence. Having this many small files can impact the $MFT size and lock up disk space. ## Troubleshooting/Cause --- Since we determined that the cause was the file we received with | as the record separator and we did not have a successful translation I compared preprocess from our old server to the new server. After adding some logging and testing I believe I've found the cause to be the regex search for the x12 trailer. ### Old Trailer Search - preprocess.py ``` foundtrailer = re.search('''%(record_sep)s \s* I[\n\r]*E[\n\r]*A .+? %(record_sep)s '''%{'record_sep':re.escape(record_sep)}, edifile[headpos:],re.DOTALL|re.VERBOSE) ``` The old version is doing a replace on record_sep with an escaped value of |. ### New Trailer Search - preproces.py:244 https://gitlab.com/bots-ediint/bots/-/blob/main/bots/preprocess.py?ref_type=heads#L245 ``` foundtrailer = re.search( f"{record_sep}" r"\s*" "I[\n\r]*E[\n\r]*A" ".+?" f"{re.escape(record_sep)}", edifile[headpos:], re.DOTALL | re.VERBOSE, ) ``` The new version is using a formatted string literal where the first separator is not escaped while the second one is escaped. Running a quick test in locally with sample print statements shows that we find the correct separator and the found trailer matched successfully, however the value was blank. {width=900 height=424} Adding in escape on the first record_sep in the regex allows the trailer to be found appropriately. {width=900 height=300} This file looped 1 time and output the appropriate translation successfully. ## Problem --- Unescapped special characters during trailer lookup can cause a successful regex match to ''(blank). This causes the infinite mailbag While True loop, filling up the disk with 0 byte data files. Thus causing the $MFT to bloat to the full disk size at which point it hard crashes. ## Proposal --- Add in an escape for x12 and edifact trailer searches. ``` re.escape(record_sep) ``` X12 https://gitlab.com/bots-ediint/bots/-/blob/main/bots/preprocess.py?ref_type=heads#L245 EDIFACT https://gitlab.com/bots-ediint/bots/-/blob/main/bots/preprocess.py?ref_type=heads#L324 Note: I'm only assuming this is also a possible with EDIFACT and haven't validated as such.