[dev] assertion failed: mailbox.c: 2180: !message_guid_isnull(&record->guid)

[Michael M Slusarz]

Quoting Jan Schneider <jan at horde.org>:

> Zitat von Jan Schneider <jan at horde.org>:
>
>> Zitat von Bron Gondwana <brong at fastmail.fm>:
>>
>>> On Tue, Nov 20, 2012, at 07:15 PM, Aleksander Machniak wrote:
>>>> On 11/20/2012 04:07 PM, steffo76 at gmx.de wrote:
>>>>>> 1353423361>* BYE Fatal error: Internal error: assertion failed:  
>>>>>> mailbox.c: 2180: !message_guid_isnull(&record->guid)^M
>>>>
>>>> I created a ticket for this yesterday.
>>>> https://bugzilla.cyrusimap.org/show_bug.cgi?id=3755. It looks like a bug
>>>> with APPEND + BINARY implementation.
>>>
>>> I'll have a look soon, promise!  I'd like to get a fix into 2.4.17
>>> and release it super-soon.  There are a couple of people who have
>>> seen it recently.  Did some client switch to using BINARY APPEND?
>>
>> The Horde_Imap_Client library did.
>>
>>> Definitely a bug!
>>>
>>> I'm in the process of moving country this week, so I'm only online
>>> sporadically...
>>>
>>> Bron.
>>
>
> Bron just fixed this in Cyrus, but since this affects major  
> functionality (saving sent-mail and drafts) with a version of a  
> major IMAP server that is shipped with major distributions, can we  
> workaround this bug? It looks like it is triggered with BINARY  
> APPEND only.

F***.  This is not something that can be easily worked around without  
totally destroying performance.  Here's why:

1. Previous scanning for null's in data could be accomplished by doing  
a strpos().  This is fast, but requires the full message data to be  
stored in PHP memory (which is bad as it can cause memory fatal errors).
2. To fix memory usage, data needs to be stored in streams.
3. Now, to search for nulls, we use a stream filter  
(Horde_Imap_Client_Data_Format_Filter_String).  This stream filter  
unfortunately has fairly deplorable performance.
Example: To import a 20MB mbox file in IMP, it would take 30-40  
seconds to process.
4. Since #2 can't be reversed, which means going back to #1 is not an  
option, a workaround (implemented in Horde_Imap_Client 2.2.0) is to  
skip null scanning of the input data and simply output the data as-is.  
  The reasoning:
   - if BINARY is available, we know the server can handle null data  
so we can just activate binary literals and output the raw data.
   - if BINARY is not available, there's nothing we can do if the data  
contains a null character since there is no workaround. So we just  
output to the server and hope for the best (it is possible that some  
servers could handle that data, or at least ignore the nulls)

These optimizations reduce the time required to import a 20MB mbox  
file by a factor of 10.

I think the current method of working around this is still the correct  
method (try with BINARY; re-do if the APPEND fails).  I see that you  
implemented a hotfix (1220a7ed6888eb61d29d8e1228d216dac3e2bb97), but  
this hotfix is too broad.  NO is a perfectly valid (and possibly  
expected) response code from APPEND so we can't lump that in with  
BAD/BYE responses.

michael

___________________________________
Michael Slusarz [slusarz at horde.org]