[sync] Sync problem

[Jan Schneider]

Zitat von Joe Greco <jgreco at ns.sol.net>:

>> Zitat von Joe Greco <jgreco at ns.sol.net>:
>>
>> >> Does the contact information (output received/output converted) in
>> >> data.txt look sane?
>> >
>> > Yes, it seemed fine, both for the affected entries and the ones around
>> > them (which were added fine).
>> >
>> >> Also search for "Maximum message size" in the sync log.
>> >
>> > Ok, I see some
>> >
>> > horde.log:Sep 24 07:24:11 HORDE [debug] [horde] Maximum message size
>> > 50000 approached during add; current size: 43348 [pid 84161 on line
>> > 501 of "/${redacted}/horde/lib/SyncML/Sync.php"]
>> > horde.log:Sep 24 07:25:25 HORDE [debug] [horde] Maximum message size
>> > 50000 approached during add; current size: 42711 [pid 84161 on line
>> > 501 of "/${redacted}/horde/lib/SyncML/Sync.php"]
>> > log.txt:DEBUG:  Maximum message size 50000 approached during add;
>> > current size: 8883
>> > log.txt:DEBUG:  Maximum message size 50000 approached during add;
>> > current size: 22267
>> >
>> > 22 instances in each log file.
>> >
>> > These appear to be individual records that aren't making it, then?
>>
>> No, this is splitting up large amounts of data into several messages
>> (packets). Did you look *around* those messages?
>
> No, because it was like 4AM and I just wanted to make sure I actually
> replied...  and because...
>
>> Are there other error
>> messages or warnings, specifically like "Item will not be sent"?
>
> ... I figured knowing where you were going with that would be useful.
> That seems to be the magic string-to-grep-for.
>
> Yes, there are "item will not be sent" messages, but only 7 in each of
> those logs.  That *looks* more or less consistent with what I think I
> am missing from the phone, and the approximate number of images that
> are probably squeaky-close to or exceed 50KB when base64 encoded.
>
> DEBUG:  Sending add from server: 20100626111600.14284swfjbo5uke8@${foo}
>
> [I haven't figured out how to translate those ID's to something that I
> can retrieve for debugging.]

It the object's UID, e.g. the event_uid column in the database etc.

> DEBUG:  Sending add from server: 20100225132336.16265hxx2q0vkz7c@${foo}
> DEBUG:  Maximum message size 50000 approached during add; current size: 28121
> WARNING:Data item won't fit into a single message. Partial sending  
> not implemented yet. Item will not be sent!
> DEBUG:  Sending add from server: 20100331071823.178223fcnt1jxea7@${foo}
> DEBUG:  Maximum message size 50000 approached during add; current size: 28121
>
> Now I see in the code that this is probably something that ought to be
> handled more gracefully:
>
>                         /* @todo: implement partial sending instead of
> 			 * dropping item! */
>
> And my impression is that this is because the attached picture for the
> contact is too big, is that probably right?

Exactly.

> If so, my thinking goes, limiting picture size when inserting is a bit
> of a quick fix, but not really correct since a syncml client can use a
> smaller max message size that would hose up any particular constant
> value one might select.
>
> It seems obvious why reporting such errors in a user-visible way is
> problematic, so we don't need to go there.
>
> Is "partial sending" in this context intended to mean sending a contact
> as more than one message, or does it mean truncating the contact so
> that it does all fit in one message?  If it means the latter, it might
> almost be interesting to see if there would be a way to resize the image
> so that it fit in the message, though that opens a new can of worms of
> course.

No, it's the former, i.e. splitting a single synchronization objects  
over several sync messages.

> I guess I could dig in and see if the syncml stuff is comprehensible to
> me, but I'm not a great PHP coder.
>
> I would also be willing to offer a bribe in the form of (reimbursement
> for) a case of beer or something like that to any developer who had a
> desire to work on this and just needed an incentive; anyone who's
> interested can talk to me about specifics such as level of drunkenness
> required.
>
> My main concern is that as it stands, there's no warning to users that
> stuff they add won't be exported correctly via sync, and simply dropping
> the whole record is a bit of a disaster.  I think I have to find some
> way to address this.  I could live with the image being dropped or being
> replaced with an error image or something.

The problem is that at the point where the size calculation happens,  
the actual data is completely abstract to the synchronization code. It  
could be anything and there is no way to find out which property of  
the object made up the size overflow, let alone to remove this property.
And it's not really *dropping* the object, it just doesn't get  
synchronized to the sync client.

Unfortunately the SyncML standard doesn't provide any means to return  
custom error messages to the client, and even for the standardized  
errors, most clients only display some meaningless, useless error  
message.

With the new Alarm framework that has been added with Horde 3.2, we  
might be able to store warnings for the user, so that he at least sees  
them when logging in to Horde next time.
But as you already notice, the *real* solution would be to implement  
partial sending. But that costs more than a few beers. ;-)

Jan.

-- 
Do you need professional PHP or Horde consulting?
http://horde.org/consulting/