[sync] SyncML roadmap, kind of

[Karsten Fourmont]

Hi,

I've been asked for some kind of roadmap for SyncML. So here are my thoughts
about it. I hope I'm not too verbose. :-)

First step is of course to get the current codebase into cvs so others can
give it a try. 

(can anybody with cvs access  untar
http://www.fourmont.com/zombie/SyncML_complete.tgz  (20kb)
in framework/SyncML. Then add the 3 doc files from my earlier posting to
docs and add them together with Command/Map.php to package.xml? Thanks!)

Here are the areas that need to be dealt with:

I: Protocol Issues
==================

Once the current code makes it to CVS people should give it a try and check
if it works. If not, the xml in /tmp/sync/, the horde log and the php error
log should be helpful. If you can provide a patch to make it work: great!

Otherwise mail the offending xml and/or logs to the list. I'll have a look
at it.

As many phones/pdas/whater should be thrown at the code. I'm especially
interested to see the <devinf> device information section that the phones
sends in its initial packet. This gets stored in
/tmp/sync/syncml_client_0.xml when you start with an empty /tmp/sync/ dir. 

Make sure to modify the <cred>...</cred> part in the syncml header if it
exists. It contains your username and password.

I'll try to come up with a brief "how to help with debugging" and add it to
the installation docs. Otherwise we might get lots of "just installed the
horde rpm and syncml doesn't work with my XXYY phone! pls hlp!!!! I'm new to
linux!!!!!" messages :-)

II: Data Issues
===============

The second main area where I expect problems are the various implementations
of the text/x-vcalendar,vcard,... formats flying around. Basically something
like

BEGIN:VNOTE
VERSION:1.1
BODY: Here's a quick note
END:VNOTE

Syncml itself doesn't understand these things: it just passes them around
between the phone and the external horde api "as they come".

If you have tricky stuff like recurring events, inline photos or other
exotic data in the file (or got them "treated" by Outlook), horde or your
phone might quickly get a hickup. So we need to test as many of these
formats as we can get our hands on.

This is more of a general horde issue than about SyncML. You can test the
behaviour by trying to import/export files with this ical/vcard data into
kronolith/nag/turba using their respective import/export menu entries.  That
should use the same codebase.

Right now as I'm writing this, I've noticed Jan creating a
framework/iCalendar/tests/charset1.phpt test file.  Excellent!


III: Code Issues
================

Of course the SnycML code is far from being finished. Here are a few things
that need work besides the stuff directly coming fomr I or II. (relevant
modules are mentioned in brackets)

1) refactoring of current codebase (SYNCML)

The current codebase is somewhat confusing and hard to deal with. I'll try
to refactor it to make it easier for other people to jump in. Any help is
appreciated. Especially suggestions about the best way to parse XML-Subtrees
into PHP-Objects.

2) better error handling (SYNCML): currently no real error handling is in
place.

3) Turba contacts support (TURBA): I haven't looked at it yet. it might just
work or not. Depends on whether VCARD in turba support is fine or not. You
can just try to sync vcard data and see if you run into problems.

4) better tasks import(NAG):
   currently nag imports only task name, description and priority. See the
two @TODO lines in nag/lib/api.php
   This should be extended to import the full info (alarm, due dates etc.).
Export should be checked as well.
   
   
5)  Collision handling: (SYNCML,NAG,MNEMO,KRONOLITH,TURBA)

  My idea for collision handling is:
  while running syncml: keep the existing entry and add the collision as a
new one somehow marked as "collision" of entry guid #xxx. (stored in
datatree)
  Then kronolith etc. should have a page to display the colliding items side
by side. There you can select which one to keep (and/or maybe edit
manually). The "losing" entry gets deleted, which is then automatically
synced as any normal delete during the next sync session. Should be rather
straightforward to implement and requires minimal effords on the syncml
side.
   
6) Get support for additional Sync Types (SyncML)
   Currently only SlowSync and TwoWaySync are supported. These are the only
two that are mandatory in SyncML and the only two my phone supports. Don't
know if we need the others. Some code is already there.
   
7) Make use of devinf information (SyncML): 
In the inital SyncML packets both sides exchange a <devinf> block with
information about device capabilites. This info should be stored and used.

 
8) Horde api enhancements (HORDE API):
a) currently the horde api only supports listby($action,$ts) to retrieve
$actions since timestamp $ts. The SyncML protocol requires that the ending
timestamp of the sync timeframe to be  exchanged _before_ the actual syncing
starts. So we need an additional parameter:
listby($action,$ts,$ts_end)

b) the syncml works as follows:
    1) the clients sends its changes to the server
    2) the server sends its changes to the client.
    
so when in step 2), the horde api is called with a request like "give me all
changes in horde since the last sync", you get the changes induced by the
client in step 1) as well. You have to somehow "tag" them to avoid echoing
(and thus duplicatinging) them back to the client. Simply storing the guids
in the session is not sufficient: the changes are made _after_ the end time
stamp (see a)) of the current sync so you'll dupe them in the next sync. 
My current implementation deals with this as follows: directly after a
client induced change is done in horde, state.php's getTSforAction is called
to find out the biggest=latest ts for the given guid. If the horde api would
provide me with this info directly (maybe just store it in a static var and
provide a getLastTS() method in Driver), that would be great and elminiate
costly and redundant DataTree calls. 
Similar in listBy where you get a list of guids but still need to retrieve
the latest History timestamps for these guids manually using the DataTree so
you can check out if the changes are a result of a client request or not. If
listBy would return an assoc array $guid->ts that would help a lot (and
solve a) as well).

IV: Testing Suite
=================
Nobody has access to all the syncml clients all the time. So there's a
always the risk that you supply a fancy enhancement patch that works fine
for you but breaks down everything for everyone else. So it would be great
to have a "testing suite" (in JUNIT terms) which simulates sync runs for all
supported phones. (Maybe done by sending specifically crafted XML files
based on the logging data of various phones and checking the results).
Before a commit of a new patch a test run is done with this suite to ensure
it doesn't break things up. Just an idea (dream) for now, suggestions how
this could be done are welcome.


If anybody want to contribute to any of these items (or contribute an item,
for that matter): that's highly appretiated!

Thanks 

 Karsten