[imp] mailbox listing speed-up

[Ziba Scott]

Hi imp devs and all,

After our recent horde/imp upgrade at umich, we've seen login times rise
from ~5-10 seconds to ~10-30 seconds.   The bulk of this time is spent
building the mailbox list in IMP_Tree (imp/lib/IMAP/Tree.php).  We've
made some changes which dramatically reduce the IMP_Tree startup time
and would love some feedback and basic sanity checking before we move
forward with them.

On an unloaded test server in our environment, imap_getmailboxes and
imap_list commands can take up to 1.7 seconds each against our Cyrus system.

First, IMP_Tree calls imap_getmailboxes for each namespace with a %
appended to get one level deep of subfolders. 

Second, IMP_Tree::_initSubscribed executes imap_list for each namespace
with an * appended to get all sublevels of folders.

Last, it recursively crawls each folder found during the first set of
imap_getmailboxes and calls imap_getmailboxes on each folder with %
again until all branches have been plumbed.

Our improvements are premised on the fact that one level searches (%)
and searches which return full trees of results (*) actually take the
same time to return from Cyrus.  So we are bypassing all other searches
and starting off with one single full (*) search from the root of the
user's mailbox.

One quirk we had to side step then was that % searches will return
objects for folders which have children that can't be directly accessed
themselves.  The * search will not.

So we just iterate through the php data structure and detect parentless
children and fake up an object for them.  Then, if tree_view is turned
on, we separate non-personal mailboxes into  IMPTREE_OTHER_KEY and
IMPTREE_SHARED_KEY.

Since the list of folders at this point has all the children, we bypass
the recursive calls to expand. 

This also partially fixes bug: http://bugs.horde.org/ticket/6703:
"problems displaying subfolders whose parent's name is a substring of
another folder name w/ a space in it"
It fixes it in the folder drop down display but not on the folder list. 
(I'll look into that after we get people logging in nice and quick like :)

If anyone wants to try it out, I've attached a diff against Tree.php ,v 
1.185 (our test env).  The actual changes aren't that big.  Blocks that
got indented for style reasons make it look bigger.  Our changes are
turned on and off with a new config option: $conf['imap']['tree_build']
= 'quick';

Our initial tests are very promising.  Tree building on login is down
from about (1.7sec/per namespace)*2 + (0.2 sec/per subfolder) to simply
1.7 seconds.  The preference permutations we've checked haven't shown
any user interface changes (except for bug 6703). 

Does anyone see problems we've missed?  What would it take to make this
patch acceptable/desirable for inclusion in imp?

Thanks in advance for your feedback.

Regards,
Ziba Scott

University of Michigan
Webmaster and Database Team


-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: Tree.php.cvsdiff
Url: http://lists.horde.org/archives/imp/attachments/20080612/db248f6c/attachment-0001.ksh