[imp] horde/imp migration problems, cpu bound on solaris

[Vasilios Hoffman]

Hi folks,

We've a 4 processor sun e450 (4x400mhz US-II) with 4 gb of ram.  It's been
running an older version of horde/imp for webmail.  from what I can
dig out of the current setup we're looking at:

apache 1.3.12
mysql 3.21.33
php 4.1.2
horde 1.0.11
imp 2.0.11

Nobody's really touched it for years (since 2000 it looks like) and
they've enough problems that it became ample time for an upgrade.  Note
that I'm also a recent hire, so I wasn't around when they first set it up.

I went through all the trouble of setting up a new Imp/Horde:

apache 1.3.27
mysql 4.0.12
php 4.3.1
horde 2.2.1
imp 3.2.1
turba 1.2

compiled w/ gcc 3.2.2 (first gcc 2.95.3 and then again w/ 3.2.2) and it
has the bells and whistles (imap support, ldap support, hashed php
sessions, imapproxy, whatever else I could find and throw at it that it
seemed to want).

Now, previously Horde/Imp was I/O bound -- those flat text inboxes scale
poorly and every page reload would drag lots of I/O off the server.  Was
hoping imapproxy would aleviate that somewhat, plus we switched to a
hashed /var/mail and newer imap, pop, and other servers which should help
a bit.  Ultimately going to move away from flat inboxes, but one step at a
time.

The new Horde/Imp.... CPU bound.  Ridiculously so.  Bringing up the login
page eats up 1% of the cpu, and a single login takes up to 8% of the
processor AND takes seconds to execute.  Seconds.  Like 4 or 5, not 1.
We tried a test on the machine, and about 35 concurrent connections would
cripple the CPU.  I/O was fine, memory was fine, but the load jumped over
50 and the machine was rendered unusable.

I can't believe that php or the new Horde/Imp setup is unusable with 35
connections on a 4-processor sun enterprise server.  I realize there is
more php to process... but... even 1 user is too slow!!

and it's definately php -- if I compile php as a DSO, I can watch libphp
during a login in top and truss, and it's chugging along with 5-8% of the
processor and no I/O.

So is this normal?  Is this the performance everybody else is getting?
Should I be talking to the php folks, or is it something specific to the
horde code?  Is there anything I can do to speed up the php processing?
I looked at the php readme's,etc, and I've configured the pants off it and
every other application involved and I don't think I missed anything
obvious.

Any feedback would be appreciated.  If there's some easy php profiler I
can use to find out where it's spinning it's wheels, that would be great
as I'd rather not profile it by hand.  I'm hoping it's something I can
fix, and not just all the new functionality slowing things down
significantly.

If it's just the new code, than please tell me what kind of machine you
all think could reasonably process say, ~100 active users, give or
take.... because currently, we can.  It's slow when it peaks like that,
the load can hit 10 or 12, but it's usable enough.  I can't imagine the
new horde/imp cuts that down to 0, I must be configuring something wrong,
etc.

Thanks in advance,

-V Hoffman

Vasilios Hoffman
Junior Unix Administrator
Wesleyan University
voice (860) 685-3142
fax (860) 685-2401