[imp] Server Farms..

[Oliver Kuhl]

Hi Michael!

Michael Bellears wrote:

> We currently run one (very!) overburdened IMP server (Currently wearing
> many hats - qmail/courier-imap/anti-virus/spamassassin/webmail on Debian
> 3) - So we have decided to migrate to a server farm.
> 
> We have been looking at the following Loadbalancers(If anyone has
> personal experience with either, or other recommendations - I would be
> grateful for any input):
> 
> ServerIron-XL -
> http://www.foundrynet.com/products/webswitches/serveriron/
> CoyotePoint - http://www.coyotepoint.com/equalizer.htm
First of all I cannot tell you something about real loadbalancers. But I 
can tell you something about our setup, which works via dns-round-robin. 
We have currently 100.000+ users able to use imp. We devided every 
service, so we have several cyrus-mailservers, 2 imp-webmail-servers 
(and another one for developing/testing) and mysql-servers on seperate 
machines.

> 1. We currently run MySQL for IMP prefs/vpopmail users/spamassassin
> prefs etc - Is it advisable to have each "real" server run it's own
> MySQL - And have each in a master/slave setup?
We have our own mysql servers holding the prefs. The load is not so much 
so it is a good idea to but your db-stuff on a stand-alone db-server. 
The advantage is scalability: you can easily add another imp-server 
without thinking about how to handle mysql-replication with 3 or 4 
servers. And if the db-server is too slow, simply create a pair with 
master-master-replication and dns-round-robin. That will easily handle 
some 100.000 users prefs. ;-)

> 2. What is the recommended method to synch config files on all "real"
> servers (Eg. Httpd.conf, horde/imp config files etc?) - Have only one
> server that admins connect to for mods, then rsync any changes to the
> other servers?
Our imp-servers are quite the same hardware. So I have a testing-machine 
with a little shell-script to sync all necessary files with the 
production machines using rsync.

> 3. What about logfiles - We would have all users mail etc on an NFS
> share - Can you do the same for logfiles?(Or do you get locking issues?)
> - From a statistical aspect, it would be a pain to have to combine each
> "real" servers logfiles, then run analysis. Also from a support
> perspective - How are support personnel supposed to know which "real"
> server a client would actually be connecting to in order to see if they
> are entering a wrong username/pass etc?
With our two imp-servers, we actually have to grep through the logs on 
both machines "by hand". NFS would work.


> 4. This is OT, but I thought I would ask anyway - Once we have a "real"
> server setup they way we want - What imaging software can be used (Quick
> recovery in the event of server failure (Eg. If Total rebuild is
> necessary), and also simple for addition of new "real" server)?. I would
> love a utility that can create a bootable CD from our SOE, giving us the
> ability to just place this CD into the new server, power up and in a
> couple of minutes we have a fully functional server ready to be placed
> into the server farm.
> 
> I have briefly looked at the following :
> 
> http://www.mondorescue.org/
> http://www.systemimager.org/
Interesting links! We are looking for something like that, too. Last 
year, there was no prob with our servers. But as database- and 
imp-servers are twins, we can act quickly by removing the failing 
machine in the dns (having a small TTL, of course). If there is a bigger 
problem, the imp-testserver can easily be configured as production 
machine. That does not sound perfect, but we can be back at service in a 
couple of minutes. Maybe a bit slower, but who matters. If your machines 
perform well normally, they should be able to take over the work of a 
failing machine without beeing completely slow.

In our case, the two imp-servers (Dual PIII at 1266, 1G RAM) have a load of 
  0.4 by average and maybe sometimes 0.9 at "rush hour". They only get 
heavy loaded (~10) while rotating the apache-logs every morning. So one 
of them should be able to handle the work of the other one at failure.

Ah, I forgot something to mention: Users get distributed to one of the 
imp-servers by dns-round-robin, as mentioned above. But they will be 
redirected to one of the servers permanent before login. So we don't 
have a session-problem here. Disadvantage: I a server fails a user has 
to login again. Advantage: Much cheaper than a hardware-loadbalancer.

So you have to decide, how available your webmail has to be.

Good luck,

	Oliver.