[horde] memcache session handler

[liamr@umich.edu]

> Are there any host-based or network-based firewalls that would be
> terminating network connections?

Nothing that changed in when it started breaking.  We've got IP Tables 
installed, and are limiting connections on port 11211 to the subnet 
that the machines are on.

> Any chance you are running out of either available local network 
> ports (cat /proc/sys/net/ipv4/ip_local_port_range to see how many you 
> are
> configured for) or that you have a bunch of connections tied up in
> FIN_WAIT state?

At this point, I can't say.  We rolled back to the single memcache 
server configuration that has been working since last summer, and I 
don't think I'll be allowed to move off of that until I've tested 
things more thoroughly.  I didn't check netstat or 
/proc/sys/net/ipv4/ip_local_port_range during my little session handler 
crisis.  I was too busy trying to figure out what I'd broken. ;)

> Are you using persistent connections?

Yes.  We've got lots of children on each of the boxes.  Should we not 
use persistent connections?

> Finally, any chance that you simply exhausted the small amount of memory
> you allocated on each box?

We had allocated 512mb on each of the 15 machines.  So, that was a 
combined cache of like 7.5gb.  We'd been using the old memcached 
session handler (that only supports one server) w/o problem since last 
summmer, and that machine only has 2GB of cache allocated.  That driver 
also doesn't have a concept of persistent connections.

Liam