[Tickets #10773] Crippling Memory Usage for certain Log entries

[bugs at horde.org]

DO NOT REPLY TO THIS MESSAGE. THIS EMAIL ADDRESS IS NOT MONITORED.

Ticket URL: http://bugs.horde.org/ticket/10773
------------------------------------------------------------------------------
  Ticket             | 10773
  Created By         | Michael Slusarz <slusarz at horde.org>
  Summary            | Crippling Memory Usage for certain Log entries
  Queue              | Horde Framework Packages
  Version            | Git master
  Type               | Bug
  State              | Unconfirmed
  Priority           | 3. High
  Milestone          |
  Patch              |
  Owners             |
------------------------------------------------------------------------------


Michael Slusarz <slusarz at horde.org> (2011-11-16 22:21) wrote:

Due to the design of the Log class, we ALWAYS generate the log message  
and send it to the Log class, even if the log level means it will  
never be output to any handler.

However, this is crippling in the case of large log outputs.  Not only  
do we need to allocate memory for the string itself, but the memory  
needs are at a minimum doubled since we generate a debug_backtrace()  
in the Horde Core log handler, and this debug_backtrace will include  
the entire contents of the log message.

My example: attaching a 2 MB file to an email message is taking  
upwards of 80+ MB /regardless/ of the Log setting.  Totally disabling  
logging in the DB library lowers the memory usage to around 50 MB.   
This 30 MB difference is because the DB logger is logging the ENTIRE  
query string sent to the server.  In my case, because I am using  
postgres, the data string size is quadrupled due to bytea escaping so  
the log message is 9MB.

Even for DEBUG logging, this is ridiculous.  And for those not using  
debug logging, they have to pay this penalty for no reasons.  Several  
thoughts:

1) A way to pre-query the logger to see if we should even go through  
the trouble of creating a log message.  Useful in cases where log  
message generation is potentially expensive.
2) Remove query logging in Db library, even for DEBUG level.  Add  
parameter to specifically request full query logging.