[Hampshire] Apache errors

All -

I've been having problems with an Apache. I'm not really sure where
to start investigating, particularly since the machine in question is
under moderately heavy load, and I don't particularly want to bring it
down for any extended period of time (although I will if I have to).

The server runs, each in a separate VH:

- a large chunk of mod_perl (the main site -- about 140kLOC),
- a svn/DAV server,
- a MediaWiki instance, and
- the occasional perl CGI (developers' test environments for the main site)

The mod_perl is the most used, accounting for a load average of
somewhere between 0.25 and 1 most of the time (see usage stats at
[1]). The other three services are a tiny fraction of the machine's
total usage, but are nevertheless important to the developers for the
system.

The mod_perl code works well most of the time. The CGI works
perfectly. The svn and MediaWiki, however, are extremely flaky. If
Apache has just been started, they are reliable. If it has been
running for more than a few minutes (or sometimes even a few tens of
seconds), then things start going wrong.

When svn goes wrong, most svn operations that require talking to the
server (svn up, svn ci being the most tested) yield the following:

hrm@carmilla:cities-trunk $ svn up
svn: REPORT request failed on '/svn/!svn/vcc/default'
svn: REPORT of '/svn/!svn/vcc/default': Could not read status line: connection was closed by server. (http://carmilla:8080)

and /var/log/httpd/error_log reports at the same time:

[Fri Aug 03 17:48:46 2007] [notice] child pid 25255 exit signal Aborted (6)

The failures on the wiki are either:

- a completely blank page being returned, with no error reported
anywhere, or

- Firefox attempting to download the page as a file ("You have chosen
to open index.php which is a: PHP script from: http://admin...").
The resulting file is empty.

We have another issue, which may be related, which is that some
large SQL queries made from the mod_perl code cause errors like:

DBD::mysql::st execute failed: MySQL client ran out of memory at /path/to/some/source/file.pm line 1729

The lines that cause this error are a $sth->execute on a SELECT
statement, or the $sth->fetchrow_array for the same statement, two
lines below.

My guess is that the server is running out of some kind of memory,
but I can't see what. Stats on the usage of the machine are at [1].
I've already changed the prefork configuration options to:

StartServers       4
MinSpareServers    4
MaxSpareServers    8
ServerLimit      256
MaxClients        16
MaxRequestsPerChild  4000

in an attempt to keep the memory usage down. This seems to work well
for some things -- we haven't had a bad case of swapping-to-death
since I changed them from the default. However, it hasn't affected any
of the above problems.

Does anyone have any suggestions as to how to investigate the
Apache/mod_perl system so that I can work out what's going wrong? Any
tweaks I can make, analysis I can run, daft ideas for what might be
the cause...

Hugo.

[1] http://cities.totl.net/stats/

PS. Fedora Core 5, Apache 2.2.0, mod_perl 2.0.2, perl 5.8.8.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 1C335860 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
               --- Guards!  Help! We're being rescued! ---