Quantcast

Enough is enough

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Enough is enough

Robin Dunn
Over the past few days while I've been working on moving the mail lists
I've noticed that there are some apache processes that are also adding
significantly to the system load.  The process would spend a lot of time
in the 'D' state and sometimes take a minute or two to complete the
request.  I later discovered that it was a SVN operation in each case.
(Not all of them, just whatever causes SVN to submit a REPORT webdav
request to the web server.)

I don't know if there is a hardware problem on the host node or if
whatever it is using for disk storage is failing, or if we've passed
some threshold in SVN where trolling through the data files for 60,000+
revisions to perform SVN operations is just too much for the
environment, but it is clear that anything on the system that is disk
intensive is suffering and it is getting worse. I am no longer convinced
that offloading the mail lists will be enough.

Today the VPS went down again, but this time it wasn't just shut down by
the hosing provider like normal.  This time the server still seemed to
be up, but too slow to process any requests or conenctions.  My existing
SSH sessions were never broken, but the displays never updated, and new
ssh connection attempts would start to login but then timeout.  I also
wasn't able to access the host's control panel page without timing out.
   So my guess is that there was either a DoS attack or some sort of
cascading hardware failure on the node.  The hosting provider hasn't yet
reported on the source of the problem, but I'm guessing the latter.  If
that's true then it's possible that they've fixed the problem and then
our stuff will run better without so much unexplained load, and although
things seem a bit better so far since the VPS came back up, I'm not yet
convinced.

In either case, I'm tired of struggling with this and if things haven't
been repaired by the current hosting provider then I think that we need
to come up with some other solution, (or maybe even if they have.)  We
can either do like we're doing for the mail lists and find some service
provider that specializes in SVN and Trac hosting, like
http://svnrepository.com/ or somebody on the list at
http://trac.edgewall.org/wiki/CommercialServices.  Or we can balance the
load by moving them to some other wx developer's server like we do with
the wiki, docs, and forum.  The former means that we may have troubles
using our trac customizations and plugins, the latter means we may run
into this problem again if it's an svn resource issue.  Any other ideas
or suggestions?

--
Robin Dunn
Software Craftsman
http://wxPython.org

_______________________________________________
wx-dev mailing list
[hidden email]
http://lists.wxwidgets.org/mailman/listinfo/wx-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Enough is enough

Grant Limberg-2
On May 17, 2009, at 10:35 PM, Robin Dunn wrote:

> Over the past few days while I've been working on moving the mail  
> lists I've noticed that there are some apache processes that are  
> also adding significantly to the system load.  The process would  
> spend a lot of time in the 'D' state and sometimes take a minute or  
> two to complete the request.  I later discovered that it was a SVN  
> operation in each case. (Not all of them, just whatever causes SVN  
> to submit a REPORT webdav request to the web server.)
>
> I don't know if there is a hardware problem on the host node or if  
> whatever it is using for disk storage is failing, or if we've passed  
> some threshold in SVN where trolling through the data files for  
> 60,000+ revisions to perform SVN operations is just too much for the  
> environment, but it is clear that anything on the system that is  
> disk intensive is suffering and it is getting worse. I am no longer  
> convinced that offloading the mail lists will be enough.
>

What about migrating away from svn to a distributed vcs.  That will at  
least transfer trolling through the repository history to the client  
side and should end up with significantly less load on the vcs server  
duties.

Grant Limberg




[hidden email]
http://www.cedrus.com/


_______________________________________________
wx-dev mailing list
[hidden email]
http://lists.wxwidgets.org/mailman/listinfo/wx-dev

unknown.gif (3K) Download Attachment
PGP.sig (202 bytes) Download Attachment
Loading...