Thread: Site outages today

We are working on isolating the reason for the outages today. Until we understand the cause and effect, there may be more hiccups, so please be patient.

There have been outages every afternoon this week. Today's were a 403 error, which is the first time I've seen that.

You do good work Joe.

X

Thanks Joe!!!

You'se guys is da Greatest!!!!

Lynn

I thought I'd give everyone a quick update.. We THINK we we're attacked by a vicious swarm of Yahoo spiders.. Joe did some research and it looks like this might be an issue.. Traffic has been way up around here lately, so it's probably a combo of the two... Or mabye not..

Joe and I are watching it pretty closely, and will try and keep everyone updated..

dw

Have you looked into a RAID array? (Get it? Spiders, RAID?)

...Ok, weak...I'll shut up now...)

Greg

Originally Posted by WireBeard

Have you looked into a RAID array? (Get it? Spiders, RAID?)

...Ok, weak...I'll shut up now...)

Greg

Well, hopefully I'm not speaking too soon... but things seem to be running better today, as far as I can see.

We believe there were actually two issues plaguing us... a shaky database server and the load being placed on us by the Yahoo Slurp Spiders. Our Hosting company has admitted they had some problems and claim they are resolved. The spiders are more problematic, however.

For those unfamiliar with these "spiders", they are search engine programs which are constantly accessing our site, collecting data so that people searching for razor terms will get results referenced to SRP. Until recently there were seldom more than 5-7 spiders working our site (Google, MSN, ASK, Yahoo, etc.) at any given instant... usually one per search engine. (If you've ever wondered why there are usually a few "guests" on the site, these spiders show up in that count, along with legitimate guests. )

Anyway, Yahoo recently changed their methodology and have unleashed a huge number of spiders (their intent being, so I've read, to recapture Search engine supremacy from Google). I've read of cases where forums such as ours have been invaded by greater than 100 simultaneous instances of the Yahoo spider. I've personally encountered as many as 37 simultaneously working our site, but there are indications that there have been much higher numbers than that at times.

Why is this bad... well, spiders place a higher load on the resources of the site (memory, database accesses, connections, etc.) than a member would... after all, they're accessing threads and sucking up posts at electronic speeds... they don't pause to read as a human would, LOL. One of our recent outages was, in fact, a result of our Host shutting us down because we were hogging too many resources.

Yahoo has been severely criticized by many webmasters, but at this time they seem not to care... I guess they're singularly focused on catching Google. Thus, many sites have simply locked them out completely... denying access to the Yahoo spiders altogether.

We, however, believe there's benefit to allowing them to keep scanning our system. We want people searching for razor related terms to find SRP in the results. Yahoo is still a widely used search engine, so it seems foolish to lock them out. Thus, we've decided to try slowing them down by requiring them to delay some number of seconds between accesses. Granted this won't help much if they descend on us with hundreds spiders at once, but it's worth a try. If the problems continue, and seem to be Yahoo spider related, we can always lock them out completely.

David and I are keeping our fingers crossed!

I'd lock them out. In my sometimes humble opinion having a site that the members can use should take priority over a second rate search engine.

Site down for ~5 hours here today