Page 1 of 2 12 LastLast
Results 1 to 10 of 12
  1. #1
    Senior Member azjoe's Avatar
    Join Date
    May 2005
    Location
    USA - Arizona
    Posts
    1,543
    Thanked: 27

    Default Site outages today

    We are working on isolating the reason for the outages today. Until we understand the cause and effect, there may be more hiccups, so please be patient.

  2. #2
    I'm Back!! Jonedangerousli's Avatar
    Join Date
    Mar 2007
    Location
    Jacksonville, FL
    Posts
    1,249
    Thanked: 1

    Default

    There have been outages every afternoon this week. Today's were a 403 error, which is the first time I've seen that.

  3. #3
    Super Shaver xman's Avatar
    Join Date
    May 2005
    Location
    Lotus Land, eh
    Posts
    8,194
    Thanked: 622

    Default

    You do good work Joe.

    X

  4. #4
      Lynn's Avatar
    Join Date
    May 2005
    Location
    St. Louis, Missouri, United States
    Posts
    8,454
    Thanked: 4941
    Blog Entries
    2

    Default

    Thanks Joe!!!

    You'se guys is da Greatest!!!!

    Lynn

  5. #5
    Senior Member dwessell's Avatar
    Join Date
    Apr 2005
    Location
    Asheville, North Carolina, United States
    Posts
    1,708
    Thanked: 328

    Default

    I thought I'd give everyone a quick update.. We THINK we we're attacked by a vicious swarm of Yahoo spiders.. Joe did some research and it looks like this might be an issue.. Traffic has been way up around here lately, so it's probably a combo of the two... Or mabye not..

    Joe and I are watching it pretty closely, and will try and keep everyone updated..

    dw

  6. #6
    Senior Member WireBeard's Avatar
    Join Date
    May 2007
    Location
    Denver, Colorado
    Posts
    947
    Thanked: 92

    Default

    Have you looked into a RAID array? (Get it? Spiders, RAID?)

    ...Ok, weak...I'll shut up now...)

    Greg

  7. #7
    As-Yet-Unscarred Newb
    Join Date
    Jul 2007
    Location
    Swift Current, Saskatchewan
    Posts
    82
    Thanked: 1

    Default

    Quote Originally Posted by WireBeard View Post
    Have you looked into a RAID array? (Get it? Spiders, RAID?)

    ...Ok, weak...I'll shut up now...)

    Greg


  8. #8
    Senior Member azjoe's Avatar
    Join Date
    May 2005
    Location
    USA - Arizona
    Posts
    1,543
    Thanked: 27

    Default

    Well, hopefully I'm not speaking too soon... but things seem to be running better today, as far as I can see.

    We believe there were actually two issues plaguing us... a shaky database server and the load being placed on us by the Yahoo Slurp Spiders. Our Hosting company has admitted they had some problems and claim they are resolved. The spiders are more problematic, however.

    For those unfamiliar with these "spiders", they are search engine programs which are constantly accessing our site, collecting data so that people searching for razor terms will get results referenced to SRP. Until recently there were seldom more than 5-7 spiders working our site (Google, MSN, ASK, Yahoo, etc.) at any given instant... usually one per search engine. (If you've ever wondered why there are usually a few "guests" on the site, these spiders show up in that count, along with legitimate guests. )

    Anyway, Yahoo recently changed their methodology and have unleashed a huge number of spiders (their intent being, so I've read, to recapture Search engine supremacy from Google). I've read of cases where forums such as ours have been invaded by greater than 100 simultaneous instances of the Yahoo spider. I've personally encountered as many as 37 simultaneously working our site, but there are indications that there have been much higher numbers than that at times.

    Why is this bad... well, spiders place a higher load on the resources of the site (memory, database accesses, connections, etc.) than a member would... after all, they're accessing threads and sucking up posts at electronic speeds... they don't pause to read as a human would, LOL. One of our recent outages was, in fact, a result of our Host shutting us down because we were hogging too many resources.

    Yahoo has been severely criticized by many webmasters, but at this time they seem not to care... I guess they're singularly focused on catching Google. Thus, many sites have simply locked them out completely... denying access to the Yahoo spiders altogether.

    We, however, believe there's benefit to allowing them to keep scanning our system. We want people searching for razor related terms to find SRP in the results. Yahoo is still a widely used search engine, so it seems foolish to lock them out. Thus, we've decided to try slowing them down by requiring them to delay some number of seconds between accesses. Granted this won't help much if they descend on us with hundreds spiders at once, but it's worth a try. If the problems continue, and seem to be Yahoo spider related, we can always lock them out completely.

    David and I are keeping our fingers crossed!
    Last edited by azjoe; 08-19-2007 at 11:20 PM. Reason: fix typos

  9. #9
    I'm Back!! Jonedangerousli's Avatar
    Join Date
    Mar 2007
    Location
    Jacksonville, FL
    Posts
    1,249
    Thanked: 1

    Default

    I'd lock them out. In my sometimes humble opinion having a site that the members can use should take priority over a second rate search engine.

  10. #10
    I'm Back!! Jonedangerousli's Avatar
    Join Date
    Mar 2007
    Location
    Jacksonville, FL
    Posts
    1,249
    Thanked: 1

    Default

    Site down for ~5 hours here today

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •