Frequent 'Server errors'

rileydog22 wrote: Reporting specific error messages is probably useful for M@X, Ivan.  It's a lot easier to fix a problem when you know what it is.

rushfan4 wrote: I don't know if this has anything to do with some of the problems, but I have noticed the last couple of days around this time, that the number of on-line forum users spikes dramatically.  I just checked and there are 256 users, most of which are not members.  I'm not sure if these are hackers or i-robots that are logging in to threads to cause the site to bog down, or something entirely different.  I don't know if this helps or not, but it was something that looked rather strange to me.

Not sure what the 'Service Unavailable' root cause is, but I wonder how much of the forum slowness and the SQL errors are attributable to CPU usage by searchbots. We've always had a lot of traffic from these crawlers, but the last few months or so the percentage of bot logins has risen to where it is usually 90%-plus of the number of logged users at any given time. I've sat and refereshed the users screen several times on different occasions to see if these guys are using reasonable policies for polite cataloging, and most of them don't seem to be. I would not be at all surprised if 98% or more of the site's traffic is due to these bots. And since virtually everything on this site is stored in a SQL database and uses ASP code, I would assume that means a lot of CPU usage.

One option (if this hasn't been explored already) would be to restrict searchbots from the forums. I know M@x needs max exposure (no pun intended) for traffic count purposes as well as the 'free' advertising that search engines offer, but I would think this would apply more to the discog section of the site than the forums. The forum data isn't as attractive for a few reasons:

- it is more volatile (changes constantly, which just perpetuates the searchbot traffic);
- is of limited value to the outside world (ever come across a snippet of a ProgArchives forum thread while Googling for something? These usually fall far down the list on searches and are also usually stale and/or not really appropriate to whatever you were searching for);
- there doesn't seem to be any logical rhyme or reason on how these things search or how long their host sites retain the information in their active engine databases.

I'm sure M@x has explored all this fully, but I have wondered for quite some time how invasive these bots are to the site as far as bandwidth drain on the server(s), database connections, and CPU usage. I suppose they help keep the hit counts up at least....

I'm afraid there are ongoing issues. M@x is working very hard to resolve them, but they are proving difficult to eradicate.

I was getting that same error yesterday in Firefox at work. 

James wrote: I've sold my Firefox error.

Thanks James for this clue ...

Don't know why exactly - but it works better when deleting the old cookies .. even if there were build new ones soon ...

Probably it's a coincidence but I'm browsing through the forum actually and  no new errors are occurring after deleting the old cookies ... let's monitor this for a while ...


Edited by Rivertree - August 01 2008 at 19:50

Ricochet wrote: It's good to see you all folks again. Yesterday I couldn't access any links except the front page and front forum page at all.

Yesterday evening was probably the worst period I've seen on the forums for timeout errors.

... I've posted this review some minute ago http://www.progarchives.com/Review.asp?id=178898 but it has disappeared from the home page... what's the problem?