Open Source Host - what's the deal?

First, I noticed that Open Source Host is no longer on the "Zen Certified Hosting" page for some reason, and I can't determine when it was removed. We use OSH for our ZC site. But my real problem is that this year our store experienced some significant sales decreases and I couldn't figure out why, until today. I found the following in my .htaccess file:

#Your site is being crawled by search engines and causing heavy load on the server. Rather than disable access to your entire site we will deny the following search engines. We will remove this portion when the load subsides. You may consider creating or altering your robots.txt or auditing your site's code for performance tuning such as the use of MySQL's pconnect function.
order allow,deny
deny from .crawl.yahoo.net
deny from .search.msn.com
deny from .crawl.##########.com
deny from .amazonaws.com
deny from .googlebot.com
deny from .cuil.com
deny from .yandex.ru

This was inserted into .htaccess in the middle of May. We're now getting into the second week of June. What in the world? We weren't told that they did this, and then they don't remove it? I checked Google's crawl stats for our site and it looks like OSH may have done something similar back in March-April.

Anyone else use OSH and experience a similar problem?

Phid:

This was inserted into .htaccess in the middle of May. We're now getting into the second week of June. What in the world? We weren't told that they did this, and then they don't remove it? I checked Google's crawl stats for our site and it looks like OSH may have done something similar back in March-April.
That kind of behaviour is inexcusable. I have never used OSH but if a host did that to me I would change hosts ASAP. I would suggest you change hosts and let OSH know how disappointed you are with the treatment you have received.

Regards,
Christian.

CJPinder:

That kind of behaviour is inexcusable. I have never used OSH but if a host did that to me I would change hosts ASAP. I would suggest you change hosts and let OSH know how disappointed you are with the treatment you have received.

Regards,
Christian.

I actually dissagree. Some search engines, Yandex in particular ignore the robots.txt file and crawl sites often and heavy ... to the extent that it can (and does) seriously affect server performance.

If reckon OSH were doing the OP a big favor by taking care of the source of the problem by modifying the .htaccess file rather than doing what most other ISP's would do, namely, simply dissabling the account and letting the user figure out exactly why and how to mitigate the problem.

Kudo's to OSH for this one.

Cheers
Rod.

ps. I have no association with, and have ever had any dealings with OSH. I'm not suggesting they are a good host or a bad host based on this single incidence.

Blocking the majors like Google, Yahoo, MSN without first discussing the matter with the client ?? Excuse me!!! :shocking:

I too would change hosts ASAP - and tell my friends ....

no longer on the "Zen Certified Hosting" page for some reason

deny from .googlebot.com
That would be a good enough reason.

g1smd:

That would be a good enough reason.

Doubtful.

The most likely reason is that one of the requirements to be listed as a certified host is that "Certified Hosts provide financial support to the Zen Cart project"

I suspect that this financial support has lapsed.

Also, once again I shall re-iterate my view.

Do you folks really think OSH really did a disservice to the OP by restricting access to search engines (one of which was the cause of excessive usage) rather than simply disabling the account and letting the user figure out why and what to do about it?

What is better.... a completely inaccessible website, or a website that allows everyone other than robots?

Given the choice, I'd go for the latter.. every time.

Cheers
Rod

ps. I do appreciate that blocking google et al along with the offending crawler is/was a bit of an overkill, but that doesn't really change the options any.
pss. I also agree that not notifying the OP is/was inexcusable - but we do only have the OP's word on this. OSH may well have sent notification that wasn't received (or even ignored).

RodG:

If reckon OSH were doing the OP a big favor by taking care of the source of the problem by modifying the .htaccess file rather than doing what most other ISP's would do, namely, simply dissabling the account and letting the user figure out exactly why and how to mitigate the problem.
They blocked Google from spidering his site and did not tell him about it! That is pretty poor customer service in my opinion.

Regards,
Christian.

I agree with most of the posts above. Whether this was a good idea or not (and rarely would blocking Google be a good idea for e-commerce sites), to do so without actually telling the client is completely unacceptable.

In this case Phid has been smart enough to spot something being wrong, and track down the problem. But even then that was time that could have been better spent developing his business than debugging hosting issues.

kuroi:

I agree with most of the posts above. Whether this was a good idea or not (and rarely would blocking Google be a good idea for e-commerce sites), to do so without actually telling the client is completely unacceptable....

Thanks kuroi, IMHO if there is a problem then the options of solving the problem should first be discussed with the customer before taking such drastic action.

RodG:

What is better.... a completely inaccessible website, or a website that allows everyone other than robots?

pss. I also agree that not notifying the OP is/was inexcusable - but we do only have the OP's word on this. OSH may well have sent notification that wasn't received (or even ignored).Assuming for the sake of argument that they did notify us (I did not receive any notification, and the site owner said he did not either), the fact remains that OSH said they would remove the block after the server load stopped. Yet, they did kept it blocked for three weeks. I would not have minded so much if OSH put a block in for a few hours, or even a day, but to fail to remove the block is what really irritates me. I only deduced the problem on my own after I googled our store's name and our store didn't come up as it used to.

Another strange thing is that our site only gets perhaps a few thousand visitors at most per day, and it's a normal gift-type of store. Nothing changed on the site from our end which would have caused an excessive increase in traffic.

In the past, Open Source Host used to be stellar. I'm not sure that Scott (I think that's his name) is still in charge there, and if he left that may explain why it's declined.

... plus if they can't handle being spidered by search engines, they're a pretty useless host, and even if they had a temporary problem with a specific spider, blocking all the important ones to the detriment of the site's traffic is a pretty inept solution.

Difficult to how they could come out of this looking anything other than very poor.

In the past, Open Source Host used to be stellar. I'm not sure that Scott (I think that's his name) is still in charge there, and if he left that may explain why it's declined.

There were management changes quite some time ago.

Phid:

Another strange thing is that our site only gets perhaps a few thousand visitors at most per day, and it's a normal gift-type of store. Nothing changed on the site from our end which would have caused an excessive increase in traffic.

I can explain this "strange thing" for you in one word. "Yandex".

Cheers
Rod

RodG:

I actually dissagree. Some search engines, Yandex in particular ignore the robots.txt file and crawl sites often and heavy ... to the extent that it can (and does) seriously affect server performance.

If reckon OSH were doing the OP a big favor by taking care of the source of the problem by modifying the .htaccess file rather than doing what most other ISP's would do, namely, simply dissabling the account and letting the user figure out exactly why and how to mitigate the problem.

Kudo's to OSH for this one.

Cheers
Rod.

ps. I have no association with, and have ever had any dealings with OSH. I'm not suggesting they are a good host or a bad host based on this single incidence.

I will have to disagree with you here.... OSH should have told the client first.... It probably would have been best if the site went down.... sounds like these people need to move to a dedicated hosts.... getting thousands of visitors a day I would be happy with a steady thousand right now and I am on a dedicated host.