Forum OpenACS Q&A: Re: Google & Co on dynamic content

Collapse
Posted by Tilmann Singer on
Regarding the cloaking issue I remember that once a competitors website did that. They registered a bunch of bogus domains and created hundreds of sub-domains, which all redirected to the competitors main page normally, but when called with a user agent of google (I actually tried it myself), they returned a list of links to all the other bogus websites instead, thus trying to fool the page rank algorythm.

I mailed google and they said they were working on techniques to automatically detect and ban such sites, and banned the offending one manually. That was a few years ago, and the source for my assumption that returning different content based on googlebot user-agent header might be a bad idea. It might as well be though that they have a way of distinguishing between sites that try to fool the pagerank mechanism and those that only return more search engine friendly content, although I can't imagine how that would work 100% reliably.

Anyway, speculations about google behaviour could be continued endless I guess, but that is not my intention. Ok, a last one: I think if we remove the restriction in robots.txt (and the site doesn't fall over when being indexed) then google will index the full site including all postings after some time, and neither query variables nor the paginator on the forums index page will scare it away.