Forum OpenACS Q&A: Re: Approach to Google-optimizing

Collapse
Posted by Jeff Davis on
I think the reason we should be more careful about noindex,nofollow is this statement from google:
1. Reasons your site may not be included.
Your pages are dynamically generated. We are able to index dynamically generated pages. However, because our web crawler can easily overwhelm and crash sites serving dynamic content, we limit the amount of dynamic pages we index.
So since they limit the amount of dynamic content they spider you want to make what they do spider unique to increase coverage (and to lower the burden on your own server).

Changing everything to have pretty urls would remedy the spidering scope issue (although it would still leave google pulling down an order of magnitude too many pages for things like bug tracker).