My understanding is that Google and other search engines don't index urls that redirect. I think its a way they use to minimize duplicate data in their indexes. Noncompliant html HEAD code is another critical factor that stops googlebot.
Tired bots? My experience with staticly generated websites containing thousands of pages and 5 levels suggests that the bots don't get tired of specific, unique information.
The openacs static-pages package changes the page depending on the requesting client. As pointed out earlier in this thread, some searchengines, such as google, attribute this behavior to manipulation and may not index the page or site accordingly.
Is it possible to configure static-pages so that urls are presented as static pages in all cases? Perhaps the quick and dirty method may be to tweak the registered robots list to none? If not possible, then a package needs to be written to address these issues.
The package should interpret static looking pages and convert/present static looking urls for the domain served. Also, in a best case scenario, there should be a way to store requested, cached dynamic pages as static files so that the server can keep up with bot requests. Not delivering the pages at static-page speed may result in lost opportunities for indexing --given that there is limited time between index updates.
