Forum OpenACS Development: ns_pools configuration with URL mapping

Collapse
1: ns_pools configuration with URL mapping
Posted by Malte Sussdorff on 09/26/07 08:37 AM
I have a fairly expensive page (taking around 10 seconds, most of it DB queries). Sadly that page is very popular and very dynamic (the page takes a category as a parameter and we have around 6000 categories currently in the system). To make matters worse, google just LOVES this page. So along comes a spider and the whole system dies, because the DB is not able to cope with the storm.

My idea now is to have a special thread pool for this page which allows a maximum of two threads to open. This should prevent the spider from hitting the DB too heavily (or am I wrong in this assumption, if we have 20 requests only two of them are executed at the same time, which would be fine with the DB).

Sadly I could not find how to map a pool to a specific URL 😊. Any clues?

Collapse
2: Re: ns_pools configuration with URL mapping (response to 1)
Posted by Gustaf Neumann on 09/26/07 09:40 AM
if it is an option for you, use an reverse proxy to direct queries from spiders to a single server with only a few connection-threads / db-connections configured.
Collapse
3: Re: ns_pools configuration with URL mapping (response to 1)
Posted by Dave Bauer on 09/26/07 12:58 PM
Why exactly are you listing 6000 items on one page?

Why would you not

1) Fix the query
2) Limit the results displayed on one page
3) Cache the query results

Which are all simpler and easier ways to solve this problem?

Heck its even easier to use template::adp::include to "save" the page, write it to the filesystem as a static adp then include that into the page than what you are trying to do. Static pages are quite speedy.

Collapse
4: Re: ns_pools configuration with URL mapping (response to 1)
Posted by Dave Bauer on 09/26/07 12:59 PM
Ah sorry, not enough coffee. There are 6000 options.

Well in that case, I still stand by my static page or caching answer. That is, cache the include, that has the category specific information, either in memeory or on disk.

Collapse
5: Re: ns_pools configuration with URL mapping (response to 1)
Posted by Malte Sussdorff on 09/26/07 07:12 PM
Thanks for the advise Gustaf. That seems like a plausible idea, but for the time being I actually got around this specific problem by not using the dotfolio method of getting URLs of users (yes, it is a long way from the categories page, but as part of the categories page a link to the user's URL is listed, and this was using site nodes to determine the URL which again was using lazy site node caching which, if you have 70 users displayed on a page results in quite some site node loading.)

Okay, my problem is solved and I will just use NGINX to redirect to a specific server for the page. I should have thought about this earlier, after all I am doing this all the time. Not enough coffee this morning with me either.

Collapse
6: Re: ns_pools configuration with URL mapping (response to 1)
Posted by Tom Jackson on 09/26/07 08:40 PM
Sadly I could not find how to map a pool to a specific URL 😊. Any clues?

Are you not using the new modules/tcl/pools.tcl file?

The config is like this: 
# Single Threadpool
# pool = fast

ns_section "ns/server/${server}/pools"
    ns_param $pool "$pool pool"

ns_section "ns/server/${server}/pool/$pool"
ns_param   maxconnections    100
ns_param   minthreads        2 ;# default = 0
ns_param   maxthreads        10
ns_param   threadtimeout     120
ns_param   map               "GET /*-thumb.jpg"
ns_param   map               "GET /images/*-thumb.jpg"
ns_param   map               "GET /test.html"

But you can always do it manually:

ns_pools register poolName serverName method url

In the config, both method and url are combined, but the url part can still be a specific url with no glob pattern. I don't believe that it is important that OpenACS uses a virtual url, it is just based upon the request url.