Forum OpenACS Development: Re: One week of using naviserver on openacs.org

Collapse
4: Re: One week of using naviserver on openacs.org (response to 1)
Posted by Gustaf Neumann on 10/13/13 02:22 PM
Dear all,

Some updates: Over the last weeks, we worked on a cleanup of the code (mostly acs-core and the packages used on openacs.org), removing calls to deprecated functions and replacing old-style Tcl with the feature set available in Tcl 8.5. Furthermore, the actual code on openacs.org implements ad_proc based on nsf::proc when available. nsf::proc is part of nsf/XOTcl2 and provides a c-based argument parser as a replacement of the classical tcl-level parser in ad_proc.

Other changes address some infrequent very long calls stemming from (a) dns hangs, (b) sometimes extremely slow calculation of the summarizing function in the listing of the search results, and (c) from a broken entries in forums on openacs.org (one entry had a tree_sortkey of NULL, returning a "thread" of all postings. The last change was just a site bug.

As a consequence of these changes, the average run time of requests on openacs.org improved by about 30% since the posting above, namely from 0.0676 to 0.0472 seconds per request

requests 136473, queued 10 (0.01%), spooled 48751 (35.72%)
avg queue time 0.0001s, avg filter time 0.0032s, avg run time 0.0472s

The times reported by munin (moving average) are now in the range of 50ms down from 80ms in August and down from 325ms before (more than a factor of 6 faster). Outlines count more in the calculation of munin, since a slow call in a 5 min interval with few other requests counts more than e.g. in a period with frequent fast requests.

The improvements might be even better, since i was dropping in the setup of openacs.org some very fast calls (using large expire times for "resources"); these fast but unnecessary calls would improve the numbers further.

I am quite happy with these results. The frequency of the changes in the oacs-5-8 branch from my side will go down, since i am mostly through with the tcl-8.5 instruction changes (and the courses of the semester start).

Best regards
-gustaf neumann

Collapse
5: Re: One week of using naviserver on openacs.org (response to 4)
Posted by Gustaf Neumann on 10/24/13 11:54 AM
Just to keep statistics about openacs.org on one place: on https://openacs.org/forums/message-view?message_id=4099229 we had:
- OpenACS.org runs the freshest code from the oacs-5-8 branch and uses not only acs-core, but as well several packages, in which we fixed many bugs not contained in the bugtracker. These changes led to much lower numbers of error entries in the error.log: 100 days ago (using aolserver + OpenACS 5.5 + pg 8.3.5) we saw abut 17.000 errors per day, last week, we were at 3000, yesterday at 428 (many of the remaining errors are from from invalid urls, etc.).
We are still improving on openacs.org: yesterday, we had e.g. 167 errors in the error.log. Most of these errors look like caused from broken web clients, sending requests with unresolved entities in hrefs (like eg. "..../?project_id=56106&f_action_20=8949&f_state=7" instead of "..../?project_id=56106&f_action_20=8949&f_state=7"). ad_page_contract complain in such cases about "nonstandard variable names".

On google's webmaster tools, we are down from 2966 errors (on Aug 6th) to 0 errors (since Sept 21th).

Collapse
6: five months of using naviserver on openacs.org (response to 5)
Posted by Gustaf Neumann on 01/13/14 02:15 PM
It is now nearly 5 months since we switched on OpenACS.org to NaviServer and pg 9.2. The following year-view shows how nicely and stable OpenACS.org runs with the newest snapshot of OpenACS 5.8.0 from cvs (oacs-5-8 branch), which will become OpenACS 5.8.1 in the future.

In Aug. and Sep. we did some more configuration and update work which are visible in the diagram (e.g. concerning the non-core packages used on this site). Since Oct. the response times are very stable, serving about the same load as one year earlier. Average response time of the server is below 50ms, although several fast requests were eliminated (e.g. by setting expiration).

all the best
-gn

Collapse
7: Re: five months of using naviserver on openacs.org (response to 6)
Posted by Jim Lynch on 01/15/14 06:04 AM
Heya Gustaf,

Did you find out what was going on (looking at by-year graph) in the first week of July, where the graph looked like it was peaking?

-Jim

Collapse
8: Re: five months of using naviserver on openacs.org (response to 7)
Posted by Gustaf Neumann on 01/15/14 09:37 AM
At least part of the problem was running openacs.org with OpenACS 5.6 and pg 8.4 for a short time. Since PostgreSQL 8.3 reached EOL in Feb 2013, we did the upgrade to 8.4 in July, hoping that the site is small enough and the machine is fast enough not to face the well known problem with the query optimizer changes in pg 8.4 (see e.g. https://openacs.org/forums/message-view?message_id=3804765). This bad performance was the final driver to push towards OpenACS 5.8.0 addressing these problems.