Forum OpenACS Q&A: Re: Down to < 60 seconds

Collapse
15: Re: Down to < 60 seconds (response to 10)
Posted by Gustaf Neumann on
wouldn't it be more straightworward and faster to search only in directories relevant for apm loading, and not to construct a list of all files and filter on these?
Collapse
16: Re: Down to < 60 seconds (response to 15)
Posted by Dave Bauer on
> wouldn't it be more straightworward and faster to search only in directories relevant for apm loading, and not to construct a list of all files and filter on these?

Probably but I was able to get the startup to a very reasonable speed with a very simple and easy to test change.

It was not clear exactly why it was setup this way. It would require a little more work. ie: Tcl library files only load from /tcl/ under a package. XQL files can be anywhere. Tcl/Adp page scripts and includable pages should not need to be loaded by the APM but the corresponding XQL files would.

So I think overall making the code smarter would be good but much more complex and time consuming to test and the simple change seems to be enough for at least my site.

Collapse
18: Re: Down to < 60 seconds (response to 16)
Posted by Gustaf Neumann on
Hi Dave,

after looking more deeper into the code, i see what you mean. The apm-loader is quite dumb and is based on the assumption that load more or less everything and filters later on. However, one can do better than what the above check does: instead of loading all files int the directory www/resources and filter these, one can already filter on the directory and avoid scanning all these files at all.

But there seems to be more useful, since several of the operations scan the files, and it is not useful to scan e.g. .zip files, or .java, etc. Furthermore, acs-lang was loaded twice.

Since the files are scanned several times during startup, it seems worth to speed it up (see my commits of today). On my test machine, some of the apm_load* operations are up to five times faster than before. The code is still straightforward. In case my filtering was too aggressive (e.g. on file extensions), it can be easily made more/less aggressive.

-gustaf neumann

Collapse
19: Re: Down to < 60 seconds (response to 18)
Posted by Gustaf Neumann on
just a datapoint from http://next-scripting.org (intel quad core, 2 years old)
Before the change: startup time 14 secs
[02/Apr/2013:17:24:44][32182.7f763a747780][-main-] Notice: nsmain: NaviServer/4.99.5 starting
[02/Apr/2013:17:24:58][32182.7f763a747780][-main-] Notice: nsmain: NaviServer/4.99.5 running
After the change: startup time 9 secs
[04/Apr/2013:12:11:36][8407.7ff48ba4c780][-main-] Notice: nsmain: NaviServer/4.99.5 starting
[04/Apr/2013:12:11:45][8407.7ff48ba4c780][-main-] Notice: nsmain: NaviServer/4.99.5 running
Collapse
20: Re: Down to < 60 seconds (response to 18)
Posted by Michael Steigman on
Have these changes been tested with the APM? I grabbed the latest from HEAD last week to work on upgrading our instance and my experience (tested twice) is that the SQL upgrade files are no longer being sourced. I backed out the changes to packages/acs-bootstrap-installer/tcl/30-apm-load-procs.tcl and the files were sourced as they should be. I haven't had time to examine the changes in detail yet as I'm focused at the moment on our upgrade. Just wanted to report back that there appears to be an issue.
Collapse
21: Re: Down to < 60 seconds (response to 20)
Posted by Gustaf Neumann on
The sql sourcing problems are fixed in CVS head since Monday 29 Apr
-g
Collapse
22: Re: Down to < 60 seconds (response to 20)
Posted by Victor Guerra on
My bad Michael... seems like i left out some changes in acs-tcl in my last commit.

http://fisheye.openacs.org/changelog/OpenACS?cs=MAIN%3Avictorg%3A20130502102746

It should work now if you apply that last commit.

Collapse
23: Re: Down to < 60 seconds (response to 22)
Posted by Victor Guerra on
And pls apply this other commit too:

http://fisheye.openacs.org/changelog/OpenACS?cs=MAIN%3Avictorg%3A20130502115722

I double checked and all scripts within the sql/upgrade directory were being unintentionally ignored.

Collapse
24: Re: Down to < 60 seconds (response to 23)
Posted by Michael Steigman on
Thanks, Victor. I'll post if I run into any other issues related to the changes.