Forum OpenACS Development: How about Apache 2.0?

Collapse
1: How about Apache 2.0?
Posted by Talli Somekh on 04/07/01 09:38 PM
Considering the major milestone of Apache 2.0 and it's inclusion of
multi-threading functionality, is there any chance that we will make
an effort to support it?

I know OpenNSD has begun as an effort to make AOLserver actually,
well, work for something other than Digital Cities. But would Apache
2.0 solve any of our problems?

I mean, other than girl problems, that is.

tal

Collapse
2: Response to How about Apache 2.0? (response to 1)
Posted by Don Baccus on 04/07/01 09:45 PM
AOLserver works very well in most environments, now that the memory
leakage problems with Tcl 8x have been solved.  Problems with FreeBSD
that folks have had aren't at all representative.

The problem with Apache 2.0 is that someone will need to support
mod_aolserver now that ArsDigita is no longer doing so.  If someone's
willing to take that on, then I see no reason why we can't support
Apache 2.0 + mod_aolserver.

Thus far no one's offered to take up support of mod_aolserver...

Collapse
3: Response to How about Apache 2.0? (response to 1)
Posted by Alex Sokoloff on 04/09/01 05:39 AM
Is it fairly certain that the memory leaks _are_ fixed with tcl 8.x and aolserver 3.3? We're running a 2.x version (I'm not sure which one) of AOLserver where I work because my boss is wary of problems with newer versions of AOLserver. The developers, especially those who've experienced the good life (?) with Perl aren't real happy with with the regular expressions in tcl 7.6. There would be some real advantages to running the newer versions of aolserver.

I've been meaning to start a thread about the pros and cons of aolserver. The subject's been hashed over quite a bit here, and I agree that it makes sense to be using aolserver at present for OpenACS, but there are some things that are not so great about it. AOLserver has very low overhead for serving page requests and database access, and it's a proven technology for high traffic web sites. But it doesn't seem to be so great for executing complex scripts. The tcl interpreter seems to be very slow in all the versions I've encountered. I did some not-very-scientific testing, (increment a counter a million times, and see how long it takes) on a couple of different versions of AOLserver on different boxes, and the results were kind of disturbing. I'll have to repeat the little test and remember to write down the results, but it was something like 30 seconds to increment the counter a million times with aolserver 3.1 on a PIII. The tcl interpreters you run from the UNIX command line that are installed on the Sun boxes we use at work also seem very slow... The one that comes with RedHat 6.2 seems to be much faster: it can count to a million in a second or two, just like the perl interpreter.

Any idea what gives here?

Collapse
4: Response to How about Apache 2.0? (response to 1)
Posted by Don Baccus on 04/09/01 07:41 AM
Well, for starters, yes, I think the memory leak problems are finally under control.  It turns out that AOL (Digital City et al) never saw them because they always run with the min threads and max threads parameters the same.  In other words, they've tuned their site to run a serious number of threads, due to serious traffic, and have no interest in giving the system back if traffic drops because no one else uses the production server.

The memleaks happened with Tcl 8x when threads exited, Tcl 8x required  a new call to release allocated memory and the AOL guys simply missed it because in their case threads live forever.

As far as speed goes, Rob Mayoff has contributed an update to AOLserver (AOLserver+ad12) that caches compiled bytecode so .tcl pages  don't have to be recompiled for each request (Tcl 8x is byte-code compiled as opposed to Tcl 7.* and earlier, which was truly and agonizingly slowly interpreted).

If you get significantly slower execute times for scripts with this patch set vs. via the shell, then something's wrong and OpenNSd.org would like to hear it.

OpenNSD will carry this patched version, along with the memleak patch,  as its base version of OpenNSD 3.x.

Lastly, AOL is still investing some time into AOLserver/OpenNSD and there's a code base for OpenNSD/AOLserver 4.x that look very promising.

Collapse
5: Response to How about Apache 2.0? (response to 1)
Posted by Ola Hansson on 04/09/01 10:53 AM
Alex,

Recently, I timed a complex and big procedure running on 8.x, RH 6.2, AOLserver 3.2ad12, K6 3 450MHz and was baffled by the incredibly high performance it yielded.

I'm working on building a backgammon service, so the proc I timed is one that tests if the move from a player is valid or not.

set elapsed_time [time "bg_valid_move_p some_arg" 10000000]

This proc calls a recursive proc that in my test case called itself 290 times checking the move against all possible moves (by building an array), and in turn, the recursive proc calls four or five helper procs on each iteration.

I was amazed that it took no more than 8 seconds to run all this TEN MILLION times...
Collapse
6: Response to How about Apache 2.0? (response to 1)
Posted by Alex Sokoloff on 04/09/01 10:09 PM
Well, I obviously need to repeat my little experiment. As, I recall, the fast TCL I ran into (the one that ships with Linux) was fast without pre-compilation - it ran quickly right off the bat when executing from the command line.

Don, I'm afraid I'm out of the loop on the various version of AOLserver out there now. Are you refering to the current release version of AOLserver, 3.3? Does the "ad12" in AOLserver+ad12 refer to the Oracle driver or something else?

Collapse
7: Response to How about Apache 2.0? (response to 1)
Posted by Alex Sokoloff on 04/09/01 10:28 PM
(Ignore my last comment about precompilation... I was confusing precompilation and caching...)
Collapse
8: Response to How about Apache 2.0? (response to 1)
Posted by Don Baccus on 04/09/01 10:42 PM
aol32+ad12 refers to patches supplied by aD, due to Rob Mayoff.  You'll need  it to run OpenACS 4.x because the code depends on his ns_cache implementation.  He'll soon be releasing ad13 (on Friday the 13th?  Hey, that would be sweet!) and that will become bundled up as our first CVS snapshot at OpenNSD.org, hopefully.

Compiling in Tcl 8x is transparent, it compiles to bytecode on demand.

  What Rob did was modify OpenNSD to cache those compiled bytecode bytes so your .tcl page doesn't have to be recompiled everytime a user  hits it, just the first time the Tcl interpreter associated with  a given OpenNSD thread runs into it.

Collapse
9: Response to How about Apache 2.0? (response to 1)
Posted by Alex Sokoloff on 04/09/01 11:04 PM
Got it. So the caching is a new feature found in the patch, but precompilation has been included in all nsd 8x (which I believe were first available in 3x versions of AOLserver)?

Also: I take it that the aD supplied version of aolserver has been the preferred one... until there's a version available on the opennsd site?

Collapse
10: Response to How about Apache 2.0? (response to 1)
Posted by Don Baccus on 04/09/01 11:17 PM
The version provided on aD is what we call "aol32+ad12" so, yes, that's the preferred version until "aol32+ad13" comes out.  At that point, Janine Sisk's going to roll it into CVS at opennsd.org.  Along with dropping Tcl support, they're not going to support any more improvements to AOLserver that aren't driven by client needs, i.e. existing sites (I imagine that would mean a previously unknown show-stopper bug).

I don't know if Rob will continue to release "adn+1" patches at aD.com  or if he'll just commit them to the opennsd.org CVS tree ... we'll probably find out when he gets ad13 ready to roll.

Collapse
11: Response to How about Apache 2.0? (response to 1)
Posted by David Kuczek on 04/10/01 10:58 AM
I would like to clear some things concerning the use of an AOLServer with openacs. My basic concern is: Which server should I use, when my future site should carry some (lots of) traffic.
<br><br>
The <a href=https://openacs.org/doc/openacs/aolserver/x10.html>Simple AOLServer Installation Guide</a> tells you only about the regular distribution at www.aolserver.com and not about the AOLserver+ad12.
<br><br>
The Installation Guide tells you "In my personal experience nsd76 is a bit faster, so choose it according to your needs" <b>while</b> Don tells you "Tcl 8x is byte-code compiled as opposed to Tcl 7.* and earlier, which was truly and agonizingly slowly interpreted".
<br><br>
If this is a contradiction we could open a thread on experiences with different servers (or distributions) relating speed, installation etc. People from the opennsd project could quickly introduce their effort and I would write a "HowTo" or "WhichToUse" for openacs.
<br><br>
This would clear some confusion I believe...
Collapse
12: Response to How about Apache 2.0? (response to 1)
Posted by Roberto Mello on 04/10/01 02:57 PM
Don is right. The AOLserver installation guide (that I wrote) is wrong. When I wrote that nsd8x was a new thing in AOLserver and it was slower, but progress has been made, which is great. I overlooked that when I made the 3.2.4 release of the docs. It will be corrected for 3.2.5.

Thanks for pointing that out. If you'd be so kind as to post that as a bug in the SDM for the documentation, I'd appreciate it.

Collapse
13: Response to How about Apache 2.0? (response to 1)
Posted by Don Baccus on 04/10/01 05:42 PM
Tcl 8x was very slow with the ACS when it first appeared because each page in your installation was recompiled for EVERY PAGE HIT.  Procs (i.e. your entire tcl directory) were compiled once and saved, but pages aren't wrapped in a proc and the server source's them directly.

Rob's modifications include caching of compiled bytecode for pages, as mentioned elsewhere.  This removed the "compile for every hit" overhead, substantially speeding things up.

The sequence, as Roberto mentions, is that he wrote our docs, Rob made  his improvements, and the docs sat unchanged.  Thanks for noting the contradiction of the docs with other statements...we need more eagle-eyed critics!

Collapse
14: Response to How about Apache 2.0? (response to 1)
Posted by Alex Sokoloff on 04/11/01 12:54 AM
Don: "Tcl 8x was very slow with the ACS when it first appeared because each page in your installation was recompiled for EVERY PAGE HIT."

And if I edit a .tcl or .adp file under the page root, how does AOLserver know that it needs to be recompiled? Do I need to re-source it somehow?

Collapse
15: Response to How about Apache 2.0? (response to 1)
Posted by Michael Feldstein on 04/11/01 02:18 AM
Returning to the original question, are the improvements in Apache's threading, etc. dramatic enough that there's reason to believe mod_AOLServer could be evolved a real alternative for production sites? Right now my impression is that nobody really trusts it to be more than a gimmick to get Apache lovers to at least *try* the ACS. It seems to me that the likelihood of getting somebody to "own" mod_AOLserver is going to depend on the potential return on the investment.
Collapse
16: Response to How about Apache 2.0? (response to 1)
Posted by Talli Somekh on 04/11/01 03:08 PM
Don, can you point to or mention some of the highlights of the promising AOLserver 4.X architecture?

tal

Collapse
17: Response to How about Apache 2.0? (response to 1)
Posted by Don Baccus on 04/11/01 04:59 PM
Alex - it uses ns_cache and uses stuff it gets from stat'ing the .tcl source file as part of the key.  In short, the .tcl file gets recached  and recompiled if it changes.

Talli - Jim Davidson posted a set of stuff he was working on to the opennsd "founders" e-mail list.  I don't know if he has published anything in public yet.  One thing folks are talking about is to drop support of Tcl7 altogether.  This would allow the server to plug into Tcl using Tcl Objects rather than Tcl strings galore, which is supposed to be a considerably more efficient way for the server to talk to the interpreters it spawns.

Collapse
18: Response to How about Apache 2.0? (response to 1)
Posted by Alex Sokoloff on 04/12/01 06:16 AM
I don't think I'm gonna get hired to benchmark scripting language performance, but here's the results of further investigation. I made a very simple test that iterated a counter: 

set start [exec date]
set counter 0
while { $counter < 1000000 } {
    incr counter
}
set end [exec date]
set result "start: $start
end: $end"

(For what it's worth, the machine was a Pentium Pro, OS RH 6.2)

Execution time

  • Running from the command line, in tclsh: 4 seconds
  • aolserver32ad12 (therefore nsd8x), run as a procedure in the tcl directory: 14 seconds
  • aolserver32ad12, placed in a .tcl page under the pageroot (first time page run): 64 seconds
  • aolserver31 using nsd76, run as a procedure in the tcl directory: 87 seconds
  • aolserver31 using nsd76, placed in a .tcl page under the pageroot (first time): 85 seconds

I didn't actually make sure ns_cache was working with aolserver32, because I think you get the same effect if the code is in one of the procs loaded at startup.

I'm told that interpreted languages interpret every iteration of a loop if they don't tokenize or precompile the script - which would explain the sluggish performance above. And just looping through a zillion (well, actually a million) iterations of one line of code is probably a misleading test. I'll try to run the same test with a more complicated script that doesn't loop so much. At any rate it looks like using the preferred version of AOLserver, and enabling ns_cache if you don't put most of your code in procedures is definitely the way to go.

Collapse
19: Response to How about Apache 2.0? (response to 1)
Posted by Don Baccus on 04/12/01 04:21 PM
Those are strange numbers, ns_cache shouldn't matter as you are running the only once.  It should be compiled then ran than the bytecodes tossed if you're not running ns_cache.  If you are, the same  except the bytecodes aren't tossed.

You are right about procs in your tcl library.

At least this shows some improvement over Tcl7.

It is very strange that tcl from the shell should be so much faster.  I'd encourage you to show these numbers in the opennsd.org forum and see if anyone has any ideas.  You might e-mail Rob Mayoff directly to see if he has any insight he could share with us.

Collapse
20: Response to How about Apache 2.0? (response to 1)
Posted by Michael A. Cleverly on 04/12/01 04:54 PM
What version of tclsh are you running from the command line? Is it also 8.3.2? Also, if you use the tcl command time command, you can get better granularity than exec'ing date.

Using a modified version of your test: 

time {  
       set counter 0
       while { $counter < 1000000 } {
           incr counter
       } 
} 10
and inside a proc: 
proc time_loop_speed {} {
    time {  
           set counter 0
           while { $counter < 1000000 } {
               incr counter
           } 
    } 10
}
I got the following results (P3-500, RH 6.2):
  inline proc
tclsh 8.3.2 3309115 ms per iteration 2934595 ms per iteration
AOLserver 3.2+ad12, nsd8x 6607447 ms per iteration 5637806 ms per iteration
Of course, it's not really an apples-to-apples comparison because tclsh is not threaded, while nsd8x is.
Collapse
21: Response to How about Apache 2.0? (response to 1)
Posted by Don Baccus on 04/12/01 06:11 PM
Those are better numbers in that the degradation between the opennsd and tclsh tests is about the same for proc and inline.  In other words, each opennsd test is about 1/2 the speed of the standalone test.  And the proc case is roughly 1/8th faster than inline in each case (rather than 4x faster as in Alex's results).

While I don't know why you guys get such different numbers, this last set is much more in line with my expectations.

You're right about the threaded vs. non-threaded issue.  I'm surprised  it would cause that level of degradation, but not surprised that the interpreter would run slower to some degree.

Collapse
22: Response to How about Apache 2.0? (response to 1)
Posted by Stephen . on 04/13/01 03:13 AM
Your AOLserver is not compiled with optimisiations. It is in a debugging configuration by default, see the makefile.
Collapse
23: Response to How about Apache 2.0? (response to 1)
Posted by Don Baccus on 04/13/01 04:04 AM
I've checked mine and it's not compiling in what I'd call a debug mode.  However, default optimizations are being used (I checked my aolserver/Makefile and aolserver/tcl8.3.2/Makefile) rather than (say) -O2 which is how PG is compiled, at least on most systems.

Hmmm...I don't have time at the moment to play with this, I wonder if Michael does? :)  Compile with -O2 and see what happens?

Collapse
24: Response to How about Apache 2.0? (response to 1)
Posted by Michael A. Cleverly on 04/13/01 06:08 PM
Recompilling with CCOPT = -O2 -DNDEBUG=1 (instead of CCOPT = -g in include/Makefile.global) gives me these times:
  • inline: 4863958 microseconds per iteration
  • proc:   4318870 microseconds per iteration
I remember reading a while back on comp.lang.tcl a post from Jeffrey Hobbs ("The Tcl Guy") that there was a roughly 30% performance hit for tclsh when compiled with --enable-threads. So these new numbers seem quite inline with what tclsh would likely be if it were compiled with threading enabled.
Collapse
25: Response to How about Apache 2.0? (response to 1)
Posted by Talli Somekh on 04/13/01 07:04 PM
I would like to promote someone to pick up mod_aolserver. If anyone is interested in doing this and can prove that they know what they are doing, please lemme know. I would like to work out some incentive.

Only people that are serious, though. And when I say incentive, I do _not_ mean I'm going to pay anyone. Something creative and worthwhile though. Perhaps some kind of hosting.

tal

Collapse
26: Response to How about Apache 2.0? (response to 1)
Posted by Don Baccus on 04/13/01 07:09 PM
Michael - thanks.  Especially thanks for catching the flag in include/Makefile.global, I didn't think to look there...

Wonder why they distribute it that way?  I think it's an oversight...

Those are pretty decent times you've got there!

Collapse
27: So back to the original question... Response to How about Apache 2.0? (response to 1)
Posted by Talli Somekh on 04/14/01 06:34 PM
Returning to Michael Feldstein's question, will the benefits (virtual domain hosting and wide acceptance and familiarity) and improvements (multi-threading) of Apache 2.0 be something that the community will embrace?

Thankfully, Petra Paler picked up mod_aolserver and after speaking with him a little bit yesterday I'm confident he's going to do a great job. So we're in good hands here and we'll just have to see how people respond.

But I'm curious to know whether there is indeed interest in Apache from two perspectives, the evangelical and the technical.

It seems that the benefits of using Apache, which seems to improve and extend at a rate far greater than AOLserver (we'll see about OpenNSD), evangelically is rather considerable. There are a whole bunch of people out there that know and support it. Also, while people hearing about OpenACS are probably eager to check it out, they are probably a little hesitant to learn three architectures (PG, ACS and AOLserver/OpenNSD). If they just had to figure out two and can run it alongside their LAMP systems by installing ACS, PG and mod_aolserver that would be neat. The *BSD problem would also be solved, but that's just a small issue with OpenNSD which I'm confident will be solved.

But that stuff is reasonably obvious and something that I can see. What about technically? Is anybody familiar enough with both systems to give a somewhat cogent analysis of Apache vs. AOLserver/OpenNSD? Are there any issues or thing to think about? It would seem that there is a fundamental performance hit with mod_aolserver since it's a module rather than an embedded scripting language, but is there and if so how significant is it?

thanks in advance.

tal