Forum OpenACS Development: Status update ... will there ever be a beta or final release?

I know many of you are probably wondering about this.  Frankly, I've
wondered at times, too ...

However, we seem to be at the point where we can see the proverbial
light at the end of the tunnel.  Here's how things look:

1. Vinod has finished his cut at an install document.  This was a big
"todo" item on my list of stuff we needed for a beta release.

2. Mat Kovach is going to bundle an AOLserver/ad3.3/nsuuencode/drivers
etc tarball that folks can use to get all that stuff in one fell
swoop.  This was also a "todo" item.  I don't know just how quickly
this will happen but it's "days or N weeks (N tiny)" not "many weeks
or months".

3. I think various packages are just about where they're going to be.
Neophytos has some search stuff to finish off, that's the only work
item I'm aware of (clue me in if I'm wrong, folks).

4. Testing - I plan to investigate some of the "package not working"
complaints posted here over the past few weeks, fixing whatever I run
into.  Soon.  Hopefully over the weekend.  After that it would be
great if some folks would step forward and test again (Simon Millward
to the rescue?)

5. Branding - this isn't going to hold up a beta release.  And I will
get that poll up shortly.  I just finished a quick client task that
had a Monday deadline and my time's not quite as crunched as it has
been the past week.

What else do we need to roll a beta and declare a "bug fix only" code
freeze?

Damn, I forgot something:

6. People have been steadily submitting patches for a variety of bugs, which I deeply appreciate.  I've got two weeks worth backlogged that I've not inspected and applied - my apologies to folks who have submitted.  Don't worry, they'll get in.

Great news, Don! Thanks for the info.

One question, regarding #2:

> Mat Kovach is going to bundle an AOLserver/ad3.3/nsuuencode/drivers 
> etc tarball that folks can use to get all that stuff in one fell 
> swoop. This was also a "todo" item. I don't know just how quickly 
> this will happen but it's "days or N weeks (N tiny)" not "many 
> weeks or months". 

Is this going to be the only AOLserver configuration that's going to be "supported"? By that...I don't care if other configurations are more difficult, but I think it's important to at least talk about them, and provide help where possible. The FreeBSD docs in the doc storage are a good start...

I ask b/c the more flexible we make the server piece, IMO, the better. I realize OpenACS has certain requirements, but (for example) I don't want to have to install a non-Debian webserver package unless absolutely necessary. Some new pieces, extra drivers, etc. Sure.

Plus, is the bundle going to be a binary tar, source, both, etc? I guess Mat may be the person to talk to here, eh? *grin*

Sounds GREAT, though, Don. I can't wait for beta...

Basically I have the latest aD sources and installed the FreeBSD patch and pounded on it for serveral weeks on FreeBSD, OpenBSD, RedHat, Solaris, and SuSE (using postgres on all, Oracle on the Linux boxes).

The source will be able to build postgres, oracle (or both) db drivers and all modules needed for openacs4.  My hope is to keep everything in one tar ball and provide package maintainers a solid ground for providing their packages (IE, the Debian, RPM, *BSD Ports, etc.) call all use the same base to create the packages so if there are problems on a platform then can feed them back to the main tree.

I'll attempt to create binaries for everything I currently have setup
(SuSE, RedHat, Solaris 8, OpenBSD, FreeBSD) in standard tarballs.  I also have a hook to install openacs in to /web/<servername>, but I don't have the openacs source in it.  It could (if wanted) make fore a nice 'run this, they connect to http://localhost:8000 to start your install' type of thing.

I'm happy with the aolserver part and a few people will be (or have) recently sent me pages and I'll work those in this weekend and start
a build-ing :)

Thanks for your hard work.  Let me again put a plug in here for the Win32 stuff, I still think some people will want to try this out on Win32.  I develop offline on Win32 and so far it works great with the exception of the lack of nsopenssl.  I had one person reply to my plea for help and say he was trying to build it; we'll see what happens there.
A win32 port is a possibility.

From what I can remember the AOLserver (and OpenACS) still are not production ready on Win2k (correct me if I'm wrong).

I've never done an serious Windows packaging before but I'd be willing to look into this if there are others what wouldn't mind helping or point me to good sources of information about it?

Mat, have you played with 3.4 at all?  What would we need to do to adopt that as our AOLserver base? (see the other thread on this)
I've tried 3.4 a bit but have not given a very deep test.  The version I tried still did not have the FreeBSD fix (no biggie).
I'll take a look at the difference between ad13 and 3.4 and post my thoughts/findings later tonite or this afternoon.
AOLServer is dropping Win32 support.  You'll need to look at a fork or a
Cygwin version for AOLServer on that platform.
Yes, they are, but from previous threads isn't the general consensus that we need to go with 3.3.1+ad13 for the near future?  Or possibly 3.4.2.

A Cygwin port is becoming more possible every day--last I heard the stumbling blocks centered around lack of full pthread support in Cygwin.  I worry about compiling the Oracle driver under Cygwin.  I'm not a newbie, but that task makes my hair stand on end.

Windows support will be very important long-term for wider
adoption of dotLRN.
It's really going to have to be via Cygwin, long-term.  Unless someone volunteers to continue to maintain Windows support in AOLserver, or to backfit important bug fixes (such as security-related bug fixes) to earlier versions.
Just wanted to report that the latest Cygwin will get through part of the make process on 4.0.  It dies now with some problems compiling the tcl portion of the server--I'm assuming that's because cygwin provides tcl itself and there is some confusion about defines.

It used to blow up a lot earlier during configure with complaints about pthreads.

Oh, yeah, and configure still complains that it doesn't know how to build shared libraries.

This doesn't mean I'm committing to do the port (wish I could, no time right now), I just thought I'd report what I'd discovered so far.

In what ways (if any) would a Cygwin-based solution be inferior
to a Windows-native AOLServer?
The only thing I can think of is difficulty compiling the Oracle driver for Cygwin.  Other than that, I would prefer a Cygwin solution.
What about the overhead of the POSIX emulation layer?  From a non-cygwin-hacker's point of view, every cygwin app I use seems slow relative to its equivalent windows port, though I have no metrics to back that up.

Besides, with cygwin, you're requiring UNIX sysadmin knowledge to some extent (watered down, yes), even though the host OS is NT.  I don't know any dyed-in-the-wool NT people who would look forward to supporting a pseudo UNIX environment, so the converts would be few from this approach.  It's likely, though, that the command line-driven, config file-editing nature of AOLServer would have filtered out that crowd already.  The question remains, however: If the admin is comfortable supporting a UNIX emulation (that possibly has bugs itself) on top of a Microsoft OS (which possibly has its own bugs pertaining to cygwin's lower-level interfaces), why would the same admin not be comfortable installing RedHat?  Of course, I suppose there are a bunch of potential beurocratic/corporate reasons, in addition to a very rational fear of the learning curve of Linux sysadmin, but I frankly question whether a windows-only shop would really consider running OpenACS on top of an emulation layer.

Of course, as a development platform for the window-phile hackers out there, cygwin would be great.  Who doesn't get a little thrill out of running a fully operational db-backed web site off of their laptop? :)

Hmm...

It doesn't sound like Cygwin will be an attractive solution for
those folks who might want to adopt dotLRN but run
Windows-only. How much work is involved with keeping up the
Windows port? What kind of a skillset is required? And is
AOLServer the only obstacle, or are there others?

Alternatively, what are the prospects of mod_AOLServer running
acceptibly well now that Apache 2.0 is (sort of ) out? And would it
be less work to keep mod_AOLServer up-to-date than it would
be to maintain a Windows port of AOLServer itself?
I've only looked briefly at the mod_aolserver source, but my impression is that it is simply a light-weight implementation of enough of AOLServer's API that you can run Tcl with database connectivity.  Last I checked, you can't even run ACS Classic >= 3.4 on mod_aolserver 1.1.  Because mod_aolserver is just a set of Tcl and db hooks plugged into the connection handling of apache, I'd imagine that keeping it up to date with AOLServer development would be significantly more difficult than maintaining a full port of AOLServer (which, I assume, is part of the reason that aD hasn't updated the package in so long).  New revisions of AOLServer itself would have a relatively good chance of NOT breaking the port in any way, but it would take a seasoned C hacker with rock solid windows experience to declare the revision to be of 'production quality' on that OS.  Remember that dropping support of a platform doesn't mean making it incompatible... it can just mean that they don't want to expend the engineering effort to certify it as a production-ready platform.  OTOH, new revisions to AOLServer in the wake of an intentional drop of the windows port may result in some very UNIX-centric optimizations, which could take some wizardry to sort out.
Michael asks how difficult it would be to have mod_aolserver running on Apache 2.0.  My understanding is that a module written for 1.x will need to be re-written for 2.x.  However, since there is an officially blessed mod_tcl for Apache, perhaps a few weeks of mod_tcl hacking could result in a reasonable facsimile of AOLserver under Apache.

Concerning mod_aolserver, I have in the past used it for sites.  It performs OK, but will need more RAM due to the fact db pools are not shared among processes.  You need more RAM for the httpd process, as well as more RAM to hold the greater number of PG sessions.  I ended up using AOLserver behind Apache/mod_proxy to implement sane virtual hosting; of course, if you have your heart set on Apache-only, well, I can't help you.

OTOH, new revisions to AOLServer in the wake of an intentional drop
 of the windows port may result in some very UNIX-centric 
optimizations, which could take some wizardry to sort out.
I think this is the central issue. The AOLserver people's number one mission in life is to make digitalcity.com and aol.com scale on as little hardware as possible, so anything they can do to pump up throughput will be on the table.

Even if it means writing code that would be difficult to back-port to Win2K.

This benefits all of us running AOLserver in a Unix or Linux environment, but could make maintaining a viable Win2K port problematical.

The major problem, though, is that there's no volunteer with the necessary time and skills to keep a Win2K version maintained. Cygwin greatly reduces the amount of work that'll be necessary.

I'd guess that long-term this is better than the Apache/mod_aolserver approach.

Maybe I am mistaken, but the idea of a Win32 port is very unattractive to me.  It means resources will be diverted to supporting a platform that just isn't that good.

You can run a decent OpenACS installation on hardware that any Windows-using corporation or school will be discarding this year.  Corporate edicts can get quite flexible when you are using castoff hardware that costs nothing.

Further, for education, OpenACS runs pretty much out of the box on MacOSX.

The argument against Unix-only was much stronger a few years ago.

Nowadays, I would think that people would be ashamed to admit that they didn't have the ability to either run the system themselves or find someone who could.

I have a source tree of AOLserver3.3ad13, plus Vinod's ns_uuencode patches, plus the UID/GID patch, plus the FreeBSD patch for exec.
I'll be making them available in the next few hours.

Just wanted to let eveybody know things are progressing :)

Obviously, if there aren't any resources then a Windows port
won't be done. I'm asking because I think it an important enough
issue to think about whether I might be able to scare up a
sponsorship for a Windows port. Regarding the issue of whether
it is important, Patrick, there are many, many organizations that
are still unabashedly Windows-only. They believe (rightly or
wrongly) that maintaining skillsets in only one OS platform keeps
down their total cost of ownership. This applies even to
universities; just because the comp sci department as some
Unix boxes doesn't mean that the (typically more conservative) IT
support departments that run the various university computing
systems are also supporting Unix (or Macs). Not having a viable
Windows port is definitely a barrier to broader adoption. Having a
solution that requires sysadmins to know Unix (i.e., Cygwin)
doesn't really solve the problem.

If there are no resources then there are no resources. It may very
well be that the community of current users (and, more
importantly, the subset of that community that actually
contributes code) doesn't care that much about whether
Windows-only organizations would consider an
OpenACS-based solution (or believes that such organizations
might be disinclined to consider OpenACS even if it were
available on Windows). But I think it's a mistake to believe that
the lack of Windows support does not represent a very
significant barrier to adoption in a very broad range of
organizations.

The problem is really that you're discussing this with the wrong community, if non-Cygwin Windows support is desired.  Few of us here are knowledgable about AOLserver internals, and of those that are apparently few or none are NT systems hackers.

So ... the AOLserver community is the one that needs to be pinged, to see if there's any support for the notion of continuing Windows support.

AOL's not and they provide almost all the development resource for AOLserver (say 99.9%) so this would be an uphill battle.

That still leaves Postgres, which is only supported under Windows via Cygwin.  A total Windows solution, assuming AOLserver support not reliant on Cygwin, means Oracle ... not necessarily a problem but something to realize.

Before beta...

(A) I'd really like OpenFTS-driver to have an easier/simpler/less confusing install model.  Right now it still requires a package parameter that is a path to the OpenFTS-tcl source directory!  No other package does that, or requires that sources for something it depends on be available at OpenACS4 runtime.

This sort of thing makes RPM (and probably .deb) packaging more difficult than it should be.  Shouldn't openfts-tcl install all the parts that openfts-driver will need at runtime?  Since some folks may say this issue is not a bug, but a deliberate design feature of openfts-driver (!), I'd prefer we didn't code freeze until it was taken care of.

(B) Does "search stuff to finish off" in your #3 include ability to search content other than notes?  Can we search bboard items, for instance?

(C) We also need to think carefully about the state of documentation.
Is the OpenACS4 documentation at a "only bug fixes needed" state already?  It doesn't seem to be, from Roberto's 05 Jan status report.

(B) "Search stuff to finish off" refers to different ways of searching content. For example searching only content items of a specific package or mime type. Unfortunately, I won't be able to finish these improvements until after our first release. The search package already supports indexing/searching of content other than notes. However, to be able to do so requires that other package maintainers implement FtsContentProvider for their packages. Please have in mind that I had planned these improvements for our *second* release cycle which for better or worse I hoped that it would happen February 2002.

(A) This was indeed a deliberate design feature of the package that would enable us to cope with different versions of openfts. We could have copied the tcl files in our package but we didn't know at the time, *August* IIRC, how often openfts would be updated. Meanwhile, tsearch was released as a contrib module for PG and I had planned to provide an alternative driver that would be easier to install but then again tsearch depends on 7.2 features and as you might know already openacs does not work with 7.2 yet due to the fact that we relied on a "bug" for making porting from oracle easier.

Neophytos and I have been e-mailing privately over the last week and I don't think we want to hold up release for his improved search package. He simply doesn't have time to finish it up beforehand.

Improving the OpenFTS install ... this would be awfully nice, and if anyone wants to tackle this feel free to e-mail me.  But, again, I don't think we should be waiting.  There are already people building production sites using versions from CVS and we've taken far longer than I wish to get to this point.

Documentation ... I've been saying that as soon as the install documentation was reasonably complete that I'd roll out a beta.  The "bug fix only" creed doesn't apply to documentation in my mind, not for this release, anyway.  A final release - yes, most certainly, we must wait until documentation's much more complete.

...but then again tsearch depends on 7.2 features and as you might know already openacs does not work with 7.2 yet due to the fact that we relied on a "bug" for making porting from oracle easier.

How much work will be involved to straighten up this bug from the OpenACS 4.x Oracle queries? Will this be changed for our first release or will the first release only run on < PG 7.2?

What will the first release be called? OpenACS 4.0?
Concerning mod_aolserver...

There has been an idea by Petru last July. He wanted to add FastCGI support to AOLserver:

https://openacs.org/bboard/q-and-a-fetch-msg.tcl?msg_id=0001uK&topic_id=OpenACS&topic=11

Is this still a current topic? If Petru's idea is to be realized and works like intended, it would solve the win32 problem or am I missing something???

Petru doesn't seem to be around any more nor working on mod_nsd (as we ended up calling it).  If I'm wrong, Petru, speak up!

Naming ... I need to get that poll up, don't I?  I've got a "todo" list sitting here right in front of me with that task on it.

PG 7.2 ... I actually tentatively made a fix and it's committed, but I've not had time to personally test it.  If you or someone else wants to try installing under PG 7.2 we'd all be thankful, I'm sure!  My worry is that we'll find other problems as we test under PG 7.2...

I'm more likely to have time to help fix problems folks find under PG 7.2 than to be able to spend time digging through packages looking for them, at the moment.  I'm very busy with client work these days.  I purposefully put off taking on paying jobs for much of the last 3-4 months of 2001 so I could concentrate on OpenACS 4 but now I've got to  replenish the 'ole bank account.

PG 7.2 ... I actually tentatively made a fix and it's committed, but I've not had time to personally test it.

I just installed OpenACS 4 (CVS 1/31/02) on PostgreSQL 7.2rc2. The first problem that I ran into was that OpenACS picks xql files based on the database-name and database-version. All the PG xql files are labelled "7.1", but the database version is now "7.2". Once I hacked db-init-checks-postgresql.tcl to recognize "7.2" as equivalent to "7.1", OpenACS 4 installed perfectly.

The other problem was related to OpenFTS. When I installed PG 7.1.3, make install-all-headers installs headers into /usr/local/pgsql/include, while PG 7.2rc2 installs headers into /usr/local/pgsql/include/server. The OpenFTS configure script has to be edited to look there. I didn't test OpenFTS beyond that yet.

Other packages seem to work well!

How does running AOLserver as a FastCGI server behind Apache solve the Win32 problem, when AOLserver is being de-supported on Win32?  The FastCGI server code does not even exist (and I don't think it's a trivial hack).  How will apache know which file to serve, considering the ACS Request Processor's non standard semantics?  You want a Windows only shop to run TWO non standard web servers?  And "requiring admins to know Unix" etc. is not the case when using Cygwin. It's a Unix wrapper around the Win32 API, of concern only to the programmers who do the port.
I am only one member of the OpenACS community, I am offering only my own opinion.

I simply find it hard to believe that OpenACS is losing huge numbers of installs due to there being no Win32 support.  I believe that we are losing the same number of installs on Win32 in absolute numbers that we are losing by not doing a VMS port, ie. not big enough to care about.

The answer in most open source communities is, if you want it and it isn't there, either build it yourself or pay someone else to do so.

The focus now should be OpenACS4 going into beta and eventually reaching release status.

Patrick, I don't think diverting core resources to a Win32 port is
even under discussion here. And most certainly, *nobody* is
suggesting syphoning off resources from producing the final
release.Those of us who are interested in the Windows port
(including particularly me) are just trying to get a sense of what
the options are to see if it's realistic to look for *additional*
resources for this purpose.

But regarding your assertion that you don't lose adopters
because of lack of Windows compatibility (or, to put it slightly
differently, you don't gain adopters that you would have with
Windows compatibility), I can say with great confidence that, at
least in the dotLRN world, you are dead wrong. I know a number
of (fairly large) clients who might consider an OpenACS-based
solution if their Win sysadmins could learn to maintain it but
won't touch a Unix-based solution, no matter how good it is.

Stephen, if you're right that Cygwin doesn't require users to know
Unix, then that *does* sound like potentially the best solution.
What kind of skill set would a hacker need to make the
necessary contributions to cygwin, and how much work might it
be?

<em>
What kind of skill set would a hacker need to make the necessary contributions to cygwin, and how much work might it be?
</em>
<p>
At this point, someone who knows a fair amount about Unix build environments (e.g. configure, gcc, libtool, shared libraries etc) and embedding Tcl.  When I try to build AOLserver on Cygwin the configure script complains that it doesn't know how to build shared libraries on the target platform, and then dies when compiling and linking some of the Tcl stuff.  I suspect this might be because Cygwin includes Tcl itself, so the include/linker paths might simply need to be adjusted.
</p>
<p>
I have confidence that someone with the right knowledge could do this without too much hassle.  I think all the essential pieces are now in Cygwin (e.g. I think the threading issues with pthreads have been corrected).
</p>
Vinod ... the query dispatcher should be selecting files based on the version being >= the specified version.  I'll try to make that change soon...

I'm around, though not getting email alerts (only checking the forums every now and then).

I'm indeed not working on mod_nsd. The plan was to work on adding FastCGI support instead, but I didn't do much work on that either (I'm not needing it, and not many people bugged me about it either).

However, that still wouldn't solve the win32 problems, since you still need AOLserver, only with a FastCGI listener instead of a HTTP one.

I recall a discussion of mod_dtcl for Apache that suggested that mod_dtcl might some day be an Apache module that supported much of the AOLserver API. Something like the author of mod_dtcl was unaware of AOLserver when he started, but when shown AOLserver, the author apparently liked a lot of what he saw. I can't find that discussion now, and looking at what mod_dtcl is today, I don't see the AOLsever connection.

Regardless, instead of a win32 native for AOLserver, it would be much more valuable to take one of the Apache tcl modules and corrupt it, I mean extend it to implement the AOLserver API.

If a Cygwin solution could be set up, how hard might it be to
package the whole thing up in a Windows installer, so that users
wouldn't have to deal with "make" or whatever Unix commands
might be necessary?