Forum OpenACS Development: BBoard development

Posted by Henry Minsky on
I am moving this discussion from email to the bboard where other people can contribute
From: "Neophytos K. Demetriou" 
X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.14-5.0 i686)
X-Accept-Language: en
MIME-Version: 1.0
Subject: OpenACS BBoard Package
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=us-ascii
X-UIDL: 56cadb50b3a0fb57efe66e1b88ef9ccf

Dear Henry,

I have seen your post that you are about to begin porting the classic
ACS BBoard package to OpenACS. As far as I can tell the acs-messaging
package is deprecated. I would suggest then that we develop a new 
package for OpenACS. I have experimented with a new BBoard package 
for a
couple of days. It is different than the current BBoard package in two

* one package instance = one forum (instead of one package instance =
many forums). Multiple instances are allowed.
* it is not based on content-repository. (but this can easily change)

I have only experimented on this package so it hasn't got the
functionality of the classic(current) BBoard but maybe you can use if
you want to start a new BBoard package. Please find the apm (oracle) 

I am currently busy with school until June 7th so I won't do any work 

Best wishes,
Posted by Henry Minsky on
Neophytos mailed me a copy of a new bboard package he has been working on. I will ask if he wants to place a copy where other people can review it, or if I can put up a copy.

I worked on a reimplementation of the bboard for a project at aD, when
ACS 4 just came out last year, and I had some thoughts about design.

Philg's original mechanism for the core bboard message threading data model (the one that's in
ACS 3.4) is actually pretty optimal, and doesn't rely on CONNECT BY,
which in turn has issues with sorting.

I am not sure if anyone has really tested the ACS 4 bboard (Oracle) for scalability of the data models with large numbers of forums and messages. They may have, I just don't know.

In any case, I found the current ACS BBoard module, with its dependencies on the acs-messaging, notifications, and content-repository, to be rather hard to work with. It is templated, which
is a big win, but I feel that the extra layers of abstraction that were added did more to increase the complexity and to make it more difficult to tune and debug the implementations, especially when you
want to customize the code.

So, I think returning to a simpler and flatter design would make sense. We should re-examine the goal of reusable code for general-comments, and for email notifications (both immediate and digestified) and see if there is some way to provide that without
making something that is difficult to work with.

I think the content repository is a useful idea for more conventional "publishing" types of tasks, but unless there is
a substantially simpler API to use it, it is overkill for the bboard,
and may have performance issues.

email notifications

Posted by Dave Manginelli on
I hope that the new (simplified) direction you are trying to take bboards won't mean that it will be isolated from other parts of the new OpenACS 4.x infrastructure such as subsites.  The subsite feature is one of the major attractions of 4.x for me.

However, I guess the modular nature of the OpenACS 4.x packaging system would mean that those of us that require such features could chose our own implementation (provided, of course, that other packages do not depend on a specific implementation of the bboard system).

Finally, if other modules are not dependent on bboard, then wouldn't it be best to at least get a straight port of the AD version going now so that we have at least some bboard capability to demonstrate immediately (i.e. one in hand vs. two in the bush).

B.T.W.  As always I recognize that those who actually DO THE WORK to code these things should have the final say, and I greatly appreciate the work you folks are doing.

Posted by Henry Minsky on
I think making the bboard subsite aware is a definite requirement, but that is orthogonal to the issue of how the message store is implemented and how the email notifications are implemented.

I also like your idea of porting the old ACS3 bboard as a quick
and dirty way to get a tuned and burned-in bboard going. That code is quite ugly, but with some slashing and burning we could probably have something usable pretty quickly. I would only support  Q&A style presentation for example.  I think I will
try to do something like that, making just the minimal concessions towards cleaning it up,  templating it and subsite-awareness, and borrowing some of the basic permissions model from the ACS 4 bboard.

Posted by Neophytos Demetriou on
You can find the simple BBoard package here.
Posted by Jon Griffin on
Posted by Michael Feldstein on
Henry, thanks for carrying the torch on this. I've been doing a fair
bit of work with bboard for educational use lately. Based on that
experience, the following functions would be highly desireable:

-Thread management. Moderators need to be able to move an
off-topic post from one bboard to another. (It seems to me that
content repository might make this easier, but what do I know?)

-Email support. This is a biggie. Participants need to be able to
both reply to a thread and start a new one via email.

-Templating, including choosing a threaded vs. a non-threaded

-A simple mechanism for peer rating of posts, a la
(i.e., check this box if you think this post is brilliant). In an ideal
world, the system would be aware of who is checking the box, so
a professor identifying a post as brilliant could be differentiated
from a student making the same judgment.

Posted by Talli Somekh on
<i>Email support. This is a biggie. Participants need to be able to both reply to a thread and start a new one via email. </i>
yesyesyesyes. This is a very biggie. Part of the bummer of ACS3.x is that one needs to revisit the website each time they wanted to respond. However, I was under the impression that this was solved in 4.x. Am I wrong?


Posted by Steve Crossan on
Re: email support

I'm currently doing some work with the Java mailet architecture within
the jakarta James project. This is a neat framework for using JavaMail
to pull apart a mail message and do something with it - in this case map mail fields onto database fields. It's simple to implement and works nicely. My idea is to allow as much interaction with the ACS via email as possible.

So far I've written prototypes of GeneralCommentMailet and ProjectHoursMailet. These allow me to send mail to <group_name>@intranet.runtime-collective and log a general comment (e.g. as a project report within a project), or <group_name> to log my hours in the intranet. Haven't done a bboard one yet but that's next.

It's ACS3.4/Oracle at the moment but all the db code is hidden in a layer behind an interface so porting to PosgreSQL/ another version of ACS should be straightforward. Happy to share work & provide further info if interested.

Posted by Henry Minsky on
I wrote the original "email-handler" code that was used in the ticket tracker to allow users to respond by email to the ticket system.

When a new ticket is assigned to a user or an already asigned ticket's state changed, the ticket system sends out email with the ticket id in the subject line (like "Coffee pot is broken again (TR#1023)"), and a reply-to address of an alias that handled incoming mail with a perl/DBI script.

If you replied to the email (and your mailer preserved the subject line), it would get shoved into the database
using the "email-handler" into a big queue, tagged with the
a symbol saying which app it was destined for. Then a periodic
daemon would sweep the queue and call the appropriate module-specific
email-handler proc for each message.

You could also play qmail games with the reply-to address to encode the ticket id in the email address itself, although we didn't implement this (but we did implement something similar for handling
bounced email intelligently by encoding the user_id in the envelope return path address, in the "bulkmail" module).

Anyway, it worked fine, and still works in ACS 3.4, I believe, if you enable
email handling and set up your qmail aliases.

We could make this work for bboard in almost exactly the same
way as it works for ticket tracker.

I don't think it's desirable to drag Java into the email handling, it is simple enough to be done in Tcl. There might be some border cases where simple minded parsing of the message doesn't really work, but
in most cases the simple minded approach seems to work.

Posted by Neophytos Demetriou on
Here are some thoughts/suggestions:

* BBoard (and all application packages in general) should be as autonomous as possible. Most of their "extra" functionality should be configurable. For example, if you want to add ratings in a bboard you should be able to do that by creating an instance of the message-ratings package under the bboard node in the site map.

* One BBoard package instance = One BBoard (You can use the site map to mount create more instances)

* As Henry said there is no reason to use the content-repository for BBoard.

Posted by Dan Wickstrom on
I would have to disagree. Having the bboard tied into the content repository has the potential for real benefits. The content repository provides the following:

  • Search interface
  • Versioning
  • Categorization
  • Common storage with other content and the ablity to add attributes to content.

I think searching is especially important, and having a common storage area for content, might make it easier to implement a search interface in postgresql. If we start tearing apart packages and implementing things like search on a ad-hoc basis, then we are really moving back towards the acs 3.x way of doing things.

It seems premature to throw out the content repository - especially since performance has not been proven to be an issue yet.

A better approach would be to do a straight port of the bboard package, and then evaluate the performance and decide if scaling and performance will be issues. Since openacs 4.x supports packages, an alternative might be to develop a bboard-lite package that doesn't depend on the content-repository or acs_objects.

Posted by Ken Kennedy on
I agree with Dan here...let's port the existing stuff, where possible,
and then look to improve it.
Posted by Neophytos Demetriou on
Dan, just to be sure, my first suggestion above is not related to the storage mechanism. It is loosely related to inter-package communication. For example, you can have many instances of the bboard package (content repository or lite) using different ratings packages.
Posted by Dan Wickstrom on
I was only responding to your last comment and Henry's earlier comments regarding the usage of the content repository for storing bboard messages.

I've been working with the CR lately, and while I agree that it has a complicated interface, I've found that you can get by with a very small subset of that interface to tie a package's content storage into the content repository.  I've looked at the bboard code and it appears that the bboard package is not a very sophisticated user of the CR.  bboard messages are stored in the content repository, and the bboard package also provides the capablity to attach images or files to bboard messages.  That's it.  Unless, I've missed something, you really only need a handful of api calls to implement the bboard interface to the CR.

On the other hand, as Henry suggests, it seems likely that there will be performance problems with the CR.  All content revisions are stored in a single table regardless of whether they are live or not, so over time, the cr_revisions table could grow to be quite large, which could lead to performance problems.

I'm not saying we shouldn't consider redesigning the CR, but I think we should at the minimum, port the existing packages and evaluate them first before we go off and change them.  Porting a package like bboard is trivial, so it seems like a waste to change it before we know how well it does or doesn't work.

There's alot of good ideas in acs 4.x, and I think we should try and find out which ones work well before we change things.

Posted by Talli Somekh on
There's alot of good ideas in acs 4.x, and I think we should try and find out which ones work well before we change things.

Considering that we are *this close* to having the whole thing ported (or at least the kernel) I think that Dan's point is very well taken.

Currently there are very few people intimately familiar with ACS positives and negatives. It would seem that once the entire system, as aD left it, is available in OpenACS there will be a whole bunch of people banging on it. So many people will be available to find the warts and identify creative ways of fixing them.

Also, any substantial fix delays the release, even by a little bit. I hate to be the nudge here, but I think it needs to be very, very clear how many people are staking a lot on this port (or at least how much we are, if anyone gives a s**t). We will be contributing quite heavily to the port once it begins because there are so many modules we'll need for current clients. So we're really banking that the core modules will be ready and working according to the ways (even if they suck) we're familiar with.

I don't want to sound like a curmudgeon or someone that is ungrateful for the considerable work the porters are doing. Also, I think these suggestions are absolutely great. I just wonder whether it would be more effective to consider these suggestions as feature requests that will be implemented for the next release.

Is this basically what you meant, Dan?


Posted by Neophytos Demetriou on
On the other hand, as Henry suggests, it seems likely that there will be performance problems with the CR. All content revisions are stored in a single table regardless of whether they are live or not, so over time, the cr_revisions table could grow to be quite large, which could lead to performance problems.
That is why I suggested/prefer the "lite" approach. I don't think that versioning or adding attributes is needed for bboard and searching can be accomplished by implementing an interface with a method that provides the data to be indexed (Site Wide Search Design Document). I think this is what the current bboard package does.
I don't want to sound like a curmudgeon or someone that is ungrateful for the considerable work the porters are doing. Also, I think these suggestions are absolutely great. I just wonder whether it would be more effective to consider these suggestions as feature requests that will be implemented for the next release.
I agree.
Posted by Bryan Quinn on
I'll contribute something on the issue of whether to do redesign now or later. The conversation you're having parallels some discussions we've had here. The debate over whether to use the content repository recapitulates a discussion several of us had last August. The design is being changed for the next version. The design for messaging, and mail, etc is also changing.

Design questions generally take a long time to resolve. You'll deliver a working product much faster if you put off the design questions and just port the queries. Its not as intrinsically interesting a project to work on then, but you get something usable faster. Also, since you are matching the ACS 4.0 requirements and design, you have a concrete base line to evaluate your results against. This makes the port much easier.

War story: We had the opportunity to make design changes for the ACS Java 4.0 port. We decided not to. The only new code and design was infrastructure to make the port possible or easier (similar to the QueryDispatcher idea). Otherwise, we kept the design the same. This allowed us to port the code in a definable and short period of time. Introducing design considerations makes it much harder to deliver code by a certain date.

What I would encourage you to do is to refactor the ACS code. If you are porting a query or method and see a way of making it clearer or removing a bug, you should change that. I wouldn't add features, as they add to your testing and documenting load.

Hope this is helpful. Much of the reason the ACS is the way it is is because we prioritized our own client needs above technical aesthetics. Unless you identify a fundamental problem in the design, such as a scaling or upgradeability problem, getting the code to the user sooner has a better payoff. The users will have your code, be doing interesting things with it, and giving you feedback on how to make it better that is often much more valuable than any internal design discussion.

Posted by Jon Griffin on
I will also weigh in.

While a company that derives revenue from a product might have the priorities you mention, this is an Open Source non-commercial port.

Things that should get fixed but aren't, never are (unless there is a major problem such as permission_p).

The problem is that you put kludge on kludge on kludge and by version 4.5 it will be a nightmare to maintain. An example is places, if anyone ports that and address book that will be a huge shame. Another reason is if you port bboard the way it is, then when you want to eliminate acs-messaging you can't because it will break something somewhere. It really is much better to design one core package that really works and not have to worry about backward compatibilty at this stage (just starting out). After the version is released of course you will have to worry about backward compatiblilty.

Posted by Don Baccus on
Yes, Bryan's philosophy matches my own very closely.  Let's get Humpty  Dumpty back together and sitting on the wall first and decide on whether we want an omelette or souffle later.  Otherwise he might rot before we make up our minds.

I feel especially strongly about this because porting to the OpenACS 4.x structure, including porting queries to Postgres, isn't that difficult.  At the same time it's rather tedious and we need to settle  in, and get that tedious work behind us.  Distractions are risky.

There are some things we *have* to redesign and rewrite, i.e. Oracle Java-dependent SMTP handling and the like.  With these things on our plate along with porting, we've got plenty of work to do for round one.

And, believe me, this is just round one.  Don't get too caught up on the need to make everything absolutely perfect right off the bat.  A bunch of us are in this for the long haul.

I do want to encourage continued discussions of this sort, though, as long as we actually make progress porting existing code in parallel.

The whole bboard issue is one which many people care about, and none of the existing aD solutions are perfect.  But we can get them up in a hurry.  Remember, the first bboard port to PG took me 10 hours spread over a Saturday and Sunday.  We can afford to throw a port of the existing 4.0 bboard away later if we decide it sucks so bad that we really need to do so.  We won't have that much effort invested in the port.

And of course there's no reason there has to be a single bboard package.  There can be more than one, which users can choose.  One advantage of using the Content Repository is that other pieces of OpenACS can plug into the content no matter what bboard package is being used.  Keep this in mind when you consider dumping the CR...different bboard packages become different skins over content.

I need to add more documentation to my little wimpypoint presentation, to make sure that people do take notes and them as features and bugs in the SDM entry for the package they're working on.

And absolute horror stories need to be fixed.  I'm doing a bit of that with the APM, which needed work anyway in order to add multiple-db support to packages in a visible way.  We've seen some progress on the  permissions front.  In the case of datamodel problems we want to work  with aD whenever this is possible.

As far as specifics I'll only comment on Henry's comment about the sort key used by Philip in the original bboard code.  It is indeed a good way to track things and is a standard way of implementing hierarchies in the SQL world.  I work with a taxonomy of North American birds that uses a very similar technique, though it is slightly more rigid, since the taxonomy's known in advance.  It works very well, as does the original bboard scheme.

Not only is CONNECT BY not portable, but it tends to lose its seductive charm as you try to do ever-increasingly complex things with it.  Datamodels that depend on CONNECT BY queries seem somewhat fragile to me.

Posted by Jon Griffin on
After Don's post I will do the following:
  • port acs-messaging/acs-mail as is.
  • find a workaround for acs-notifications stuff (i.e. integrate it with acs-mail somehow)
Posted by Bryan Quinn on
FWIW, I think what Don says is exactly right. I'm always interested to read about the "absolute horror stories." :-)

Jon, what you mention is a serious problem in software development that affects the ACS as well. There are "broken windows" or pieces of code that have problems, are hacks, or have questionable designs that persist for a long period of time. The presence of these windows is demoralizing and causes other parts of the system to slip. The Pragmatic Programmer has a great discussion of this, and other software engineering problems.

We try to identify such problems early and find ways to resolve them, but it is fundamentally difficult, espescially with short schedules. Creating a package that offers parallel functionality later is an acceptable way to address this, particularly if you were able to work out an interface in advance. Using the SDM to record the presence of broken windows with as much context as possible when you encounter has worked out great for us; it really is a fantastic tool. Keeping a thread running about the problems that you find is a great way to start a conversation and get ideas, and many of us at aD find it very interesting to follow along with what is posted here. Once you've got the problems out on the table, you can prioritize and address them in future versions.

Tangent: The conversations here are exactly what I hoped we'd see when ACS 4.0 got released, but, with few exceptions, never really happened. One of the claimed advantages of open source is that people will read and hack the code and let you know about these things. I think aD has always been open to such conversations and contributions in the form of patches, but what has been lacking was a concrete goal to organize non aD-developers around to really hack the code. Supporting multiple databases is a goal that has the quality of providing that motivation and excitement that was missing for so long.

Posted by Henry Minsky on
Thanks Don (and Dan and others) for the voice of experience, and Jon for pushing the porting of these modules through. I agree with people's observations that there is a relatively large return on the small (relative to design and implementation of new code) effort invested in porting working existing code. Having something to show quickly will be a big boost for future development. An improved bboard can then be done on top of that or as a parallel package.
Posted by Tom Mizukami on
Bryan, you and aD just don’t get opensource development. It is not just happenstance that substantive discussion is occurring here and real discussion has all but dried up on the aD boards. Take a look at the ACES thread, this is aD software but the real discussion is here because developers believe in the openness of the project here and have given up on the process at aD. Just take a look at the development of 5.0, and you wonder where the real conversations have gone. You don’t need a VP of community development or organization of non-aD developers, what aD needs to do is to believe in the opensource process and let “claimed advantages” turn into real advantages.
Posted by Pascal Scheffers on
(...) Things that should get fixed but aren't, never are (unless there is a major problem such as permission_p).

The problem is that you put kludge on kludge on kludge and by version 4.5 it will be a nightmare to maintain. (...) when you want to eliminate acs-messaging you can't because it will break something somewhere. (...)
-- Jon Griffin

I would like to disagree with that. First, though, I have to admit that I do not know the 4.x bboard design other than what I've picked up on this thread and seen in general-comments. From the issues I see in this thread I come to the conclusion that:
  1. There is a lot of interest in the 'perfect bboard', and I am also counting previous discussions on messaging, IM, email support for bboards, etc.
  2. the problems are so severe that there will be no way around re-engineering a to get a better messaging/bboard system.
So, yes, I do think the messaging/bboard/CR will be rebuilt, possibly from scratch, *hopefully* as soon as the port is done. That is what point-releases are for, this sounds like an OpenACS 4.1(?) thing. I think the core team is strong enough to pick that up and they will have support. To me that means it is good to have a straight port now.

Although generally annoying, breaking parts of the existing API in the next core release will happen no matter what. If that would not happen, we will likely get serious problems with design-rot - we are not gods, after all :-) Sure, that means that you will break some or a lot of packages and those will have to be upgraded to the new API. It sucks, but that's life. OTOH, a lot of things will not change so much that parts of the old API can't still exist for backward compatibility. That will give package maintainers some time to catch up.

Don said that a bunch of us are in this for the long haul. That would include me, although I've only recently joined. There appears to be no (opensource) competitor for what OpenACS does. There are probably more people like me who want to set up their semi-private community websites that will hopefully survive a slashdotting.

26: Re: Brian's contribution (response to 1)
Posted by Steve Crossan on
Brian - I hear your point and it's well made (about attending to client needs) but on the other hand, aren't you having to do the redesign now anyway (with the move to ACS5)?
Posted by Bryan Quinn on
I want to reply to Tom and Steve, but don't want to tangent this thread any further. Its been a great discussion, and if people have further questions, please start a new thread or email me.

Tom: I think we are misunderstanding each other and I should have added some more context. Really all I'm saying is that a goal is a necessary condition for organizing an open-source community. ArsDigita never really supplied that goal for ACS4, and I suspect that is one reason why we were frustrated. Supporting multiple databases is a great goal, and the work that OpenACS is doing is great! There is more I could write considering the theory of this, but I'll save it for another time.

Steve: That is correct. We identified a set of business goals that required the redesign.