Forum OpenACS Development: BBoard development

Neophytos mailed me a copy of a new bboard package he has been working on. I will ask if he wants to place a copy where other people can review it, or if I can put up a copy.

I worked on a reimplementation of the bboard for a project at aD, when
ACS 4 just came out last year, and I had some thoughts about design.

Philg's original mechanism for the core bboard message threading data model (the one that's in
ACS 3.4) is actually pretty optimal, and doesn't rely on CONNECT BY,
which in turn has issues with sorting.

I am not sure if anyone has really tested the ACS 4 bboard (Oracle) for scalability of the data models with large numbers of forums and messages. They may have, I just don't know.

In any case, I found the current ACS BBoard module, with its dependencies on the acs-messaging, notifications, and content-repository, to be rather hard to work with. It is templated, which
is a big win, but I feel that the extra layers of abstraction that were added did more to increase the complexity and to make it more difficult to tune and debug the implementations, especially when you
want to customize the code.

So, I think returning to a simpler and flatter design would make sense. We should re-examine the goal of reusable code for general-comments, and for email notifications (both immediate and digestified) and see if there is some way to provide that without
making something that is difficult to work with.

I think the content repository is a useful idea for more conventional "publishing" types of tasks, but unless there is
a substantially simpler API to use it, it is overkill for the bboard,
and may have performance issues.

email notifications

I think making the bboard subsite aware is a definite requirement, but that is orthogonal to the issue of how the message store is implemented and how the email notifications are implemented.

I also like your idea of porting the old ACS3 bboard as a quick
and dirty way to get a tuned and burned-in bboard going. That code is quite ugly, but with some slashing and burning we could probably have something usable pretty quickly. I would only support  Q&A style presentation for example.  I think I will
try to do something like that, making just the minimal concessions towards cleaning it up,  templating it and subsite-awareness, and borrowing some of the basic permissions model from the ACS 4 bboard.

I posted some thoughts regarding messaging/mail at: https://openacs.org/bboard/q-and-a-fetch-msg.tcl?msg_id=0001aT&topic_id=12&topic=OpenACS%204%2e0%20Design

<i>Email support. This is a biggie. Participants need to be able to both reply to a thread and start a new one via email. </i>
<p>
yesyesyesyes. This is a very biggie. Part of the bummer of ACS3.x is that one needs to revisit the website each time they wanted to respond. However, I was under the impression that this was solved in 4.x. Am I wrong?

tal

I wrote the original "email-handler" code that was used in the ticket tracker to allow users to respond by email to the ticket system.

When a new ticket is assigned to a user or an already asigned ticket's state changed, the ticket system sends out email with the ticket id in the subject line (like "Coffee pot is broken again (TR#1023)"), and a reply-to address of an alias that handled incoming mail with a perl/DBI script.

If you replied to the email (and your mailer preserved the subject line), it would get shoved into the database
using the "email-handler" into a big queue, tagged with the
a symbol saying which app it was destined for. Then a periodic
daemon would sweep the queue and call the appropriate module-specific
email-handler proc for each message.

You could also play qmail games with the reply-to address to encode the ticket id in the email address itself, although we didn't implement this (but we did implement something similar for handling
bounced email intelligently by encoding the user_id in the envelope return path address, in the "bulkmail" module).

Anyway, it worked fine, and still works in ACS 3.4, I believe, if you enable
email handling and set up your qmail aliases.

We could make this work for bboard in almost exactly the same
way as it works for ticket tracker.

I don't think it's desirable to drag Java into the email handling, it is simple enough to be done in Tcl. There might be some border cases where simple minded parsing of the message doesn't really work, but
in most cases the simple minded approach seems to work.

I would have to disagree. Having the bboard tied into the content repository has the potential for real benefits. The content repository provides the following:

• Search interface
• Versioning
• Categorization
• Common storage with other content and the ablity to add attributes to content.

I think searching is especially important, and having a common storage area for content, might make it easier to implement a search interface in postgresql. If we start tearing apart packages and implementing things like search on a ad-hoc basis, then we are really moving back towards the acs 3.x way of doing things.

It seems premature to throw out the content repository - especially since performance has not been proven to be an issue yet.

A better approach would be to do a straight port of the bboard package, and then evaluate the performance and decide if scaling and performance will be issues. Since openacs 4.x supports packages, an alternative might be to develop a bboard-lite package that doesn't depend on the content-repository or acs_objects.

Dan, just to be sure, my first suggestion above is not related to the storage mechanism. It is loosely related to inter-package communication. For example, you can have many instances of the bboard package (content repository or lite) using different ratings packages.

There's alot of good ideas in acs 4.x, and I think we should try and find out which ones work well before we change things.

Considering that we are *this close* to having the whole thing ported (or at least the kernel) I think that Dan's point is very well taken.

Currently there are very few people intimately familiar with ACS positives and negatives. It would seem that once the entire system, as aD left it, is available in OpenACS there will be a whole bunch of people banging on it. So many people will be available to find the warts and identify creative ways of fixing them.

Also, any substantial fix delays the release, even by a little bit. I hate to be the nudge here, but I think it needs to be very, very clear how many people are staking a lot on this port (or at least how much we are, if anyone gives a s**t). We will be contributing quite heavily to the port once it begins because there are so many modules we'll need for current clients. So we're really banking that the core modules will be ready and working according to the ways (even if they suck) we're familiar with.

I don't want to sound like a curmudgeon or someone that is ungrateful for the considerable work the porters are doing. Also, I think these suggestions are absolutely great. I just wonder whether it would be more effective to consider these suggestions as feature requests that will be implemented for the next release.

Is this basically what you meant, Dan?

talli

I'll contribute something on the issue of whether to do redesign now or later. The conversation you're having parallels some discussions we've had here. The debate over whether to use the content repository recapitulates a discussion several of us had last August. The design is being changed for the next version. The design for messaging, and mail, etc is also changing.

Design questions generally take a long time to resolve. You'll deliver a working product much faster if you put off the design questions and just port the queries. Its not as intrinsically interesting a project to work on then, but you get something usable faster. Also, since you are matching the ACS 4.0 requirements and design, you have a concrete base line to evaluate your results against. This makes the port much easier.

War story: We had the opportunity to make design changes for the ACS Java 4.0 port. We decided not to. The only new code and design was infrastructure to make the port possible or easier (similar to the QueryDispatcher idea). Otherwise, we kept the design the same. This allowed us to port the code in a definable and short period of time. Introducing design considerations makes it much harder to deliver code by a certain date.

What I would encourage you to do is to refactor the ACS code. If you are porting a query or method and see a way of making it clearer or removing a bug, you should change that. I wouldn't add features, as they add to your testing and documenting load.

Hope this is helpful. Much of the reason the ACS is the way it is is because we prioritized our own client needs above technical aesthetics. Unless you identify a fundamental problem in the design, such as a scaling or upgradeability problem, getting the code to the user sooner has a better payoff. The users will have your code, be doing interesting things with it, and giving you feedback on how to make it better that is often much more valuable than any internal design discussion.

Yes, Bryan's philosophy matches my own very closely.  Let's get Humpty  Dumpty back together and sitting on the wall first and decide on whether we want an omelette or souffle later.  Otherwise he might rot before we make up our minds.

I feel especially strongly about this because porting to the OpenACS 4.x structure, including porting queries to Postgres, isn't that difficult.  At the same time it's rather tedious and we need to settle  in, and get that tedious work behind us.  Distractions are risky.

There are some things we *have* to redesign and rewrite, i.e. Oracle Java-dependent SMTP handling and the like.  With these things on our plate along with porting, we've got plenty of work to do for round one.

And, believe me, this is just round one.  Don't get too caught up on the need to make everything absolutely perfect right off the bat.  A bunch of us are in this for the long haul.

I do want to encourage continued discussions of this sort, though, as long as we actually make progress porting existing code in parallel.

The whole bboard issue is one which many people care about, and none of the existing aD solutions are perfect.  But we can get them up in a hurry.  Remember, the first bboard port to PG took me 10 hours spread over a Saturday and Sunday.  We can afford to throw a port of the existing 4.0 bboard away later if we decide it sucks so bad that we really need to do so.  We won't have that much effort invested in the port.

And of course there's no reason there has to be a single bboard package.  There can be more than one, which users can choose.  One advantage of using the Content Repository is that other pieces of OpenACS can plug into the content no matter what bboard package is being used.  Keep this in mind when you consider dumping the CR...different bboard packages become different skins over content.

I need to add more documentation to my little wimpypoint presentation, to make sure that people do take notes and them as features and bugs in the SDM entry for the package they're working on.

And absolute horror stories need to be fixed.  I'm doing a bit of that with the APM, which needed work anyway in order to add multiple-db support to packages in a visible way.  We've seen some progress on the  permissions front.  In the case of datamodel problems we want to work  with aD whenever this is possible.

As far as specifics I'll only comment on Henry's comment about the sort key used by Philip in the original bboard code.  It is indeed a good way to track things and is a standard way of implementing hierarchies in the SQL world.  I work with a taxonomy of North American birds that uses a very similar technique, though it is slightly more rigid, since the taxonomy's known in advance.  It works very well, as does the original bboard scheme.

Not only is CONNECT BY not portable, but it tends to lose its seductive charm as you try to do ever-increasingly complex things with it.  Datamodels that depend on CONNECT BY queries seem somewhat fragile to me.

FWIW, I think what Don says is exactly right. I'm always interested to read about the "absolute horror stories." 😊

Jon, what you mention is a serious problem in software development that affects the ACS as well. There are "broken windows" or pieces of code that have problems, are hacks, or have questionable designs that persist for a long period of time. The presence of these windows is demoralizing and causes other parts of the system to slip. The Pragmatic Programmer has a great discussion of this, and other software engineering problems.

We try to identify such problems early and find ways to resolve them, but it is fundamentally difficult, espescially with short schedules. Creating a package that offers parallel functionality later is an acceptable way to address this, particularly if you were able to work out an interface in advance. Using the SDM to record the presence of broken windows with as much context as possible when you encounter has worked out great for us; it really is a fantastic tool. Keeping a thread running about the problems that you find is a great way to start a conversation and get ideas, and many of us at aD find it very interesting to follow along with what is posted here. Once you've got the problems out on the table, you can prioritize and address them in future versions.

Tangent: The conversations here are exactly what I hoped we'd see when ACS 4.0 got released, but, with few exceptions, never really happened. One of the claimed advantages of open source is that people will read and hack the code and let you know about these things. I think aD has always been open to such conversations and contributions in the form of patches, but what has been lacking was a concrete goal to organize non aD-developers around to really hack the code. Supporting multiple databases is a goal that has the quality of providing that motivation and excitement that was missing for so long.

Bryan, you and aD just don’t get opensource development. It is not just happenstance that substantive discussion is occurring here and real discussion has all but dried up on the aD boards. Take a look at the ACES thread, this is aD software but the real discussion is here because developers believe in the openness of the project here and have given up on the process at aD. Just take a look at the development of 5.0, and you wonder where the real conversations have gone. You don’t need a VP of community development or organization of non-aD developers, what aD needs to do is to believe in the opensource process and let “claimed advantages” turn into real advantages.

Brian - I hear your point and it's well made (about attending to client needs) but on the other hand, aren't you having to do the redesign now anyway (with the move to ACS5)?