Forum OpenACS Development: openacs.org upgrade: things do be done

I post this in the development forum, as most of the stuff that needs to be done is development after all. Let's give a quick overview:
  • Bug Tracker must be enhanced in speed
  • Forums entry page needs to get faster
  • Search is not working
  • Neither is CVS view
  • We should cleanup ETP instances or at least think about how to best deal with them.
Collapse
Posted by Don Baccus on
I'm working on the list builder pagination speed issue, which will address the forums index page issue.  It will also allow us to paginate the bug index page which addresses the first item on your list.

Someone really needs to pick up the search issue that's crucial for our image.

Collapse
Posted by Don Baccus on
Sorry my work on the list builder will speed the forum thread view page, not the forums front page.  The forums front page is probably suffering from COUNTing threads in the query.  This is very fast in Oracle, slow in PG and always will be for reasons I won't get into here.  We should modify the forums datamodel to track thread and reply counts via triggers rather than do COUNTs in queries ...
Collapse
Posted by Don Baccus on
Some minor things ...

1. There's some annoying HTML appearing in the summary forum description, apparently that's quoted now.  Should we allow HTML in the forum description?

2. Forum search doesn't use FTS for PG???  Totally not scalable.

3. the little doggie icon doesn't show up in my mozilla tabs now, did we lose some CSS or something?

And /repository doesn't seem to be up?
Collapse
Posted by Malte Sussdorff on
/repository is back
Collapse
Posted by Torben Brosten on
re. 3. copy over the www/favicon.ico file

4. What happened to the openacs.org/repository/ ? There was an important doc there that identifies packages and their maturity status.

5. h r e f tags are currently not permitted in html forum posts

Collapse
Posted by Malte Sussdorff on
3) Favicon.ico file is back
4) Repository is back
5) href attribute is allowed
Collapse
Posted by Don Baccus on
Why doesn't the favicon show up when I'm in the forums?  Missing master tag or some such somewhere?

Also just to clarify (though it was probably clear to you) the non-FTS forums search works, it's the site-wide FTS based search that doesn't work.  The FTS engine's not been installed?

Collapse
Posted by Don Baccus on
Preview of a forum post doesn't work, and this is true on 5.1 tip, too.
Collapse
Posted by Randy O'Meara on
It appears that the "Thread:" hyperlink contained in these forum email notifications is incorrect, at least in one instance.

The post that started this thread has message_id=238612. However, the Thread: hyperlink in the email of the post has message_id=238610. Launching a browser on the link results in a "there was a server error processing your request" message.

This is the only message (I've checked several, including the first in another thread) that seems to have this problem.

Collapse
Posted by Don Baccus on
I think we're proving the importance of testing our software by USING IT, today :)
Collapse
Posted by Torben Brosten on
I can't close a bug (1801). maybe not all permissions were transfered?

Don, regarding favicon, my understanding is that mozilla/netscape only checks the first time it visits the site.. so once it's corrected, it still won't show on that particular browser. I miss the little puppy too. =(

maybe it's time to update browser?

Collapse
Posted by Don Baccus on
I have made a temporary hack (NOT to be CVS committed) to the forums threads summary page to speed the non-cached query by a whole bunch (a couple of seconds to load now rather than 18).

I've had to disable caching of rows for that page temporarily but I think the faster average time and losing the horribly slow worst case after each post is worth it.

This is not related to fixing list builder's pagination ... strictly a short-term hack.

Collapse
Posted by Andrew Piskorski on
The dates and times are gone from all the Forums postings. This is very bad, please again display both the date and the time of every single post in the thread.
Collapse
Posted by Dossy Shiobara on
I second this request. It's very important to know when a message or reply was posted. Jacob Nielsen would agree with me! 😊
Collapse
16: Use YYYY-MM-DD format (response to 1)
Posted by Andrew Piskorski on
Also, the Forums posting history page for each user shows dates like "04/06/04" - what is this nonsense? Is that June 6th, or July 4th? OpenACS always has the sense to use ISO YYYY-MM-DD dates in the past, why change to a ridiculous ambiguous date format now?
Collapse
Posted by Andrew Piskorski on
(Oops, I meant to say, "Is that June 4th, or April 6th?")
Collapse
Posted by Ben Koot on
Has anybody found a solution how to fix this issue.

I just launched a travel trade forum and received this comment....

08/09/05 means 08 September in 85% of the world and 09 August in 15% of it. Even that 15% (the USA) still officially uses the international system (just check what the US immigratiuon people use on their forms) so could we, please, use an unambigious system? The one the travel industry uses worldwide is day (in figures)/month (in letters)/year (in figures) thus:
08 Aug 2005. Clear and unambigious. Can we have it?

Thanks
Ben

Three queries that are up for improvement (taken from the frontpage):

4603 ms       select       dbqd.edit-this-page.tcl.etp-procs.etp::get_content_items.get_content_items: select nsdb0

select * from
  (select
  i.item_id, i.name, tree_sortkey as sort_order,
  to_char(r.publish_date, 'Mon DD, YYYY') as publish_date,
  (select object_type from acs_objects
    where object_id = i.item_id) as object_type,
          etp__get_relative_url(i.item_id, i.name) as url,
          etp__get_title(i.item_id, r.title) as title,
          etp__get_description(i.item_id, r.description) as description

  from cr_items i left join cr_revisions r on (i.live_revision = r.revision_id)
  where i.parent_id = etp__get_folder_id('3981') and i.name != 'index'
  ) attributes
where
1 = 1

order by
  sort_order

##################################

812 ms       select       dbqd.www.templates.widgets.postings.messages_select: select nsdb0

select fm.message_id,
      fm.forum_id,
      ff.name as forum_name,
      fm.subject

from (select message_id,
            forum_id,

Did I mention forum preview is broken?. Apparently preview did not like my content (it was too large), so it just cut it down in half. Either way, here are the other two queries in full beauty.

812 ms       select       dbqd.www.templates.widgets.postings.messages_select: select nsdb0

select fm.message_id,
      fm.forum_id,
      ff.name as forum_name,
      fm.subject

from (select message_id,
            forum_id,
            subject,
            parent_id,
            posting_date,
            last_child_post
      from forums_messages_approved
      order by last_child_post desc)  fm, forums_forums ff

where fm.forum_id = ff.forum_id
and fm.parent_id is null
and ff.package_id = 3061
and ff.forum_id <> 46408
--and ff.package_id = 3928
and ff.enabled_p='t'
-- and last_child_post > (current_timestamp - 7)
--order by forum_name
limit 11


###################

664 ms       select       dbqd.edit-this-page.tcl.etp-procs.etp::get_content_items.get_content_items: select nsdb0

select * from
  (select
  i.item_id, i.name, tree_sortkey as sort_order,
  to_char(r.publish_date, 'Mon DD, YYYY') as publish_date,
  (select object_type from acs_objects
    where object_id = i.item_id) as object_type,
          etp__get_relative_url(i.item_id, i.name) as url,
          etp__get_title(i.item_id, r.title) as title,
          etp__get_description(i.item_id, r.description) as description

  from cr_items i left join cr_revisions r on (i.live_revision = r.revision_id)
  where i.parent_id = etp__get_folder_id('4053') and i.name != 'index'
  ) attributes
where
1 = 1

order by
  sort_order

limit 4

Collapse
Posted by Malte Sussdorff on
The query on the forums frontpage is really slow, maybe someone has a clue on how to fix it ?:

 12213 ms  	 select  	 dbqd.forums.lib.forums.forums-chunk.select_forums: select nsdb0

select forums_forums_enabled.*,
       (select count(*)
        from forums_messages_approved
        where forums_messages_approved.forum_id = forums_forums_enabled.forum_id
        and parent_id is null) as n_threads,
       to_char(last_post, 'YYYY-MM-DD HH24:MI:SS') as last_post_ansi,
       case when last_post > (now() - interval '1 day') then 't' else 'f' end as new_p
from forums_forums_enabled
where forums_forums_enabled.package_id = '3061'
and (
        forums_forums_enabled.posting_policy = 'open'
     or forums_forums_enabled.posting_policy = 'moderated'
     or 't' = acs_permission__permission_p(forums_forums_enabled.forum_id, '7797','forum_read')
    )
order by forums_forums_enabled.nam
Collapse
Posted by Jade Rubick on
Perhaps we should keep track of all of these things by posting them in bug-tracker, under openacs.org? http://www.openacs.org/bugtracker/openacs.org/
Collapse
Posted by Don Baccus on
I'm planning to work on speeding this up in the next few days, along with the pagination stuff. Essentially I'll keep track of forum threads and thread replies with counters in the appropriate database row, like the old ACES forums did ...
Collapse
Posted by Dave Bauer on
Don, I added thread counting to forums including a prelimiary upgrade script for postgresql. It seems to be working except the count is missing for the development forums,I just noticied. It brings the forum index page down around 1 second.
Collapse
Posted by Dave Bauer on
I will add the reply date to the UI.
Collapse
Posted by Malte Sussdorff on
Thanks a lot for the hard work on forums Dave. This rocks.
Collapse
Posted by Dave Bauer on
Ok, since noone can decide on the style for the forums pages, I added some CSS code, that is driven by the package parameters of forums. This fixes the wide table problem. It also allows colors if the site owner chooses. I also added the posting date to the display. I fixed the home page by removing a pl/sql call in the where clause in an ETP procedure query. Now the query takes 14ms instead of over 3000ms.
Collapse
Posted by Matthew Geddert on
Some comments about the forums UI for openacs.org:
<ul>
<li>I've found on my sites that don't need threaded views of postings - i.e. where its just a flat list (such as openacs.org) that it simplifys things greatly to remove the reply link from each posting and only leaving that on the bottom (i.e. respond to the original post). It also removes UI "clutter" a bit (and reduces the number of queries needed to create a forum thread view - since they only need to look for one parent_id - i.e. the original post). I would recommend putting the forward button next to who posted the message</li>
<li>Having the who posted what text be smaller is also really nice visually (<code>font-size: small; color: #CCC;</code> or something like that)</li>
<li>Likewise It helps if there is some visual seperation of posts. A simple #EEE hr line is enough. Having smaller who posted what might be enough visual seperation. I am partial to a UI i developed for one site i.e. which uses Lists and CSS to do all this display stuff (no need to tables) http://www.ministryquest.com/community/forums/message-view?message_id=24159</li>
<li>Also, if the title is just "RE: posting title" there is not need to post the title of the new post. Again, unnecessary visual clutter - its easy to remove this when looping through the posts</li>
</ul>
<p>These aren't urgent of course... Here are my recommendations that comply more with html and css standards to a previous post on this thread</p>
<pre>

<div style="border-bottom: 1px solid #EEE; padding: 10px; margin-bottom: 20px;">

<h3 style="font-size: normal; font-weight: bold; margin ">Re: openacs.org upgrade: things do be done</h3>

<p>Ok, since noone can decide on the style for the forums pages, I added some CSS code, that is driven by the package parameters of forums. This fixes the wide table problem. It also allows colors if the site owner chooses. I also added the posting date to the display. I fixed the home page by removing a pl/sql call in the where clause in an ETP procedure query. Now the query takes 14ms instead of over 3000ms.</p>

<p style="font-size: small; color: #999999"><cite>Posted by <a href="">Dave Bauer</a> on December 06, 2004 03:39 PM</cite> [ <a href="">forward</a> ]</p>

</div>

</pre>

Some comments about the forums UI for openacs.org: I've found on my sites that don't need threaded views of postings - i.e. where its just a flat list (such as openacs.org) that it simplifys things greatly to remove the reply link from each posting and only leaving that on the bottom (i.e. respond to the original post). It also removes UI "clutter" a bit (and reduces the number of queries needed to create a forum thread view - since they only need to look for one parent_id - i.e. the original post). I would recommend putting the forward button next to who posted the message Having the who posted what text be smaller is also really nice visually (font-size: small; color: #CCC;) or something like that) Likewise It helps if there is some visual seperation of posts. A simple #EEE hr line is enough. Having smaller who posted what might be enough visual seperation. I am partial to a UL or OL lists i developed for one site i.e. which uses Lists and CSS to do all this display stuff (no need to tables) http://www.ministryquest.com/community/forums/message-view?message_id=24159 Also, if the title is just "RE: posting title" there is not need to post the title of the new post. Again, unnecessary visual clutter - its easy to remove this when looping through the posts These aren't urgent of course... Here are my recommendations that comply more with html and css standards to a previous post on this thread: <div style="border-bottom: 1px solid #EEE; padding: 10px; margin-bottom: 20px;"> <h3 style="font-size: normal; font-weight: bold; margin ">Re: openacs.org upgrade: things do be done</h3> <p>Ok, since noone can decide on the style for the forums pages, I added some CSS code, that is driven by the package parameters of forums. This fixes the wide table problem. It also allows colors if the site owner chooses. I also added the posting date to the display. I fixed the home page by removing a pl/sql call in the where clause in an ETP procedure query. Now the query takes 14ms instead of over 3000ms.</p> <p style="font-size: small; color: #999999"><cite>Posted by <a href="">Dave Bauer</a> on December 06, 2004 03:39 PM</cite> [ <a href="">forward</a> ]</p> </div>
Collapse
Posted by Matthew Geddert on

ARGH, i need to get used to this new system. Sorry about the duplicate posts it is impossible to see what I'm talking about without decent formatting. If one of you with permission can delete the two previous posts and this part of this message I would appreciate it. By the way pressing edit on preview removed some of the content on this message

Some comments about the forums UI for openacs.org:

  • I've found on my sites that don't need threaded views of postings - i.e. where its just a flat list (such as openacs.org) that it simplifies things greatly to remove the reply link from each posting and only leaving that on the bottom (i.e. respond to the original post). It also removes UI "clutter" a bit (and reduces the number of queries needed to create a forum thread view - since they only need to look for one parent_id - i.e. the original post). I would recommend putting the forward button next to who posted the message
  • Having the who posted what text be smaller is also really nice visually (font-size: small; color: #CCC; or something like that)
  • Likewise It helps if there is some visual separation of posts. A simple #EEE hr line is enough. Having smaller who posted what might be enough visual separation.
  • Also, if the title is just "RE: posting title" there is not need to post the title of the new post. Again, unnecessary visual clutter - its easy to remove this when looping through the posts

These aren't urgent of course... Here are my recommendations that comply more with html and css standards to a previous post on this thread. Since its has the same title the stuff in H3 tags would be omitted (this would be done either in tcl or via the multiple loop). This would preferably be done via a .css file and classes.



<div style="border-bottom: 1px solid #EEE; padding: 10px; margin-bottom: 20px;">

<h3 style="font-size: normal; font-weight: bold; margin ">Re: openacs.org upgrade: things do be done</h3>

<p>Ok, since noone can decide on the style for the forums pages, I added some CSS code, that is driven by the package parameters of forums. This fixes the wide table problem. It also allows colors if the site owner chooses. I also added the posting date to the display. I fixed the home page by removing a pl/sql call in the where clause in an ETP procedure query. Now the query takes 14ms instead of over 3000ms.</p>

<p style="font-size: small; color: #999999; padding: 0px; margin: 10px 0px 0px 0px;"><cite>Posted by <a href="">Dave Bauer</a> on December 06, 2004 03:39 PM</cite> [ <a href="">forward</a> ]</p>

</div>

Collapse
Posted by Don Baccus on
Sniff I helped too Malte :( :( Just kidding ... I agree with the comment about removing the REPLY if the presentation's flat, along with the post number etc. We need to fix preview of course. Dave, when will we get to see that css, it looks like it's not set at the moment? I changed the front page query to group recent posts by forum name, looks more sensible IMO and it's still fast. And of course Dave needs to remember to add the STABLE attribute to the offending ETP PL/pgSQL procs so all sites, not just openacs.org, can be nice and fast without using magic object numbers :) I have figured out, finally, how to make pagination in the list builder 1) continue to use the paginator 2) to not require any changes to existing client code and 3) to STILL BE ROCKIN' FAST and to have the slowest of the two needed queries to display a page be cachable. Plan to implement it in the next day or two.
Collapse
Posted by Malte Sussdorff on
Sorry Don, for sure I'm thankful to your outstanding work as well 😊. And anyone else who was helping with comments, critizism, suggestions and pushing us along.
Collapse
Posted by Andrew Piskorski on
I disagree, the "reply" links even when the Forums display is flat ("Q&A" style) are still useful, and should be kept. Forums keeps track of the threaded discussion underneath, and uses that when displaying only part of the full thread, so replying to a particular post rather than to the thread as a whole has a specific and useful effect.
Collapse
Posted by Malte Sussdorff on
In addition to the "reply" links, we should also add the "in reply to" links back in, so even in Q&A style you can see who has replied to what thread.
Collapse
Posted by Dave Bauer on
This is the CSS version. You are looking at it. What I have done is supress all the borders and colors since there wasn't any consensus on whether they are helpful or distracting. The main benefit is that the posts in a thread are not all in a huge table, but each post is in a seperate DIV so that extra wide posts do not cause really long lines on regular posts.
Collapse
Posted by Alfred Essa on
thanks everyone!
Collapse
Posted by Matthew Geddert on

Thanks Dave!

I can still see two bugs:

  1. The top level post still shows the old date format - presumably direct from the db.
  2. When you "post a reply" (the screen I am entering this text into) the context bar forum link doesn't contain the forum_id. It just shows: <a href="./forum-view?forum_id=">OpenACS Development</a>
Collapse
Posted by Matthew Geddert on
Opps - thanks to Don, Malte, and everybody else as well that have volunteered their talent and time to make this transition!
Collapse
Posted by Andrew Piskorski on
The openacs.org login page is giving me the "remember my login on this computer" check box, however, it is expiring logins once per day or so. The old openacs.org did not do this, it let me stay logged in forever. Probably someone forgot to change one of the site-wide settings after installing the new software.
Collapse
Posted by Malte Sussdorff on
There is a distinction in openacs between persistent login (the system remembers permanently who you are) and the time it let's you use the site without having to submit a password. As the old behaviour did not know this distinction I changed the lifetime that you can surf the site without a password to permanent.
Collapse
Posted by Matthew Geddert on
The reply links on each message showing who responded to what message are nice if they are used consistetly (otherwise you miss some of the discussion on a subthread of a post - note that this message for example would not show up under yours since i used the button at the end of the thread). I know openacs keeps track of these things underneath if you use the reply to a particular response to a post feature - but that's precisely why I (and this is a personal opinion), and it looks like Don as well from his comment, think its unnecessary complication, db search as well as UI wise. Just becase it can do it doesn't mean its best to do it... and if you have a flat forum view i don't see why you would want threads...
Collapse
Posted by Rocael Hernández Rizzardini on
I tried and got this :(

Database operation "0or1row" failed (exception NSDB, "Query was not a statement returning rows.")

ERROR:  function acs_message__new("unknown", "unknown", timestamp with time zone, "unknown", "unknown", "unknown", "unknown", "unknown", "unknown", "unknown", integer, "unknown", "unknown", "unknown", "unknown", "unknown") is not unique
HINT:  Could not choose a best candidate function. You may need to add explicit type casts.

SQL:
        select acs_message__new (
        '239150',        -- 1  p_message_id

Collapse
Posted by Mark Aufflick on
Excellent work everyone!

I'm unfortunately not being much of an openacs-er at the moment due to my current contract as a Perl programmer, but like to check into openacs.org at least once a week for stimulating reading :)

I saw the "we are down for upgrading" message the other day and pitied you folk (since I have done a 4.2 to 5.1 upgrade myself, though not on the scale of openacs.org).

Everyone deserves a nice cold beer - I'll have one for you :)

PS: the default format should probably be preformatted text, and openacs.org causes a spellcheck error...

Collapse
44: (response to 1)
Posted by xx xx on
I need the direct download area at http://www.openacs.org/projects/openacs/download/download/ for an automated installler.

Can anybody restore this feature?

is the post list on the main page ordered by most recent post (top-down)? seems not ...
Collapse
Posted by Don Baccus on
That box on the main page is ordered by forum, then by the last child post descending. So within the list for each forum it is indeed most-recent-on-top. Originally it ordered posts chronologically and didn't group them by forum, I thought that was quite ugly and after discussing it in IRC with a couple of folks changed it to group by forum as it does not.
I see, makes more sense now than before, just that I was used to the previous way, thanks for the change.
Collapse
Posted by Joel Aufrecht on
I changed the openacs.org process to use the openacs user instead of the maltes user.

There are several errors in the openacs.org log on startup:

[08/Dec/2004:20:17:22][10927.4143927968][-main-] Notice: nsmain: AOLserver/4.0.9 starting
...
[08/Dec/2004:20:17:59][10927.4143927968][-main-] Error: Aborting transaction due to error:
Database operation "0or1row" failed (exception NSDB, "Query was not a statement returning rows.")

ERROR:  duplicate key violates unique constraint "acs_sc_impl_alias_un"
CONTEXT:  PL/pgSQL function "acs_sc_impl_alias__new" line 12 at SQL statement

SQL:
        select acs_sc_impl_alias__new(
                'FtsContentProvider',          -- impl_contract_name
                'journal_issue',                        -- impl_name
                'datasource',                  -- impl_operation_name
                'etp::revision_datasource',    -- impl_alias
                'TCL'                          -- impl_pl
        )
(three times)

[08/Dec/2004:20:18:05][10927.4143927968][-main-] Error: Error executing initialization code block search_engine_driver in /var/lib/aolserver/openacs.org/packages/openfts-driver/tcl/openfts-driver-in
it.tcl: Unable to locate /usr/local/src/Search-OpenFTS-tcl-0.1.
    while executing
"error "Unable to locate $openfts_tcl_src_path.""

...
[08/Dec/2004:20:18:07][10927.4115127216][-driver-] Notice: driver: accepting connections
Total start time is 45 seconds.

This error also occurs from time to time:
[08/Dec/2004:20:13:31][3638.4133436336][-sched-] Warning: irc::logger::apply_xslt - Could not parse /web/openacs.org/data/eavesdrop/log/2004-12-09.rdf: error "not well-formed (invalid token)" at lin
e 430 character 27
"<foaf:chatEven      < <--Error-- rdf:li>
      <foaf:chatEvent rdf:ID=""
[08/Dec/2004:20:13:31][3638.4133436336][-sched-] Error: can't read "rdf": no such variable
can't read "rdf": no such variable
    while executing
"$rdf delete"
    (procedure "irc::logger::apply_xslt" line 31)
    invoked from within
"irc::logger::apply_xslt  -rdf_log $irc_rdf_log  -xsl_style [parameter::get  -parameter xsl_stylesheet  -package_id $package_id  -default "[acs_package..."

Collapse
Posted by Joel Aufrecht on
Automated install from the repository seems to be broken. Several people have reported an error such as Error: Error loading APM file form url https://openacs.org/repository/5-1/logger-1.1b2.apm: child process exited abnormally child process exited abnormally while executing "exec [apm_gunzip_cmd] -q -c $file_path | [apm_tar_cmd] tf - 2>/dev/null" However, the repository appears to be complete and normal, with manifests and apm files that are correct and can be accessed normally via browsing or wget.
Collapse
Posted by Malte Sussdorff on
Whoever set the automated generation of the repository up, could you give a clue what needs to be done? The reason why there is a repository is because I copied the files over from the old installation.
Collapse
Posted by Joel Aufrecht on
I've moved some of the bugs and problems in this thread to the bug tracker. I also set up a development site at angora.furfly.net.
Collapse
Posted by Joel Aufrecht on
I support using a date format of "08 Aug 2005." Where do we need to make the change? I changed all the acs-lang strings on openacs.org, but that didn't fix forum.