Forum OpenACS Development: What is the general usefulness of the dir_all_users view?

The directory package has a view:
create view dir_all_users as
   select p.party_id as user_id, p.email, pe.first_names, pe.last_name, p.url
     from parties p, persons pe
    where p.party_id = pe.person_id;
 replaced by all_users now in acs-kernal
There is a view cc_users in acs-kernel which provides this basic info, plus much more, but is signifigantly slower. Would it be a good idea to move this view into the core of OpenACS and fix directory to use this more general view? Obviously some applications will not be able to take advantage of this simpler view, but where it can be used there is a performance advantage.
Just to give people a flavor of how bad the cc_users view can be in postgresql, here is the plan generated for a fairly straightforward query:
explain select * from cc_users u, forums_messages f where u.user_id = f.user_id and message_id = 100;
NOTICE:  QUERY PLAN:

Nested Loop  (cost=1812.21..2367.32 rows=1 width=1442)
  ->  Index Scan using forums_messages_pk on forums_messages f  (cost=0.00..3.05 rows=1 width=923)
  ->  Materialize  (cost=2321.71..2321.71 rows=3404 width=519)
        ->  Hash Join  (cost=1812.21..2321.71 rows=3404 width=519)
              ->  Hash Join  (cost=1618.34..2059.76 rows=3406 width=503)
                    ->  Hash Join  (cost=1410.51..1792.44 rows=3403 width=479)
                          ->  Seq Scan on users u  (cost=0.00..288.27 rows=10227 width=241)
                          ->  Hash  (cost=1401.99..1401.99 rows=3408 width=238)
                                ->  Hash Join  (cost=1146.20..1401.99 rows=3408 width=238)
                                      ->  Seq Scan on parties pa  (cost=0.00..188.43 rows=10243 width=145)
                                      ->  Hash  (cost=1137.68..1137.68 rows=3408 width=93)
                                            ->  Hash Join  (cost=304.61..1137.68 rows=3408 width=93)
                                                  ->  Seq Scan on acs_objects o  (cost=0.00..655.31 rows=27031 width=85)
                                                  ->  Hash  (cost=296.10..296.10 rows=3408 width=8)
                                                        ->  Seq Scan on group_element_index  (cost=0.00..296.10 rows=3408 width=8)
                    ->  Hash  (cost=182.27..182.27 rows=10227 width=24)
                          ->  Seq Scan on persons pe  (cost=0.00..182.27 rows=10227 width=24)
              ->  Hash  (cost=168.29..168.29 rows=10229 width=16)
                    ->  Seq Scan on membership_rels mr  (cost=0.00..168.29 rows=10229 width=16)
cc_users is horrible in Oracle, too;  we have a 4.2 Classic site that's seeing significant slowness from it.  I haven't had a chance to do much with it, but it's definitely on my list.  Suggestions welcome!
It's not quite so bad on openacs.org:
Nested Loop  (cost=536.44..913.66 rows=1 width=379)
  ->  Nested Loop  (cost=536.44..909.41 rows=1 width=263)
        ->  Nested Loop  (cost=536.44..907.39 rows=1 width=247)
              ->  Nested Loop  (cost=536.44..904.52 rows=1 width=166)
                    ->  Hash Join  (cost=536.44..902.48 rows=1 width=64)
                          ->  Seq Scan on persons pe  (cost=0.00..105.27 rows=5927 width=28)
                          ->  Hash  (cost=534.96..534.96 rows=591 width=36)
                                ->  Hash Join  (cost=173.32..534.96 rows=591 width=36)
                                      ->  Seq Scan on parties pa  (cost=0.00..116.45 rows=5945 width=28)
                                      ->  Hash  (cost=171.84..171.84 rows=591 width=8)
                                            ->  Seq Scan on group_element_index  (cost=0.00..171.84 rows=591 width=8)
                    ->  Index Scan using users_pk on users u  (cost=0.00..2.03 rows=1 width=102)
              ->  Index Scan using acs_objects_pk on acs_objects o  (cost=0.00..2.84 rows=1 width=81)
        ->  Index Scan using membership_rel_rel_id_pk on membership_rels mr  (cost=0.00..2.01 rows=1 width=16)
  ->  Index Scan using forums_messages_pk on forums_messages f  (cost=0.00..4.23 rows=1 width=116)
Bad enough but at least some indexes are picked up. Jeff, did you VACUUM ANALYZE your test database?
Yeah, I vacuumed it. One thing on openacs.org is that I changed acs__magic_object_id to be with(iscachable). Making the same change on my side gives something that is still pretty horrible:
Nested Loop  (cost=11105.20..14252.40 rows=1 width=1482)
  ->  Index Scan using forums_messages_pk on forums_messages f  (cost=0.00..3.02 rows=1 width=983)
  ->  Materialize  (cost=13829.18..13829.18 rows=33616 width=499)
        ->  Hash Join  (cost=11105.20..13829.18 rows=33616 width=499)
              ->  Merge Join  (cost=0.00..2494.57 rows=33621 width=354)
                    ->  Index Scan using parties_pk on parties pa  (cost=0.00..812.76 rows=33637 width=146)
                    ->  Index Scan using users_pk on users u  (cost=0.00..1093.40 rows=33621 width=208)
              ->  Hash  (cost=11021.16..11021.16 rows=33616 width=145)
                    ->  Hash Join  (cost=2448.11..11021.16 rows=33616 width=145)
                          ->  Hash Join  (cost=1812.82..9713.52 rows=33621 width=129)
                                ->  Hash Join  (cost=1055.56..8642.72 rows=33628 width=99)
                                      ->  Seq Scan on acs_objects o  (cost=0.00..6014.54 rows=230454 width=91)
                                      ->  Hash  (cost=971.49..971.49 rows=33628 width=8)
                                            ->  Seq Scan on group_element_index  (cost=0.00..971.49 rows=33628 width=8)
                                ->  Hash  (cost=673.21..673.21 rows=33621 width=30)
                                      ->  Seq Scan on persons pe  (cost=0.00..673.21 rows=33621 width=30)
                          ->  Hash  (cost=551.23..551.23 rows=33623 width=16)
                                ->  Seq Scan on membership_rels mr  (cost=0.00..551.23 rows=33623 width=16)

I have not committed the acs__magic_object with(iscachable) since I am scared it will break something (oh, and isstrict should be there as well I think). Should I commit it?

I did a little more digging on the cc_users query plan and discovered that the problem seems to be that the genetic query optimization just gives a terrible result. The default threshold (GEQO_THRESHOLD) is 11 elements in the from clause and it turns out cc_users by itself is right on the threshold so when queried by itself it has a nice query plan but when joined it goes over the threshold and the (as far as I can tell -- horrible beyond words) genetic query optimizer takes over and we get these bad plans.

On openacs.org I did the following:

openacs.org-dev=# set geqo_threshold to 20;
SET VARIABLE
openacs.org-dev=# explain select * from forums_messages fm, cc_users u where fm.message_id = 55001 and u.user_id = fm.user_id;
NOTICE:  QUERY PLAN:

Nested Loop  (cost=0.00..17.27 rows=1 width=379)
  ->  Nested Loop  (cost=0.00..15.23 rows=1 width=351)
        ->  Nested Loop  (cost=0.00..13.21 rows=1 width=335)
              ->  Nested Loop  (cost=0.00..10.34 rows=1 width=254)
                    ->  Nested Loop  (cost=0.00..8.30 rows=1 width=152)
                          ->  Nested Loop  (cost=0.00..6.26 rows=1 width=144)
                                ->  Index Scan using forums_messages_pk on forums_messages fm  (cost=0.00..4.23 rows=1 width=116)
                                ->  Index Scan using persons_pk on persons pe  (cost=0.00..2.02 rows=1 width=28)
                          ->  Index Scan using group_elem_idx_element_idx on group_element_index  (cost=0.00..2.03 rows=1 width=8)
                    ->  Index Scan using users_pk on users u  (cost=0.00..2.03 rows=1 width=102)
              ->  Index Scan using acs_objects_pk on acs_objects o  (cost=0.00..2.85 rows=1 width=81)
        ->  Index Scan using membership_rel_rel_id_pk on membership_rels mr  (cost=0.00..2.01 rows=1 width=16)
  ->  Index Scan using parties_pk on parties pa  (cost=0.00..2.02 rows=1 width=28)
Which is a nice plan and the query is then quite fast.

I posted on pgsql-performance about this problem and got an answer from Tom Lane about what is really going on here. Says Tom Lane on pgsql-performance:

This is not actually using GEQO. The reason you are seeing an effect from raising geqo_threshold is that geqo_threshold determines whether or not the view will be flattened into the upper query. For this particular query situation, flattening the view is essential (since you don't want the thing to compute the whole view). The relevant source code tidbit is

/*
 * Yes, so do we want to merge it into parent?    Always do
 * so if child has just one element (since that doesn't
 * make the parent's list any longer).  Otherwise we have
 * to be careful about the increase in planning time
 * caused by combining the two join search spaces into
 * one.  Our heuristic is to merge if the merge will
 * produce a join list no longer than GEQO_RELS/2.
 * (Perhaps need an additional user parameter?)
 */

AFAICS, your only good solution is to make geqo_threshold at least 14, since you want a 7-way join after flattening.

regards, tom lane
"iscachable" only affects how often the function is called in an individual query (as in "once per unique set of parameters") so it's safe to use on any function that only depends on its input parameters (i.e. doesn't SELECT from the db or call uncachable functions etc).

So, yes, feel free to commit.

BTW there are more extensive cache options in 7.3 I think - at least they were discussed.  Maybe not 'til 7.4.

yes, the caching options got in. ("with iscachable" = immutable in the new stuff.)
These attributes inform the system whether it is safe to replace multiple evaluations of the function with a single evaluation, for run-time optimization. At most one choice should be specified. If none of these appear, VOLATILE is the default assumption.

IMMUTABLE indicates that the function always returns the same result when given the same argument values; that is, it does not do database lookups or otherwise use information not directly present in its parameter list. If this option is given, any call of the function with all-constant arguments can be immediately replaced with the function value.

STABLE indicates that within a single table scan the function will consistently return the same result for the same argument values, but that its result could change across SQL statements. This is the appropriate selection for functions whose results depend on database lookups, parameter variables (such as the current time zone), etc. Also note that the CURRENT_TIMESTAMP family of functions qualify as stable, since their values do not change within a transaction.

VOLATILE indicates that the function value can change even within a single table scan, so no optimizations can be made. Relatively few database functions are volatile in this sense; some examples are random(), currval(), timeofday(). Note that any function that has side-effects must be classified volatile, even if its result is quite predictable, to prevent calls from being optimized away; an example is setval().