Forum OpenACS Development: Re: What is the general usefulness of the dir_all_users view?

It's not quite so bad on openacs.org:
Nested Loop  (cost=536.44..913.66 rows=1 width=379)
  ->  Nested Loop  (cost=536.44..909.41 rows=1 width=263)
        ->  Nested Loop  (cost=536.44..907.39 rows=1 width=247)
              ->  Nested Loop  (cost=536.44..904.52 rows=1 width=166)
                    ->  Hash Join  (cost=536.44..902.48 rows=1 width=64)
                          ->  Seq Scan on persons pe  (cost=0.00..105.27 rows=5927 width=28)
                          ->  Hash  (cost=534.96..534.96 rows=591 width=36)
                                ->  Hash Join  (cost=173.32..534.96 rows=591 width=36)
                                      ->  Seq Scan on parties pa  (cost=0.00..116.45 rows=5945 width=28)
                                      ->  Hash  (cost=171.84..171.84 rows=591 width=8)
                                            ->  Seq Scan on group_element_index  (cost=0.00..171.84 rows=591 width=8)
                    ->  Index Scan using users_pk on users u  (cost=0.00..2.03 rows=1 width=102)
              ->  Index Scan using acs_objects_pk on acs_objects o  (cost=0.00..2.84 rows=1 width=81)
        ->  Index Scan using membership_rel_rel_id_pk on membership_rels mr  (cost=0.00..2.01 rows=1 width=16)
  ->  Index Scan using forums_messages_pk on forums_messages f  (cost=0.00..4.23 rows=1 width=116)
Bad enough but at least some indexes are picked up. Jeff, did you VACUUM ANALYZE your test database?
Yeah, I vacuumed it. One thing on openacs.org is that I changed acs__magic_object_id to be with(iscachable). Making the same change on my side gives something that is still pretty horrible:
Nested Loop  (cost=11105.20..14252.40 rows=1 width=1482)
  ->  Index Scan using forums_messages_pk on forums_messages f  (cost=0.00..3.02 rows=1 width=983)
  ->  Materialize  (cost=13829.18..13829.18 rows=33616 width=499)
        ->  Hash Join  (cost=11105.20..13829.18 rows=33616 width=499)
              ->  Merge Join  (cost=0.00..2494.57 rows=33621 width=354)
                    ->  Index Scan using parties_pk on parties pa  (cost=0.00..812.76 rows=33637 width=146)
                    ->  Index Scan using users_pk on users u  (cost=0.00..1093.40 rows=33621 width=208)
              ->  Hash  (cost=11021.16..11021.16 rows=33616 width=145)
                    ->  Hash Join  (cost=2448.11..11021.16 rows=33616 width=145)
                          ->  Hash Join  (cost=1812.82..9713.52 rows=33621 width=129)
                                ->  Hash Join  (cost=1055.56..8642.72 rows=33628 width=99)
                                      ->  Seq Scan on acs_objects o  (cost=0.00..6014.54 rows=230454 width=91)
                                      ->  Hash  (cost=971.49..971.49 rows=33628 width=8)
                                            ->  Seq Scan on group_element_index  (cost=0.00..971.49 rows=33628 width=8)
                                ->  Hash  (cost=673.21..673.21 rows=33621 width=30)
                                      ->  Seq Scan on persons pe  (cost=0.00..673.21 rows=33621 width=30)
                          ->  Hash  (cost=551.23..551.23 rows=33623 width=16)
                                ->  Seq Scan on membership_rels mr  (cost=0.00..551.23 rows=33623 width=16)

I have not committed the acs__magic_object with(iscachable) since I am scared it will break something (oh, and isstrict should be there as well I think). Should I commit it?

I did a little more digging on the cc_users query plan and discovered that the problem seems to be that the genetic query optimization just gives a terrible result. The default threshold (GEQO_THRESHOLD) is 11 elements in the from clause and it turns out cc_users by itself is right on the threshold so when queried by itself it has a nice query plan but when joined it goes over the threshold and the (as far as I can tell -- horrible beyond words) genetic query optimizer takes over and we get these bad plans.

On openacs.org I did the following:

openacs.org-dev=# set geqo_threshold to 20;
SET VARIABLE
openacs.org-dev=# explain select * from forums_messages fm, cc_users u where fm.message_id = 55001 and u.user_id = fm.user_id;
NOTICE:  QUERY PLAN:

Nested Loop  (cost=0.00..17.27 rows=1 width=379)
  ->  Nested Loop  (cost=0.00..15.23 rows=1 width=351)
        ->  Nested Loop  (cost=0.00..13.21 rows=1 width=335)
              ->  Nested Loop  (cost=0.00..10.34 rows=1 width=254)
                    ->  Nested Loop  (cost=0.00..8.30 rows=1 width=152)
                          ->  Nested Loop  (cost=0.00..6.26 rows=1 width=144)
                                ->  Index Scan using forums_messages_pk on forums_messages fm  (cost=0.00..4.23 rows=1 width=116)
                                ->  Index Scan using persons_pk on persons pe  (cost=0.00..2.02 rows=1 width=28)
                          ->  Index Scan using group_elem_idx_element_idx on group_element_index  (cost=0.00..2.03 rows=1 width=8)
                    ->  Index Scan using users_pk on users u  (cost=0.00..2.03 rows=1 width=102)
              ->  Index Scan using acs_objects_pk on acs_objects o  (cost=0.00..2.85 rows=1 width=81)
        ->  Index Scan using membership_rel_rel_id_pk on membership_rels mr  (cost=0.00..2.01 rows=1 width=16)
  ->  Index Scan using parties_pk on parties pa  (cost=0.00..2.02 rows=1 width=28)
Which is a nice plan and the query is then quite fast.

I posted on pgsql-performance about this problem and got an answer from Tom Lane about what is really going on here. Says Tom Lane on pgsql-performance:

This is not actually using GEQO. The reason you are seeing an effect from raising geqo_threshold is that geqo_threshold determines whether or not the view will be flattened into the upper query. For this particular query situation, flattening the view is essential (since you don't want the thing to compute the whole view). The relevant source code tidbit is

/*
 * Yes, so do we want to merge it into parent?    Always do
 * so if child has just one element (since that doesn't
 * make the parent's list any longer).  Otherwise we have
 * to be careful about the increase in planning time
 * caused by combining the two join search spaces into
 * one.  Our heuristic is to merge if the merge will
 * produce a join list no longer than GEQO_RELS/2.
 * (Perhaps need an additional user parameter?)
 */

AFAICS, your only good solution is to make geqo_threshold at least 14, since you want a 7-way join after flattening.

regards, tom lane