acs_object_party_privilege_map or whatever the heck it is called is what you should join against - look for examples with grep elsewhere that use "exists" to check that the correct permission exists for the semantics you're trying to enforce with your query.
Calling permission_p from the Tcl level for each row is bound to be the slowest approach.
General rule of thumb:
1. To return permissions a user has on an object, call permission_p in the query for each row returned. In other words, to return (say) "admin_p" for an object a user has read perm on.
2. To restrict a query to return rows restricted by a permission check, use an "exists" clause as described above.
NEVER call permission_p in the where clause of your query. NEVER. Too slow.