The all_party_privilege_map queries can be expensive, too. "nested loop" means just what it says, and when you have nested loops nested a few deep you're far, far from the O(log2(N)) comfort zone. We've been zapped with this on SSV2 ...
So unfortunately, with things as they are one must experiment.