Yes - indeed, that was part of the idea. Trade joining against a large table for joining against a smaller table.
Also you could actually keep joining against acs_object_context_index, and transform that table just into a flat (object_id,context_id) mapping.
Anyway - all said and done I don't think this would be a big win after all.
The bitstring privilege approach still feels like there should be something to gain from, but indeed the question is 'how much'.