The view is now literally a couple of orders of magnitude faster in some (not all) cases in Oracle, varies little in performance whether an object gets a privilege directly or through inheritance, and can be much, much more than a couple of orders magnitude faster in PostgreSQL (the view would occasionally eat all virtual memory and cause Postgres to get very unhappy.)
I don't think your statement's really true anymore. I added the permissions check into SloanSpace's file-storage folder display query on their test server (they'd taken it out for the very reason you state) and it only slowed the query by about 10-15%. That's not a bad penalty at all for the fine-grained control you get (if you need it.)
I forget how many files were in their system at the time but I know there were over 850 URLs alone and that the vast majority of entries were not URLs. 10-20K files at least and a half-million objects ...
YMMV of course.