4.6 will cache the permission check (strange, I thought I'd checked that in but apparently haven't, I will later today though). This should help considerably.
We've also talked about specifying a special directory for static files that don't need to be checked, but that won't happen until 4.7.
