Forum OpenACS Development: caching files

Collapse
Posted by Malte Sussdorff on
As many probably already know I use NGINX for load balancing and acting as a reverse proxy. Today I made sure that the portraits of the users can be served from NGINX instead of hitting the backend servers all the time.

I did this by redirecting the thumbnail view from /shared/portrait-bits.tcl to /image/$image_id/thumbnail and set up a directive in NGINX to cache the /image URL. Works like a charm.

Now I am thinking though of taking this one step further and provide caching for all files in the content repository. My idea here is to have something similar. I would create a page which takes the item_id and which does the permission checking . It then redirects to a separate page like /shared/file-cache.vuh which works on the revision_id instead of the item_id. Why revision_id? Because the revision_id will not change its content, whereas the item_id could.

NGINX would then be configured to cache everything coming from file-cache.vuh. A couple of thoughts though.

a) I would configure file-cache.vuh as an internal location, so only if its referred from the Backend server will it serve the file. This is to make sure people have a hard time tricking the permission checking.

b) We need to access the files probably like /shared/file-cache/$revision_id/$filename_with_extension so NGINX serves the correct headers for the file.

Before I go down that road, has anyone solved this problem already e.g. using Pound and lives to tell the tale? I have the slight suspicion that this idea lied behind the creation of image.vuh but i am not sure.

Collapse
2: Re: caching files (response to 1)
Posted by Dave Bauer on
I don't understadn this design at all.

This statement particularly is a problem
"This is to make sure people have a hard time tricking the permission checking." impossible to trick permission checking is better.

Basically anything that needs permission checking per user cannot be cached.

Cacing at a proxy server and private applications don't work together.

It works great for publiclly accessible content.

Collapse
3: Re: caching files (response to 2)
Posted by Malte Sussdorff on
Well, your sentence is not entirely correct. You can cache the files if you make sure you can access them only from the OpenACS page that does the permission checking.

https://blog.kovyrin.net/2006/11/01/nginx-x-accel-redirect-php-rails/

The above link gives a good explanation how this can be achieved with NGINX. My initial sentence came from the fact that you could hack the referrer URL if you knew how the OpenACS system is setup. Also take a look at X-Sendfile

https://blog.lighttpd.net/articles/2006/07/02/x-sendfile

The goal of my question is not about authentication, that is covered (see above), my goal is to have the content repository objects be served as full URLs, so without Form values like ?object_id=1234

Collapse
4: Re: caching files (response to 1)
Posted by Dave Bauer on
/file/${item_id}/${filename}

works. I thought I added file.vuh to acs-subsite.

Although it doesn't key on revision_id.

Could fix that I supposed.
like file/r/${revision_id}/${filename}

Collapse
5: Re: caching files (response to 4)
Posted by Malte Sussdorff on
That is great. Sadly now we are running into the problem (on the caching server) which caused us not to use one big directory for the CR in the first place: We run out of directory names 😊. Therefore I was thinking to use /file-cache.vuh which is called like /file-cache/$content/${filename} where content is the path to the file in the content repository. Then create a new parameter "behind caching server" and if set to "yes" both image.vuh and file.vuh would redirect to /file-cache and we are "fine". As discussed in IRC we need to make sure that the caching server does not allow access to /file-cache.vuh directly and that the files are only served if permission is given. https://openacs.org/forums/message-view?message_id=1163926 is a good description on how to do this with NGINX, which is similar to the X-Sendfile statement http://blog.lighttpd.net/articles/2006/07/02/x-sendfile.