Forum OpenACS Q&A: Status of file storage module?

Collapse
1: Status of file storage module?
Posted by Janine Ohmer on 08/29/00 12:20 AM
Has anyone tried to use the file storage module yet?  I'm thinking
about setting up a small site that would use it, and I'd like to have
an idea of what shape it's in before I start, since some people have
found that some of the more obscure modules aren't fully ported yet.
Collapse
2: Response to Status of file storage module? (response to 1)
Posted by Dan Wickstrom on 08/29/00 02:39 AM
I haven't looked at it since I ported it, but it should work fine.  If you run into any problems with it, let me know.
Collapse
3: Response to Status of file storage module? (response to 1)
Posted by Michael Feldstein on 08/29/00 04:07 PM
I've used it a bit. It seems to work fine, although I haven't really pounded on it.
Collapse
4: Response to Status of file storage module? (response to 1)
Posted by Ben Adida on 08/30/00 12:06 AM
We've used it intensively here at OpenForce. It doesn't scale well at all. This is partly due to BLOB storage (which is a hack, we all admit it), but mostly to some heinous complex queries in there that were never meant to be so complex.

Plus, there's my philosophy that storing these files in the database is a mediocre idea at best. There is no need for transactional control over the file content. Thus, we have built new-file-storage, which stores all files in the filesystem, has a more logical permission scheme (permissions linked to files instead of versions of files), and is generally much faster. I'm happy to distribute this code, I just didn't want to add yet another new module to the distribution.

Collapse
5: Response to Status of file storage module? (response to 1)
Posted by Don Baccus on 08/30/00 04:23 AM
Is aD planning to rewrite file-storage for 4.0?  If not, incorporating  Ben's replacement is a good idea (I wrestled with this module for an Oracle-based client and it is loathsome, though the time I had allocated for that portion of the project precluded doing anything other than some really simple improvements).

I see no reason to store the files in the DB, either.  In fact, for my  own website I store portraits in the filesystem rather than the DB for similar reasons, and I'm the guy who wrote the LOB hack!

Collapse
6: Response to Status of file storage module? (response to 1)
Posted by Michael Feldstein on 08/30/00 04:56 AM
I dunno. To my mind, if I'm putting a file on a web site, it's because I want to share it. And if I want to share it, I probably want version control.

I believe that aD is indeed working on a new version of file storage (if the slightly old documentation is to be believed, anyway), but it will have *more* emphasis on version control, not less. This suits my needs but clearly doesn't suit everyone's.

Collapse
7: Response to Status of file storage module? (response to 1)
Posted by Ben Adida on 08/30/00 05:32 AM
The new file storage does *not* cut back on versioning. The versions are still there, but the permissions map to files, instead of versions of files, because it really doesn't make sense for someone to have access to one version, but not the next.
Collapse
8: Response to Status of file storage module? (response to 1)
Posted by Janine Ohmer on 08/30/00 05:36 AM
But does having version control necessarily mean you have to store the body of the file in the database?  I don't see any particular reason why you couldn't keep your versions on disk, using a renaming convention, and then maintain the rest of the information in the database.

The biggest problem with storing files in the database, IMHO, is that it makes your dump files *huge* very quickly.  Since versions don't change once they're uploaded, if they were kept in disk files they would be backed up once and then skipped by incremental backups.  In the database, where most of us do backups of full data dumps, you end up backing up those unchanged versions over and over and over...

Collapse
9: Response to Status of file storage module? (response to 1)
Posted by Michael Feldstein on 08/30/00 09:25 PM
Doing version control on a per-file basis rather than a per-version basis makes a lot of sense, so long as users can't easily get confused about which version is the most recent one. As for storing files in a file system vs. a database, that's an issue that's way beyond me. I'm happy to let you hacker gods make that decision. 😊

BTW, out of curiosity, how does CVS store source code? In a database or a file system?

Collapse
10: Response to Status of file storage module? (response to 1)
Posted by Dan Wickstrom on 08/30/00 09:45 PM
For each source file, cvs keeps all the versions for the source file in one flat file.  I believe that the individual versions are stored as differences to minimize the size of the cvs archive file, and conncurrency is maintained by using file locking.