Forum OpenACS CMS: content repository physical folders keeps growing

On a server for a customer I found myself out of disk space. I tought it was strange, because files in content repository are few compared to disk space: a sum on file size in fs_files returned around 3GB, but size of content-repository-content-files goes over 17GB.

Is it possible that files were not deleted when issuing a file_delete command?

I checked the count of revisions grouped on item_id in content_repository, because I was suspecting multiple revisions leftovers from file deletes/adds, but each item_id shows a count of 1, so no multiple revisions...

It seems really strange to me... How could I check files from content-repository-content-files who are not related to any content? I need this so I can identify folders I can remove safely.

The table "cr_files_to_delete" is empty right now, but this should be correct in current situation... I should try to remove some file and check it again.

I could find the init tcl file where the proc is scheduled.

One question: beside checking the logs, in "ad_schedule_proc" docs it says I should be able to see currently scheduled procs at "/admin/monitoring/schedule-procs.tcl" location on the server, but it seems this is not true anymore. I've also tried to grep for a "schedule-procs" file, but I couldn't find any. Which is the best current method to look at currently scheduled procs on a server?

From that page I can see "cr_delete_scheduled_files" is correctly scheduled, and that when a file is deleted, "cr_files_to_delete" table is updated correctly.

I think I have found my problem: as my application manages music playlists composed by many files, I developed a script which allows for the uploading of a big zip file to the server, containing many mp3s.

As something can go wrong during the insertions of the songs (because of wrong format or other issues), this operation can fail when some of the files were already put into content repository. As the operation is made into a db_transaction, all changes to the db are rolled back, but the same doesn't happen to the physical files in content-repository, which will remain there forever (no explicit deletion issued, so no entry into "cr_files_to_delete").

Would it be reasonable to change "cr_delete_scheduled_files" so it sweeps every file not having an entry into content repository tables? It would be more aggressive, but we could make it parametrical...

• uses the indexing as sketched above
• uses the "find" command instead of the function from the tcllib (more than 10 times faster).

all the best
-g

Here http://wiki.tcl.tk/776 they say "globfind" command could overcome some limitations of ::fileutil::find and achieve better performance. I was thinking about trying that command instead, but it is not part of the tcl OpenAcs requires right now, so I would need your opinion.

For my current situation, I fixed the proc by using "exec" and the "find" command under linux. It isn't cross-platform, so I will leave it only here for reference.

ad_proc cr_check_orphaned_files {-delete:boolean} {

Check for orphaned files in the content respository directory, and
delete such files if required. Orphaned files might be created, when
files add added to the content repository, but the transaction is being aborted.

@param -delete delete the orphaned files

} {
set cr_root [nsv_get CR_LOCATIONS CR_FILES]
set root_length [string length $cr_root]

# check for missing trailing slash on directory
if {[string index $cr_root end] != "/"} {
append cr_root /
}

set result ""
# get any not-hidden file into cr directory...
foreach f [exec find $cr_root -type f \( ! -iname ".*" \)] {
set name [string range $f $root_length end]
# ...skip names which are not-numerical...
if {![regexp {^[0-9/]+$} $name]} continue
# ...check if we have a revision under this filename...
if {[db_0or1row _ {
select 1 from cr_revisions
where content = :name limit 1}]} continue

# ...otherwise it is an orphan.
lappend result $f

if {$delete_p} {
file delete $f
}
}

return $result
}

All the best

Tried with last version. I saw "find" needs folders to end with "/" to search into them. I added this little check to the proc and committed. Proc seems to work properly.

Bye!

Therefore, i've implemented a much more scalable approach based on a file-creation log that keeps track of freshly created files (per default the files created on one day). These files can be efficiently checked together with "cr_delete_scheduled_files" without putting much stress on either the file-system or the database.

see: http://cvs.openacs.org/changelog/OpenACS?cs=oacs-5-8%3Agustafn%3A20131231162110

The function "cr_check_orphaned_files" can still be used for cleaning up orphaned files from times before the cr-file creation log was in place (that is from the birth of OpenACS until now).

Please test on your systems as well. Note that the change requires an update of acs-tcl and a restart of nsd.

all the best for 2014!
-gustaf neumann

I will fiddle a bit with file insertion/deletion and let you know!

Thanks a lot for this fine solution to the issue!