Forum OpenACS Q&A: Upload large files on Aolserver - Is moving to Naviserver the only way?

Hello everyone,

I came across a problem almost identical to the one debated here https://groups.google.com/forum/#!topic/aolserver/hhZYKNx-jDE.

In short: I need to allow the upload of big files on my server (up to 1 GB). The upload script is working fine, but whenever it gets executed, nsd process suffers memory growth, which will eventually lead to the process being killed by the OS.

I read in the discussion I linked that Naviserver addressed this problem properly, then every speculation about Aolserver stopped, so I took my development server and tried to install naviserver in place of aolserver using the new script from here https://openacs.org/xowiki/naviserver-openacs.

Production and development servers run Debian Squeeze and OpenAcs5.5, so I was expecting some kind of failure, also because I didn't remove any of the previous software to put me in the most recoverable situation once in production... but it worked! The only issue was the missing bzip2 package, then all went fine. I edited configurations from the oacs-HEAD installation to match the ones of my instance and OpenAcs5.5/postgres8.4 was up and running on a fresh Naviserver.

This is what I ask: am I being too confident about the results of my experiments? Should I expect strange behaviors further on? But more important: is this the only pratical way to allow large upload for an OpenACS website?

In the end I found something not working: is with delivery threads. Seems some internals regarding mutexes have changed and file storage download features won't work anymore if you use OpenAcs 5.5 on Naviserver...
My recommendation is to move ahead and upgrade from OpenACS 5.5 to OpenACS 5.8.0. OpenACS 5.5.1 is more than 4 years old. Sooner or later, you might want to upgrade, and upgrading the kernel is usually not a big thing. We recently upgraded OpenACS 5.5 to OpenACS HEAD and then to the oacs-5-8 branch, and the upgrade of the core was very smooth (see https://openacs.org/forums/message-view?message_id=4071801 for the exactly used components). You do not have to upgrade the database (although i would recommend this anyhow).

If for some reason, you can't upgrade to 5.8.0 now, you can check for the differences (diff -r) between your version and the components of the oacs-5-8 branch, and grep for NaviServer. There are probably just a handful of changes.

Concerning bgdelivery: you can probably just upgrade xotcl-core to get this fixed. My first guess would be:
http://cvs.openacs.org/browse/OpenACS/openacs-4/packages/xotcl-core/tcl/bgdelivery-procs.tcl?r1=1.24&r2=1.25&u=3&k=kv
but advancing to newer versions buys you a lot of other advantages (e.g. use of the NaviServer build-in writer threads, which are much better than classical bgdelivery),

Hope, this helps,
-gustaf neumann

Thanks for the help Gustaf, yeah, I should probably just upgrade.

I was giving a last try by switching the memory allocator to one which allows to reclaim the memory once used, as I could read around... just didn't want to migrate/reload data, but they're not a lot and won't be a big deal.

In order to use a different memory allocator, one has to alter the C-implementation in Tcl... and it might turn out that it does not change much.

The Tcl implementation favors speed over space consumption and over-allocates memory eagerly (essentially doubling the allocated size of a string allocation when it needs more space). With bad luck, if you have a string of 1MB and you append one byte, then it allocates 2MB. This works in practice much better as it sounds. However, if this string is truncated later to e.g. 10bytes, these 10 bytes are still sitting in the 2MB allocation.... in such cases, a different malloc won't help at all. The bad things of the zippy malloc are (1) that it never releases memory and (2) that it leads to a large vsize (where people often argue that this does not matter), while the good thing is that it uses very little mutex locks. Check [1] for some results of using tcl with different memory allocators.

If you are concerned about memory consumption, NaviServer is as well on longer running server the better choice, since it has less memory leaks and it is less exposed to stress reactions causing eager thread creations. This can be seen partly by the following chart showing the memory size of the aolserver/NaviServer running openacs.org:

From Aug to June one sees the memory consumption of the aolserver with reboots in Nov and Dec. In July we first upgraded pg 8.3 to 8.4, which brought the machine nearly to a halt due to the slow permission queries, leading to the mentioned stress reactions. After upgrading to pg 9 + NaviServer, one sees essentially a flat line of memory consumption (with the zippy allocator) using a total of currently 27 threads (including writer threads etc.) The aolserver variant used typically 17 threads, but used more memory. One can certainly further reduce the long range memory consumption by using other mallocs, but this is currently not a concern.

[1] https://next-scripting.org/xowiki/docs/misc/thread-mallocs/index1

My conditions are quite narrow being on a cost-aware hosting solution. I'm very curious about the level of service I could achieve!
The biggest bloat is OpenACS, loading many deprecated/unneeded/redundant procs into blueprints, keeping large nsvs, etc. OpenACS was just gowing over the years without much cleanup. Therefore, we are currently working to reduce the usage of deprecated functions such we can move this in the future into a package containing just the deprecated stuff for the people, still needing it.

Try to load navi-server with a plain sample configuration, you'll see how fast it starts and how much/little memory it consumes (this certainly depends on the configuration parameters) to get a boundary, what you might reach.

Btw. i've added bzip2 to the list of packages in install-ns.sh

Upgraded! It has been quite straightforward thanks to the new scripts.

I report my experience:

OpenAcs installer, during database configuration, probably makes use of unavailable syntax for postgresql 8.x, because failed the first time. Could be a configuration-specific problem tough, as I had to mess around with the conf in the past. Anyway, as I was already towards the idea of upgrading to postgres 9.x and to a supported distro, I switched my Debian repos from squeeze to wheezy and upgraded the distribution so I could installe postgres 9.1 .

After that, I followed the exact steps found in https://openacs.org/xowiki/naviserver-openacs to obtain an oacs-HEAD tree in /var/www

Note that Debian doesn't install sudo by default, so install-oacs.sh should probably have it into requirements to run smooth on Debian.

I then made some tweakings to conform the installation to the 'Debian way' and feel more comfortable:
- moved root of the server to /var/lib/naviserver
- linked config.tcl into /etc/naviserver/conf.d/
- created a sh script to store variables as user, group and port for the instance as was for the repo installation
- modified configurations so they matched instance name, db etc...
- copied /etc/init.d/aolserver4 to /etc/init.d/naviserver startup script and modified so it starts and stops naviserver properly, one instance for each configuration found in /etc/naviserver/conf.d/
- updated the code of the instance to oacs-5-8 branch by cvs
- created a db for the instance
- pointed browser to the instance and installed as usual

Seems a lot of fiddling, but in fact was more a matter of waiting for downloads or compilation...

Now for the interesting part:

I can testify that memory consumption DROPPED. After two simultaneous uploads of 400MB each, the system occupies less than 100MB in resident memory, as stated by top, and inflates and shrinks under need. Before it was constantly increasing every time we tried to upload big files.

What can I say? Very very well! The fruit of all this years of work from you is very at hand now and doesn't require super-human skills to be enjoyied.

Upgraded! It has been quite straightforward thanks to the new scripts.
Great!

OpenAcs installer, during database configuration, probably makes use of unavailable syntax for postgresql 8.x, because failed the first time. Could be a configuration-specific problem tough, as I had to mess around with the conf in the past. Anyway, as I was already towards the idea of upgrading to postgres 9.x and to a supported distro, I switched my Debian repos from squeeze to wheezy and upgraded the distribution so I could installe postgres 9.1 .

actually, the new syntax of the postgres files works fine on pg 8.0, but recursive queries need 8.4. So, the first step should be to upgrade older instances (see https://openacs.org/xowiki/Postgres_9). New postgres versions might complain about upgrades containing the old syntax, therefore, one should upgrade with the old pg-version to be on the safe side.

To summarize, two kind of errors might show up:

  1. old pg version does not accept new SQL (argument passing need 8.0, recursive queries need 8.4)
  2. new pg version does not accept old SQL (backslash escaping in plain strings)

Note that Debian doesn't install sudo by default, so install-oacs.sh should probably have it into requirements to run smooth on Debian.
sudo is not really needed for the installation, it's convenient later... anyhow, added theses dependencies for Debian and redhat to the script

I then made some tweakings to conform the installation to the 'Debian way' and feel more comfortable:

- moved root of the server to /var/lib/naviserver
- linked config.tcl into /etc/naviserver/conf.d/

keeping the server out of the "standard debian paths" has as well advantages. sooner or later we should have a debian naviserver package, that might conflict with your adapted installation.

- created a sh script to store variables as user, group and port for the instance as was for the repo installation
what is this? ownership + permissions?

- modified configurations so they matched instance name, db etc...
- copied /etc/init.d/aolserver4 to /etc/init.d/naviserver startup script and modified so it starts and stops naviserver properly, one instance for each configuration found in /etc/naviserver/conf.d/
Is there a reason for not using/adapting the provided upstart scripts?

- updated the code of the instance to oacs-5-8 branch by cvs
- created a db for the instance
- pointed browser to the instance and installed as usual
here i am as well confused: for a new installations, this should not be necessary. for an upgrade (keeping old data), the dump/restore is missing.

I can testify that memory consumption DROPPED. After two simultaneous uploads of 400MB each, the system occupies less than 100MB in resident memory, as stated by top, and inflates and shrinks under need. Before it was constantly increasing every time we tried to upload big files.

Since you are using though the script a local version of tcl, I'll send you a patch and short instructions how to use different mallocs...

all the best
-g

Debian squeeze version of postgres is 8.4, but I clearly remember having enabled some backward compatibility options for OpenAcs5.5, so it must really be something on my end.

Standard server root for packaged openacs in debian-like systems is /usr/share/openacs, which felt weird to me... I haven't really respected the debian convention, just put myself in a more familiar situation.

Another thing Debian did in the package and I replicated was the shell script. user and group are the ones the server is executed by. My replication was quite blind in this regard, just wanted to stick to the old package's style. Of course it is not necessary.

Debian doesn't install upstart by default, but sysvinit. Honestly tough, this is not the reason why I prepared the init.d script, I am just more used to it and wanted to feel at home...

In the end, as the data in previous site was not much, I decided to go for a clean install rather than upgrade. As I wanted to name the instance in a different way than oacs-HEAD, after openacs was installed I created another database for the instance and modified the configs of the new instance accordingly.

Thanks a lot for patch and hints!

Notice that the install-oacs script has a section of settings, where one can define the install-dir, user/group names, db-name etc. So it is probably easier to change these settings than to fiddle around afterwards.

Maybe we should provide command line switches to ease adaptation, but having a common set of parameters helps communication in forums, especially fro beginners.

Redhat went from sysvinit (fedora < 9) over upstart (fedora 9-14) to systemd (fedora > 14; currently 19). optionally spitting out sysvinit scripts should be no rocket science...