Forum OpenACS Development: File-storage upload slow in OpenACS 4.6.3

The slowness occurs in the call to

<fullquery name="new_lob_file">
      <querytext>
      select file_storage__new_file (
                :title,                -- title
                :folder_id,            -- parent_id
                :user_id,              -- creation_user
                :creation_ip,          -- creation_ip
                true                    -- indb_p
                );
      </querytext>
</fullquery>

in file-add-2.tcl. I've tried calling this proc manually in psql, and it virtually never ends.

This only happens once the DB has some data in it. It used to never happen.

Before I start digging into this, I'd like to know if anyone is aware of this problem, and have already dug into or even fixed this in a later version.

Thanks,
/Lars

DaveB, are you load-testing file-storage now that you're working on it?

Maybe you could get Don and/or Jeff to help you with that.

/Lars

I know that Jeff Davis put in a few fixes to speed up object creation. I wonder if this would affect file-storage uploads. Those changes are in OpenACS 5.0. I will try to do a comparison to see if it has any effect.

Here is something from the list. Can anyone test or confirm this?

"Earlier this year (sometime in February) I described a problem I had with file uploads taking excessively long time within an intranet environment, with effective speeds 750-800Kbit/sec on a 100MBit/s ethernet.

This week I spent some time investigating the problem further. My initial suspicion was that IOBuffers are not really allocated and we end- up reading only a few characters at a time. After checking the socket read/write code in AOLServer (using printf statements) my initial hypothesis was overulled (buffer allocation works OK!).

To cut a long story short, I found out that the culprit was the Silly Window Syndrome (SWS; see Stevens, Vol.1) exhibited by the TCP stack in WinNT4 clients when sending bulk data to the web server (the IE and Netscape browsers may also be at fault here).

The work-around was to change the tcp_deferred_ack_interval parameter in Solaris 8; this parameter indicates the delay in milliseconds before sending a deferred ACK (the default for windows is 200ms, for Solaris 8 is 100 ms and for most linux distributions 50ms; I set it to 10ms and the file uploads are 10 times faster!).

I believe the new low value of 10ms is OK for LANs (where RTT for TCP packets are less than 10ms) but it may not work for WANs.

I hope that anybody who would like to speed-up file uploads to AOLServer may find this information useful. "

It does a create_item first off ... first guess is the tree sortkey stuff as has been discussed above.  When Jeff has time to munge 5.1 perhaps he can introduce his changes to file storage first, and we can see if this fixes the problem?

Of course the 5.0 basic object creation tree sortkey speed-up would also need to be applied to Collaboraid's system, just speeding up the creation of one of the two tree sortkeys involved won't solve the problem.

The other troublesome call in file_storage__new_file is the call to update the last modified attribute of the object and all its context_id parents.  For one thing this doesn't seem right, OF sort of snuck it in while working on .LRN.  The last modified attribute in the original aD object model was intended to track the last direct modification of an object, not those who inherit permissions from the object.  For a second thing, the recursive nature of the implementation might be slow ...

But I think the first place to look is the sortkey stuff.  When Jeff gets the mods made to the CR (which should help the performance of several packages) we can do some performance testing to see if the update last modified procedure is also a problem.