Forum OpenACS Q&A: Problems downloading large files...

Collapse
Posted by Robert Locke on

We have a client which is using ETP to store some fairly large (3M+) binary files.

They have been receiving complaints from some site visitors saying that they cannot completely download these files. There are no browser errors, the saved files are just incomplete (they discover this the hard way because these files are memory images for special printers and a bad download does not load properly).

I looked through the server logs and discovered the following (the error is "i/o failed"):

[06/Nov/2002:09:44:53][14453.5913606][-conn2534-] Notice: RP (30837.797 ms): error in rp_handler: serving GET /home/downloads/reseller_downlo\ads/mainA100802_V20Pre25
	ad_url "/home/downloads/reseller_downloads/mainA100802_V20Pre25" maps to file "/servers/vipcolor/packages/edit-this-page/www/template\s/vipcolor/download/content.adp"
errmsg is i/o failed
[06/Nov/2002:09:44:53][14453.5913606][-conn2534-] Error: GET /home/downloads/reseller_downloads/mainA100802_V20Pre25  i/o failed
    while executing
"ns_writefp $ofp"
    ("write_blob" arm line 33)
    invoked from within
"switch $type {

            blob_get {

		if {[info exists storage_type]} {
                    switch $storage_type {
                 ..."
    invoked from within
"db_exec_lob write_blob $db $full_statement_name $sql"

Has anyone seen this before? Any ideas how to resolve this?

Thanks...

Collapse
Posted by Robert Locke on
Here are some entries from the access logs which indicate the failures referenced above. Only the last access listed below was successful (ie, the file was completely downloaded). There are many other failed downloads in the logs. Interestingly, many seem to stop at 160000 bytes:
access.log.002:65.218.30.138 - reseller [05/Nov/2002:09:41:41 -0800] "GET /home/downloads/reseller_downloads/mainA100802_V20Pre25 HTTP/1.1" 200 160000 "http://www.vipcolor.com/home/downloads/reseller_downloads/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
access.log.002:151.203.38.99 - reseller [05/Nov/2002:10:32:20 -0800] "GET /home/downloads/reseller_downloads/mainA100802_V20Pre25 HTTP/1.1" 200 64584 "http://www.vipcolor.com/home/downloads/reseller_downloads/" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0)"
access.log.002:216.154.131.113 - reseller [05/Nov/2002:11:48:58 -0800] "GET /home/downloads/reseller_downloads/mainA100802_V20Pre25 HTTP/1.1" 200 3818901 "http://www.vipcolor.com/home/downloads/reseller_downloads/" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)"

Any ideas?

Collapse
Posted by C. R. Oldham on
Is this AOLserver 3.3.1+ad13?
Collapse
Posted by Robert Locke on
Hi there...

Yes... AOLserver/3.3.1+ad13 (aolserver3_3_1_ad13), mostly stock OACS 4.5 install.

Do you think changing "cr_write_content" to use "ns_returnfile" as suggested by Jeff Davis in this thread would make a difference?

Thanks...

Collapse
Posted by Don Baccus on
Switching to ns_returnfile should be *within* cr_write_content, you shouldn't remove the abstract call because it works with blobs as well as files.

And ... since apparently you're storing your large data as blobs rather than in the file system cr_write_content will need to use write_blob anyway.

Is there any reason not to believe the error message, i.e. that i/o failed?  As in a socket hiccup or something similar?

Collapse
Posted by Robert Locke on
Hi Don...

> Switching to ns_returnfile should be *within*
> cr_write_content, you shouldn't remove the abstract call
> because it works with blobs as well as files.

Absolutely, that's what I meant by 'changing "cr_write_content" to *use* "ns_returnfile"'.

I actually am storing the data in the file system, NOT the database. The original "cr_write_content" calls "db_write_blob" even when the storage type is "file". "db_write_blob" then calls "db_exec_lob write_lob", which then calls "ns_writefp $ofp" when storage_type = "file", which is in my original error message posted above. So I am using the file system.

I guess I would believe the i/o error message except that it seems to happen quite frequently. Also, someone else posted a similar problem here, so maybe there's something more to this.

In any case, I updated "cr_write_content" to use "ns_returnfile" as per Jeff's suggestion. Ultimately, it boils down to using ns_returnfile instead of ns_writefp to deliver the content to the end user. I haven't dug deeper into the internals of those calls, but do you think it would make any difference?

Thanks...

Collapse
Posted by Tom Jackson on

I wrote a proxy module for AOLserver that used ns_writefp, and I would get these io errors. But what it boiled down to was the user would hang up on the connection, usually an image file would get cut short in the process. I'm guessing that ns_writefp doesn't check for the other end closing the connection. Does ns_returnfile do this type of check?

So the question is, are users giving up on a download? How are they getting a partial file without the browser reporting the problem?

Collapse
Posted by Robert Locke on
Hi Tom...

I asked those very questions and was only told that the browsers never complained.  They only discovered the downloads were incomplete after attempting to use the file.

I will ask again though.  And I will let you know if using ns_returnfile improves the situation.

Thanks...