Forum OpenACS Q&A: lynx, cookies and ACS sessions

Collapse
Posted by Connie Hentosh on
The fact that I can't find a thread on this topic in this forum or arsdigita leads me to believe that I set up something wrong.

I setup OpenACS and tried to access using lynx. Everytime I would log in it would give me the login screen suggesting I may have cookies turned off. I have used lynx before on cookie sites and it seems to work fine. So I tried with netscape and it works. I download the python program httpMonitor to see what was the difference... and it seems that lynx doesn't like cookies with commas in them. It truncates the value at the comma.

The spec at http://home.netscape.com/newsref/std/cookie_spec.html also states that the cookie cannot contain semi-colon,comma or white space.

Here is a sample of the output of http sniffer:


RESPONSE
HTTP/1.0 http://192.168.32.21:8000/pvt/home.tcl 302 Found
Set-Cookie:
ad_session_id=132,0,ukaj9bTODpXp0LGxZViYMR1QPQoT5NAD,988017574;
Path=/; Max-Age=86400
Location:
http://192.168.32.21:8000/register/index.tcl?return_url=%2fpvt%2fhome%2etcl
Content-Type: text/html; charset=iso-8859-1
MIME-Version: 1.0
Date: Mon, 23 Apr 2001 09:19:34 GMT
Server: AOLserver/3.2+ad12
Content-Length: 357
Connection: close

REQUEST
GET
http://192.168.32.21:8000/register/index.tcl?return_url=%2fpvt%2fhome%2etcl
Host: 192.168.32.21:8000
Accept: text/html, text/plain, application/vnd.rn-rn_music_package,
application/x-freeamp-theme, audio/mp3, audio/mpeg, audio/mpegurl,
audio/scpls, audio/x-mp3, audio/x-mpeg, audio/x-mpegurl,
audio/x-scpls, audio/mod, image/*, video/*, video/mpeg
Accept: application/pgp, application/pgp, application/pdf,
message/partial, message/external-body, application/postscript, x-be2,
application/andrew-inset, text/richtext, text/enriched,
x-sun-attachment, audio-file, postscript-file, default, mail-file
Accept: sun-deskset-message, application/x-metamail-patch, text/sgml,
video/mpeg, image/jpeg, image/tiff, image/x-rgb, image/png,
image/x-xbitmap, image/x-xbm, image/gif, application/postscript,
*/*;q=0.01
Accept-Encoding: gzip, compress
Accept-Language: en
User-Agent: Lynx/2.8.4dev.7 libwww-FM/2.14
Referer: http://192.168.32.21:8000/
Cookie2: $Version="1"
Cookie: ad_browser_id=129; ad_session_id=130; ad_user_login=3


So did I miss something?

Should the values for the cookies be encoded?

Has anyone else had issues with lynx and ACS?

Collapse
Posted by Don Baccus on
Wonderful.  The cookie code comes straight from ACS Classic, to be honest I've never paid attention to it except for fixing the problem with "second to last visit" a few months back, a problem that plagued every ACS Classic as well as OpenACS site on the planet due to a most silly programming error on the part of some aD hacker.

I just read the spec you pointed to, and indeed it appears that the cookie should be encoded in order to exclude commas.

On the other hand, only lynx doesn't like it apparently.  Or, at least  Konqueror, Mozilla, Explorer and Netscape in their bazillion release variants seem to work OK.

The spec you reference looks like the original Netscape cookie spec, so one immediate question comes to mind - is that the spec that's the official spec today, or was it superceded later on?  Is it possible the restrictions on commas was removed later?

Collapse
Posted by Connie Hentosh on
Sorry... did another pre tag instead of a /pre tag closed it..
Collapse
Posted by Ola Hansson on
I have experienced problems with Opera 5.0b6 on Linux when separating "Cookie items" by white space but it was functioning normally on IE and NN.
All three browsers were able to handle a colon as delimiter.
Collapse
Posted by Ola Hansson on
Trying to close pre tag... Commas didn't work either (on Opera).
Collapse
Posted by Jonathan Marsden on
I see this behaviour too, using Lynx to browse openacs.org and www.arsdigita.com. So as far as I can see, it is not you doing something wrong. You appear to me to have found a genuine bug.

Encoding the cookie values seems a good solution to me, though you'd have to allow for "converting" old non-encoded cookies too, unless you wish to break all current stored cookies!

Cookies are set in only 17 or so .tcl files within OpenACS 3.2.4. Hand hacking these places to use ns_urlencode would be pretty straightforward from a quick scan of the relevant code (do something like

   grep -ri Set-Cookie *

at the top of your ACS tree to find them).

Finding the places where cookies are searched for in the headers (to attempt decoding in each place) is marginally more interesting, but I suspect that something like


  egrep -ri "ns_set get .*Cookie" *

will find them, and again there are not too many places to hack in a call to ns_urldecode.

Collapse
Posted by Connie Hentosh on
Doing a search on developer.netscape.com Give me this page: http://developer.netscape.com:80/docs/manuals/js/client/jsref/document.htm Which states the same thing. Doing a search on Google gives me this: http://portal.research.bell-labs.com/~dmk/cookie.html They mention RFC 2965. I am going through it to see what I can find out.. looks like they suggest it be URL encoded.
Collapse
Posted by Connie Hentosh on
Looking at RFC 2965... it seems that the comma's have to go... at least to be compatible with the future spec. Here is the snippet from the spec:

   Note: For backward compatibility, the separator in the Cookie header
   is semi-colon (;) everywhere.  A server SHOULD also accept comma (,)
   as the separator between cookie-values for future compatibility.

(closed it that time :)
Collapse
Posted by Don Baccus on
Oh, sheesh ... well, at least you're doing better closing "pre" tags now than aD did reading cookie specs when the current scheme was adopted!

Thought aD'ers often drop by here, I think you should bring this directly to their attention, either via web/db or by e-mailing someone  like Bryan Quinn.

I realize now why the only opera user of my personal site has had problems with my cookies...

Collapse
Posted by Connie Hentosh on
I took a look at the arsdigita site using httpMonitor.py. It seems that now there is only one cookie and it is URL encoded. It works with Netscape but breaks with lynx for different reasons. For some unknown reason when I use lynx going through httpMonitor.. it hangs on the request to the ars site (Never receiving a response to the HTTP request.) If I don't go through the proxy and then enter the forum and click on "ask a question" After getting the login screen and loggin in I receive the following:

Problem with Your Input

   to ArsDigita
     _________________________________________________________________

   We had a problem processing your entry:
     * You must supply a value for topic

   Please  back  up  using  your  browser,  correct it, and resubmit your
   entry.
     
   Thank you.
     _________________________________________________________________
     
     
    webmaster@arsdigita.com

So that is different... but I still can't see the cookie since httpMonitor does work.

Anyone have a recommendation on a debug proxy that does a good job of printing out cookies and headers and the such? I am not convinced that httpMonitor is transparent to the session since there were some issues of Netscape not redirecting correctly.

Collapse
Posted by Connie Hentosh on
I took a look at the arsdigita site using httpMonitor.py. It seems that now there is only one cookie and it is URL encoded. It works with Netscape but breaks with lynx for different reasons. For some unknown reason when I use lynx going through httpMonitor.. it hangs on the request to the ars site (Never receiving a response to the HTTP request.) If I don't go through the proxy and then enter the forum and click on "ask a question" After getting the login screen and loggin in I receive the following:

Problem with Your Input

   to ArsDigita
     _________________________________________________________________

   We had a problem processing your entry:
     * You must supply a value for topic

   Please  back  up  using  your  browser,  correct it, and resubmit your
   entry.
     
   Thank you.
     _________________________________________________________________
     
     
    webmaster@arsdigita.com

So that is different... but I still can't see the cookie since httpMonitor does work.

Anyone have a recommendation on a debug proxy that does a good job of printing out cookies and headers and the such? I am not convinced that httpMonitor is transparent to the session since there were some issues of Netscape not redirecting correctly.

Collapse
Posted by Connie Hentosh on
Well that is just jim crackin dandy. Did I do that double post with my back button?
Collapse
Posted by Connie Hentosh on
I tried this before... decided to give it one more try.. I added double quotes around the value of the cookie by modifying ad-utilities.tcl.preload in proc ad_set_cookie.

Changing:

	set cookie "$name=$value"
to:
	set cookie "$name="$value""
This failed in my previous test ... but looking at the proxy output I see that lynx now gets the full session id with commas! Now it seems that ad_get_cookie needs to be modified to accept values possibly surrounded by quotes.

Can anyone suggest the proper patch to the ad_get_cookie so it will also be backward compatible?

Also, is there anything that works with TCL that makes it easy to jump to the definition of a proc? Does tags work with TCL?

Thanks.

Collapse
Posted by Michael A. Cleverly on
"Also, is there anything that works with TCL that makes it easy to jump to the definition of a proc? Does tags work with TCL?"
I'm not sure what you mean, exactly. It's easy to find out the definition of a Tcl procedure, what arguments it takes, etc. (Tcl's introspection facilities are great!). info body proc_name will give you the source code for the proc, and info args proc_name will tell you what arguments it takes.

(Or, did you mean something else, like the API browser?)

Collapse
15: emacs, tags and tcl (response to 1)
Posted by Jerry Asher on
emacs, the tags system, and etags can support tcl, pl/sql and acs development easily. Here's http://www.arsdigita.com/bboard/q-and-a-fetch-msg?msg_id=000Jr9 a discussion of tags, tcl, and the acs. I imagine it shouldn't be too difficult to add support for pgsql.
Collapse
Posted by Dan Wickstrom on
I've been using this for pgsql:

find . -type f -name "*.sql" -exec etags --regex="/create[ 	]+table[ 	]+([^ 	]+)//" --regex="/create[ 	]+function[ 	]+([^ 	]+)//"
You could also extend this to find indices, views and just about anything else you would like to find quickly.
Collapse
Posted by Janine Ohmer on
Just to muddy the waters further :), I use Lynx to download from
arsdigita.com, which requires a login before download, and it
always works for me.  But with openacs.org, when I log in I get
thrown back to the login screen again, as Connie reported.  So
why do I have success when Jonathan said it also fails for him
on arsdigita.com???
Collapse
Posted by Richard Li on
The RFC section that is quoted above actually refers to how servers accept cookies, as opposed to how user-agents process cookies. I actually don't see any place in RFC 2695 which specifies (or prohibits) a specific delimiter inside a cookie (I could be missing it, though; any pointers would be helpful). Regardless, you're right that these things should be url_encoded at the very least, and that's exactly what ACS 4.0.1 and later does:
<~> telnet developer.arsdigita.com 80
Trying 216.34.106.248...
Connected to developer.arsdigita.com (216.34.106.248).
Escape character is '^]'.
HEAD / HTTP/1.0

HTTP/1.0 200 OK
Set-Cookie:
ad_session_id=28646925%2c0%20%7b145%20988119842%205F3C876E8F07ECB37869B5B9C5EECB2DB17AE730%7d;
Path=/; Max-Age=1200
Content-Type: text/html; charset=iso-8859-1
MIME-Version: 1.0
Date: Tue, 24 Apr 2001 13:24:03 GMT
Server: AOLserver/3.2+ad10
Content-Length: 6967
Connection: close
I'm not familiar with the OpenACS 3.2.x code, but in ACS 4.x we introduced a set cookie abstraction procedure (ad_set_cookie). If this exists inside OpenACS 3.2.x, you could fix this proc instead of running through the entire ACS as Jonathan suggests.

One more note is that getting cookies to work right is atually a pain in the butt because most client browsers don't follow 2965 to the letter, and you'll notice that ad_set_cookie redundantly sets headers to cover all the known cases. It's taken us a couple of tries to get it right (and it looks like OpenACS hasn't gotten all of the fixes yet). In fact, even the ACS 4.2 implementation isn't quite right because in rare cases it might fail in WAP gateways, so we're using ! as a delimeter now in ACS Java 4.0.2 and later. I actually disagree with the comment that it's just a "silly programming error on the part of some aD hacker", because I actually think that programming cookies is a difficult thing and requires anal-retentive testing :).

Collapse
19: RFC for cookies (response to 1)
Posted by Connie Hentosh on
       I actually don't see any place in RFC 2695 which specifies (or
       prohibits) a specific delimiter inside a cookie (I could be
       missing it, though; any pointers would be helpful).
RFC 2965 points to the RFC 2616. 2616 states (around page 16) what is valid in a token and what is a delimeter. If there is a delimeter in the value, it needs to be placed in quotes. I noticed that ACS 4 doesn't have quotes around the value. I wonder if that would solve your WAP issue.
       I'm not familiar with the OpenACS 3.2.x code, but in ACS 4.x 
       we introduced a set cookie abstraction procedure
       (ad_set_cookie).
ad_set_cookie is in 3.2, but not used in ever place. Some code set the cookie headers directly.
Collapse
Posted by Connie Hentosh on
Curious though, were the WAP gateways failing on a delimeter that was encoded?
Collapse
21: lynx cookie patch (response to 1)
Posted by Connie Hentosh on
Making the following changes in ad-utilities.tcl.preload permits lynx users to login. Modify ad_set_cookie to add quotes around the value:
    set cookie "$name="$value""
And then modify ad_get_cookie to look like this:
    if { $include_set_cookies == "t" } {
        set headers [ns_conn outputheaders]
        for { set i 0 } { $i < [ns_set size $headers] } { incr i } {
            if { ![string compare [string tolower [ns_set key $headers $i]] "set-cookie"] && 
                    [regexp "^$name="([^"]+)"" [ns_set value $headers $i] "" "value"] } {
                return $value
            }
        }
    }

    set headers [ns_conn headers]
    set cookie [ns_set iget $headers Cookie]
    if { [regexp "$name="([^"]+)"" $cookie match value] } {
        return $value
    }
    if { [regexp "$name=([^;]+)" $cookie match value] } {
        return $value
    }
If you dont need to be backward compatible with older cookies set previous to the patch... then you can omit the last if statement.
Collapse
Posted by Connie Hentosh on
Whoops.. I just noticed that my backslashes got stripped. :(
Collapse
Posted by Jonathan Marsden on
Connie, could you try reposting the code with doubled backslashes
please, or (perhaps better) could you just submit it as a patch to the
SDM?

I could use this... I *think* I can tell where they got missed
out, but it would be nice to see them, rather than having to guess,
and put them back in until it works 😊

Assuming this works (and I can't see why it shouldn't), we should
probably get this into OpenACS 3.2.6.

Meanwhile (Ben/Roberto/Don), if someone could apply this change to
OpenACS.Org itself, that would be helpful.

Collapse
Posted by Connie Hentosh on
Okay. I added the patch to the SDM for the lynx cookie issue. It is #11 against 3.2.5.

So should one also submit a bug report... or just the patch witht the SDM?

Collapse
Posted by Jonathan Marsden on
Great.

Bug report and patch is ideal.  There are patches to the SDM so you
can tell it that patch#N fixes bug#M etc, but they are not live on
OpenACS.Org (Ben -- are they working well enough to use here?).

So for now, just submit a bug report against the SDM module, and in
the description say "fixed by Path #whatever".  I think that's the
best we can do with the SDM as it now stands.

Collapse
Posted by Jon Griffin on
The problem is OpenACS still hasn't applied the patch!

So now I have to use my windows box, download what I need, then scp it to my linux box.

Please fix this on the OpenACS site!