Forum OpenACS Q&A: Stupid Aolserver trick with ns_httpget and lynx cookies

Well, I finally did it. I have two aolserver sites running - and needed to sync up some data between them. I considered using db handles to insert data directly to the remote machine, but since there was already a web page to do it manually, I took the coward's route and started playing with ns_httpget. After a few minutes I realized that the standard "ACS requires cookies" gotcha was in play, so I hunted for a ns_httpget that handled extended headers, e.g. cookies.

The cookie spec told me that I needed to send a "Cookie:" header with all the cookies expected for the session.

The hacked-up version of http.tcl (cf ~aolserver/modules/tcl/http.tcl) from Holly Jerry was able to handle cookies in an rqset (ns_set). I had cookies from a lynx session. So, I copied my .lynx_cookies file to the ~aolserver directory, made aolserver the owner of the file, and added the following lines:

proc ns_Chttpget {url {timeout 30} {depth 0} {rqset ""}} {
    if {[incr depth] > 10} {
	return -code error "ns_Chttpget: Recursive redirection: $url"
    # Perform the actual request.

ns_log Notice "ns_Chttpget: trying $url"
regexp {([^:]+)://([^:/]+)(:([0-9]+))?(/.*)} $url match protocol 
server x port path
ns_log Notice "ns_Chttpget: $protocol $server $x $port $path"
set LynxFile [open /home/aolserv/.lynx_cookies r+ ]
set cookie {}
while {[gets $LynxFile line] >= 0} {
regexp {([^	]+)[	]+([^	]+)[	]+([^	]+)[	]+([^	]+)[	]+([^	]+)
[	]+([^	]+)[	]+([^	]+)} $line lynxline lx_domain lx_false lx_path 
lx_false2 lx_date lx_name lx_value

if {[string compare $server $lx_domain] == 0} {
ns_log Notice "Setting cookie: $lx_name=$lx_value"
append cookie "$lx_name=$lx_value; "
close $LynxFile
ns_log Notice "cookie: $cookie for X${server}X AKA X${lx_domain}X"
	if {$rqset != ""} {
		ns_set cput $rqset "Cookie" $cookie
		set http [ns_Chttpopen GET $url $rqset $timeout]
	} else {
		set rqset2 [ns_set create]
		ns_set cput $rqset2 "Cookie" $cookie
		set http [ns_Chttpopen GET $url $rqset2 $timeout]

    set rfd [lindex $http 0]
    close [lindex $http 1]
    set headers [lindex $http 2]

basically everything syncs up to the original hollyjerry version starting at "set rfd ..."

Also, first, in vi editor I did a :1,$s/ns_http/ns_Chttp/g just to avoid interfering with the original ns_http family of procs. Yes, the "C" stands for cookie. So shoot me ...

It ain't pretty, but it's working for me :)

Ideally, this would get extended out to actually WRITE the cookies that get sent back, but I don't need that functionality myself. Maybe someday Real Soon Now ... With all the fragmentation in the community, and also given that IMHO we should be using the http2.3 module rather than ns_http - it's hard to figure out who or where to contribute to!!! I can't post to the aolserver list - the majordomo won't accept me...

Final note after previewing - all the regexp statements are wrapped - they should only be one line long each!!

I had the same problem for a long time with the AOLserver list majordomo. You should email Kriston Rehberg ( - he's the list administrator.
It's a good list with some excellent people active on it - I've learned plenty from lurking there.
Ironically, that's my homepage on explorer :-) Thanks for the tip, maybe I can actually post to it after I contact him!
Is there an approach that doesn't require altering any files?  Requiring a hacked http.tcl would render my file useless:

I'm working on a smoke-test page to be included with a module.  It tries to reach and and access each page, posting forms and whatnot, using util_httpget and util_httppost.  It then parses the results.  It dies as soon as it hits a page that requires cookies.  Is there a way to get the necessary information from the parent page (smoke-test.tcl, the page which hosts the util_httpget commands) and then includes that in the httpget?  It looks like I just need to give util_httpget a header argument like "Cookie: ad_session_id = foo".  Is this right?  If so, can I get or derive foo?


to get the Cookie header you can search the output of ns_conn headers for "Set-Cookie" headers. Depending on your ACS version there may be more then one of them - my OpenACS 3.2.5 installation returns three at the first request: ad_browser_id, ad_session_id and last_visit; a current OpenACS 4 installation seems to only return one: ad_session_id (which is good, especially if the browser is set up to ask for confirmation for each cookie).

If you want to do more fancy things with forms and session stuff you might want to check out tclwebtest. It is designed to run as stand-alone application from the commandline, but with a little tweaking it might be possible to also run it from within aolserver. It's propably overkill for your task, but I couldn't resist adding a little commercial for it here ;-)
Thanks for the tip. For the record, here's a method that works:
ns_set cput $header_set_id Cookie [ns_set iget [ns_conn headers] cookie]

set return_page [util_httpget $target_url $header_set_id ]