Forum OpenACS Development: POST Method can be broken when using host node map

Hi everybody,

There's a bug in request processor that I've lost already three days and couldn't fix it. I've put it in bugtracker here:

http://www.openacs.org/bugtracker/openacs/bug?bug_number=3132

What happens is, if you are using host node map and you try to POST a form, the URL in POST is different from the system, what is causing the vars not to be parsed. I've proposed first this patch:

http://www.openacs.org/bugtracker/openacs/patch?patch_number=861

However, I could see it has a problem: when you are using some kind of proxy and you have a huge form, there's the character limit for URL's, wich causes the connection to be cancelled.

I've tried a lot of different aproaches, but I guess this is beyond my skills. Can somebody help me on that?

Collapse
Posted by Dave Bauer on
I am confused.

If you post to the host-node mapped URL it should "just work". There should not be any redirects on post to a different URL.

Do you have forcehostp parameter set to 1? If you are using host node map, that needs to be turned off.

Collapse
Posted by Eduardo Santos on
Actually it doesn't work.

You see, all the checks and URL's in the host-node mapped subsite are relative to itself. If you have, as an example, a page called /subsite/add-edit, the user will see the add-edit link only, but the real URL is /subsite/add-edit. The form action attribute, in that case, will have the full URL (/subsite/add-edit) and it's going to post to it. When the form is reloaded and rendered, it's going to have the add-edit link only, and the vars that were posted to /subsite/add-edit will be lost.

It also happens in the confirm-template page at acs-templating and many other places, such as register page. The first solution I thought was to parse the vars in the request, as the above patch shows, but it fails if you have a big form. So I ran out of ideas.

Collapse
Posted by Dave Bauer on
Why is the subsite coded into the POST url?

Can you give an example of a page where this happens? If you use relative URLs in the HTML the browser will construct the full URL based on the HTTP location.

Collapse
Posted by Eduardo Santos on
Sure, I can give you an example.

Check out these URL's:

http://mercado.softwarepublico.gov.br/register/

http://teste.softwarepublico.gov.br/mpv/register/

In both of them, the action link for register is /mpv/register. The request processor is going to redirect the request from /mpv/register to /register if you use host-node mapped URL, but the POST will fail because the bug I've mentioned above.

Right now the only way it works is because this patch I've supplied.

Collapse
Posted by Dave Bauer on
Which URL is the susbsite and which is the host node mapped URL?

I am not sure if the request processor itself is the problem or some other code is generating the URLs for the redirect that actually cause the problem. Of course, using the request processor to fix the URL might be a good centralized solution.

Collapse
Posted by Eduardo Santos on
Sorry for the short info. The first one is the mapped URL:

http://mercado.softwarepublico.gov.br/register/

The second one is the absolute URL:

http://teste.softwarepublico.gov.br/mpv/register/

I don't know if the request processor is the problem either. I had the same thoghts as you did in this case. However, the subsite must work in both cases (supplying THE host-node mapped URL and the absolute one), and in order to do that, I can't see any solution that doesn't involve the request processor.

Collapse
Posted by Dave Bauer on
If I visit

http://mercado.softwarepublico.gov.br/register/

The form HTML looks like:
form class="margin-form" style="margin: 0px;" action="/register/" method="post" name="login"

Which looks correct. The post goes to the same URL. There should not be any redirects occuring. That is there should not be an external browser redirects. It should map the URL internally.

If I visit

http://teste.softwarepublico.gov.br/mpv/register/

the form HTML looks like
form class="margin-form" style="margin: 0px;" action="/mpv/register/" method="post" name="login"

I think I'll review your patch again and see if can understand what the request processor does in this case.

I've run into this as well. In a couple of cases where I needed a quick solution, I used

[ad_conn vhost_subsite_url]/my-page

to create the form action URL to avoid the redirect. Have you tried that ad_conn property?

Collapse
Posted by Eduardo Santos on
Hi Michael,

In the form creation there are some things you can do to avoid this, yes. However, I'm looking for a more general solution, that would be the right way to do it. Otherwise, I would have to fix all the forms in the system, and that can be very difficult to trace.

That's way I think request processor should do it.

I have not looked into the problem, but from the discussion, it is not clear to me, what the problem is: is it (a) the different host-name due to the host-node map, or (b) the redirection issue of a POST due to URL length limitations.

For (b) the only clean solution is to store the POST-data temporarily on the server and to refer to it in the redirection-URL, and resolve the reference in the request processor. It should not be hard to implement (famous last words), but there are a few hairy issues, and the likelihood to decrease the number of your friends by breaking their code is greater than 0.

Hi Gustaf,

The problem to me seems more related to (b). Let me try to make myself more clear.

1 - Let's assume you have one subsite in yourdomain.com named subsite. So, the subsite URL will be http://yourdomain.com/subsite.

2 - Let's also assume that you have a host-node mapped do otherdomain.com pointing to your subsite. In such way, the subsite URL would be http://otherdomain.com

3 - We create a form under the URL add-edit. so this form has two URL's: http://yourdomain.com/subsite/add-edit and http://otherdomain.com/add-edit.

In the form code, the action link is allways going to be /subsite/add-edit, no matter how you access the site. This mean, if I type the URL http://otherdomain.com/add-edit and http://yourdomain.com/subsite/add-edit, in both ways the form action is going to be /subsite/add-edit, wich is going to be treated for request processor.

Ok, so we have this situation: the RP is going to execute this filter code:


# -------------------------------------------------------------------------
# Start of patch "hostname-based subsites"
# -------------------------------------------------------------------------
# 1. determine the root of the host and the requested URL
set root [root_of_host [ad_host]]
set url [ad_conn url]
# 2. handle special case: if the root is a prefix of the URL,
# remove this prefix from the URL, and redirect.
if { ![empty_string_p $root] } {
if { [regexp "^${root}(.*)$" $url match url] } {

if { [regexp {^GET [^\?]*\?(.*) HTTP} [ns_conn request] match vars] } {
append url ?$vars
}
if { [security::secure_conn_p] } {
# it's a secure connection.
ad_returnredirect https://[ad_host][ad_port]$url
return "filter_return"
} else {
ad_returnredirect http://[ad_host][ad_port]$url
return "filter_return"
}
}
# Normal case: Prepend the root to the URL.
# 3. set the intended URL
ad_conn -set url ${root}${url}

# 4. set urlv and urlc for consistency
set urlv [lrange [split $root /] 1 end]
ad_conn -set urlc [expr [ad_conn urlc]+[llength $urlv]]
ad_conn -set urlv [concat $urlv [ad_conn urlv]]
}
# -------------------------------------------------------------------------
# End of patch "hostname-based subsites"
# -------------------------------------------------------------------------

Ok, so here the URL is changed acording to the host. If you are using the host-node mapped subsite, it's going to cut the subsite URL (/subsite/add-edit) and deliver to the user the form URL (add-edit). It works fine for redirection, but there's a problem here: no matter how you access it, the POST is allways going to be:


POST /subsite/add-edit HTTP/1.1

When you are using the host-node mapped URL, the POST is going to be redirected to /add-edit, but the vars, that where in the POST body, are going to be lost, as they are associated to the /subsite/add-edit URL.

It seems like somebody have seen it before for the GET method, as we can see in these code lines:


if { [regexp {^GET [^\?]*\?(.*) HTTP} [ns_conn request] match vars] } {
append url ?$vars
}

So I proposed a similar patch, wich should work mostly:


--- openacs-4/packages/acs-tcl/tcl/request-processor-procs.tcl 2008-03-22 15:31:32.000000000 -0300
+++ acs-tcl/tcl/request-processor-procs.tcl 2008-04-16 11:01:58.000000000 -0300
@@ -525,12 +525,21 @@
set url [ad_conn url]
# 2. handle special case: if the root is a prefix of the URL,
# remove this prefix from the URL, and redirect.
- if { $root ne "" } {
+ if { ![empty_string_p $root] } {
if { [regexp "^${root}(.*)$" $url match url] } {

if { [regexp {^GET [^\?]*\?(.*) HTTP} [ns_conn request] match vars] } {
append url ?$vars
- }
+ }
+ # Handle POST bug: When the URL in POST request is different from the absolute one,
+ # the vars aren't sent to the page. So we past them as URL vars
+ if {[regexp {^POST (.*) HTTP} [ns_conn request] match post_url]} {
+ if {$post_url ne $url} {
+ set query [ns_getform]
+ set query "?[export_entire_form_as_url_vars]"
+ append url $query
+ }
+ }
if { [security::secure_conn_p] } {
# it's a secure connection.
ad_returnredirect https://[ad_host][ad_port]$url

However, this patch has a major issue: even that the absolute URL is not delivered to the user, as the form is going to be processed sometimes internally, if you are using some redirecting machine such as a proxy, you have to face Apache maximum character limit for GET method, wich is 4.000 as I could see. In that way, some forms are not going to be sent and the user is going to see the proxy error message.

So, you must be asking: why does it send a GET method with the vars? I'm still trying to figure this out, but I guess it has something to do with the ad_returnredirect in the patch. After all the filters are applied, it tries to redirect to the absolute page, without the subsite URL, and this is where the vars are lost.

I've tried some different approaches: when you detect a difference between the post url and redirect url, one could use this code:


util_httppost url formvars [ timeout ] [ depth ] [ http_referer ]

However, it also seems to fail. So, I ran out of options here. I don't want to lose any friends :) and neither want to break other people's code, and that's why I'm trying to find a better solution with you guys.

I am going to use this a lot more, and in order to do this I have to fix this issue.

So, if somebody can help, I'll be very thankfull.

:)

Collapse
Posted by Dave Bauer on
The rendered HTML does not include the subsite part of the URL when you reference the host node mapped url. I posted the results of viewing the source previously. This is why I am confused. The form target should only refer to the relative url which should cause the browser to calculate the url relative to the host node mapped url.
Ok Dave, you are right. The rendered HTML form does not include the subsite part of the code.

I guess I found out what was the error about. I was using the confirm-template switch, and the file acs-templating/resources/forms/confirm-button.tcl had the following code:


set __return_url__ [ad_conn url]

This would work, but the RP patch has the following lines:


# Normal case: Prepend the root to the URL.
# 3. set the intended URL
ad_conn -set url ${root}${url}

# 4. set urlv and urlc for consistency
set urlv [lrange [split $root /] 1 end]
ad_conn -set urlc [expr [ad_conn urlc]+[llength $urlv]]
ad_conn -set urlv [concat $urlv [ad_conn urlv]]

These lines where causing conflict. Even thought the subsite URL was not included in the form action, when you go to the confirm template page, the above lines where including the subsite URL in form action, and maybe that's why my form was broken.

I've change acs-templating/resources/forms/confirm-button.tcl to:


set __return_url__ [ns_conn url]

I don't know if this is an workaround, because when I watch the log and see the ad_conn url behavior, it changes from not include the subsite URL (/add-edit) to include it (/subsite/add-edit). If this is the right behavior, I guess this fix in confirm-button.tcl must be enough.

Anyway, I'm sorry for all the trouble I've caused you guys and thank you very much for the help.

Collapse
Posted by Dave Bauer on
Ah this leads to the key. Thanks for being persistent.

Anything that calls ad_conn url probably could generate this bug. Maybe ad_conn url needs to be fixed?

Most likely there are other things like ad_return_url which may also need to be fixed.

It would be nice to finally make host-node mapping work consistently.

Hi Dave,

It seems like the ad_conn have to be fixed. Actually, i don't know if this would fix, but the code that changes ad_conn behavior is this:


# Normal case: Prepend the root to the URL.
# 3. set the intended URL
ad_conn -set url ${root}${url}

# 4. set urlv and urlc for consistency
set urlv [lrange [split $root /] 1 end]
ad_conn -set urlc [expr [ad_conn urlc]+[llength $urlv]]
ad_conn -set urlv [concat $urlv [ad_conn urlv]]

I don't know if this is necessary. Maybe it is for the redirect, but I guess there's somebody better than myself to talk about this.

As we are talking about it, there's another thing that needs to be fixed in order do the host-node map to work better. There should be an way that the singleton packages are rendered in the same URL, no matter the domain. As an example, http://mydomain.com/dotlrn and http://otherdomain.com/dotlrn should both serve dotlrn. The same for acs-lang and others.

Maybe a memoized list with the singleton packages URL before the patch would solve it?

Collapse
Posted by Dave Bauer on
For singleton packages, notifications, etc that makes sense. But dotlrn is not a singlton and I don't think there's a way you could make dotlrn appear under two different subsites.
Collapse
Posted by Eduardo Santos on
Hi everybody,

Sorry to bring this thread back, but I've just upgraded my system to oacs-5-5 and the problem came back. I didn't remember what I've done to fix this, and lost some days again in the work. I have two solutions for the problem wich should include to patch request processor, and I wanted to know your toughts on this:

1 - Fix the ad_conn problem;

2 - Serve always singleton packages.

Fix the ad_conn means that the URL shown by ad_conn will always be the URL from request, even if we are in a subsite. It seems the best behavior on this case to me and solve the POST problem I've mentioned.

Serve the singleton packages mean that singleton packages should be served on all subsites. If it doesn't happen an user doesn't have how to change his locale, as an example.

You can see the full patch here on the bug I've posted on bugtracker: http://www.openacs.org/bugtracker/openacs/patch?patch_number=915

Let me know if you have any comments on this. Also let me know if we can get these changes into kernel.

Best regards

Collapse
Posted by Torben Brosten on
Eduardo Santos,

site_node::conn_url was created to help deal with host node map issues related to "ns_conn url"

When used in place of "ns_conn url" most related issues go away.

For example, the site context bar works using this. The context bar works when accessed from the main site domain and host node map domains.

Maybe if site_node::conn_url were applied to acs-core in more places, more host node map issues would disappear.

The code is located at the bottom of this page:
http://fisheye.openacs.org/browse/OpenACS/openacs-4/packages/acs-tcl/tcl/site-nodes-procs.tcl?r=HEAD

cheers,

Torben

Collapse
Posted by Dave Bauer on
It looks good but I don't think accessing the database is a good idea for a procedure to be used as you suggest.

Also why does it check for nsunix. Is there any such driver anymore?

Hey Torben,

I didn't know this proc, and it seems to be a good solution. However, it doesn't solve the original problem: ad_conn is set up with the wrong url. In order for this to work we would have to replace all [ad_conn url] calls by site_node::conn_url. It seems a huge job for me.

The idea of patching request processor is to provide the information it should in the first place, and make sure all the calls work in the site.

There's still the singleton packages problem, wich is not solved by this proc. Think about the user trying to access acs-lang to change his locale. This is also an issue right now.

Thanks for the info and for your opinion. I'll use this proc to fix context bar, wich was not working on my site. Maybe we should add this fix to the core too (context bar)?

Hi Eduardo Santos,

Maybe a revision of this code should be added to ad_conn?

It seems the most appropriate place..

Could the singleton issue be resolved using an index.vuh file with internal redirects?

Collapse
Posted by Torben Brosten on
Hi Dave,

Accessing the db with cache is about the fastest way I could do this. Besides having neglected to add the cache.. (my bad), there's probably some global variables with the info in it.. Please, "core-ize" this code. Maybe add it to ad_conn url?

I'm not sure if there's an nsunix anymore. If there is, I just wanted to cover all cases.

cheers,

Torben