Forum OpenACS Development: Re: Issues with site-wide SSL configuration

8: Re: Issues with site-wide SSL configuration (response to 1)

Posted by Jose Mendez on 04/05/19 12:28 AM

Sad to report that the upgrade to OpenSSL (v1.0.2.r) and recompiling of the nsopenssl driver did not resolve our issues. After testing it on our development environment we did the upgrade in Production with the same results.

Initially we only updated the software without enabling the site-wide configuration. We ran it for a day and our HTTPS-restricted applications showed no crippling problems but did notice some connections would be in a CLOSE_WAIT state for some time and would eventually go away.

This morning we forced HTTPS site-wide, and as the activity on the site increased we started to see intermittent delays until the site became unresponsive. netstat showed a large number of connections in a CLOSE_WAIT state. After some time, we received a "fatal signal 11" error and the site crashed. The crash is not something we had experienced before. We suspect this is because we had not left it running in this state for as long as we did today.

After rolling back the site-wide configuration, we restarted the service and noticed that HTTPS connections would still hang from time to time. In our original configuration non HTTPS-restricted URLs are redirected back to HTTP and these redirects would also take a long time. This is something we had not experienced before. It took restarting the server to alleviate this condition.

At this point, we have ran out of ideas on what to look for and troubleshoot. We are going to start testing the NGINX solution and hope we won't run into similar issues. I welcome any other tips or troubleshooting ideas in the meantime.

9: Re: Issues with site-wide SSL configuration (response to 8)

Posted by Gustaf Neumann on 04/05/19 09:03 AM

You can get a stable configuration with both other approaches.

- We run a large site behind nginx with up 5mio page views per day, all requests via HTTPS, backend currently via HTTP (change to HTTPS on the backend is on the agenda). When running behind a proxy, also the backend requires some changes. There are as well more changes involved for secure cookie handling for secure cases, in case you need this. There is a small wiki page [1] for current OpenACS (which is certainly not applicable for your methusalem), which might point to some problem areas.
- with NaviServer+nsssl, the reverse proxy issues won't arise, but you have to handle the porting steps mentioned above in advance. E.g. OpenACS.org runs with current OpenACS+NaviServer. This is a low traffic site, between 100k and 300k requests per day (the same nsd handles openacs.org, dotlrn.org and soon fisheye.openacs.org).

Both approaches run very stable, we every experiences the issues you are mentioning.

all the best
-gn
[1] https://openacs.org/xowiki/running_behind_a_proxy

10: Re: Issues with site-wide SSL configuration (response to 9)

Posted by Jose Mendez on 04/08/19 04:44 PM

Thanks Gustaf.

We got NGINX running in our test environment. Listening on port 80 and 443 with the AOLServer back-end listening on 8080. Installation and setup was straight forward.

The one thing we noticed was that our code uses [ad_conn location] for navigation links and we were getting the backend port (8080) in our links. To address this, we pulled some procs from a newer version of ACS (util::split_location and ns_parseurl) and added code in ad_conn to handle getting the location without the port number.

I noticed that in the version of ACS where we pulled the procs from, that this would be handled by a utility proc (util_current_location) and not in ad_conn directly. Should we be concerned about adding this to ad_conn?

Best

-Jose

11: Re: Issues with site-wide SSL configuration (response to 10)

Posted by Maurizio Martignano on 04/08/19 05:08 PM

Hello Jose,
I believe you can fix this issue by setting the following parameter

proxy_set_header X-Forwarded-For $remote_addr;

in the server section of your Nginx configuration file.
Try to google how to use the 'X-Forwarded-For' parameter in some Nginx configuration examples.

Hope it helps,
Maurizio

12: Re: Issues with site-wide SSL configuration (response to 10)

Posted by Gustaf Neumann on 04/09/19 02:38 PM

This is what i referred to with "When running behind a proxy, also the backend requires some changes". Your old version of ACS has to be made aware that it is not directly communicating with the peer, but with a proxy. So it will think that the other side can directly reply to its own address, or that the peer is the proxy server, which will be noted as well in the access.log containing just the IP addresses of the proxy instead of the IP address of the real peer.

Current OpenACS has the commands (see [1]) for handling such cases:
- [ad_conn peeraddr]
- [ad_conn behind_proxy_p]
- [ad_conn behind_secure_proxy_p]

Check out the source code of current OpenACS, when something special have to be done in the reverse proxy cases, some of these might not be relevant for you.

-gn
[1] https://openacs.org/xowiki/running_behind_a_proxy