Forum OpenACS Development: can't connect to openacs.org port 80: network is unreachable

1: can't connect to openacs.org port 80: network is unreachable

Posted by Iuri Sampaio on 11/25/18 12:21 AM

Attempting to run /acs-admin/install/install, installing packages from OpenACS remote repository, it returns the following.

##
# can't connect to openacs.org port 80: network is unreachable
##

Debugging the error, I got the same error when I ran NaviServer ad_proc ns_http.

Example:
ns_http queue -timeout 30:0 -method GET -headers d5 https://openacs.org/repository/

I've noticed that it was just a collateral effect from another error. The actual cause isn't in the application level.

However, curl and wget are properly installed.

What else could be missing as pre-requisite to make ns_http work?

ERROR:
can't connect to openacs.org port 80: network is unreachable
while executing
"{*}$queue_cmd"
(procedure "::nsf::procs::util::http::native::request" line 99)
invoked from within
"util::http::${impl}::request -method $method -body $body -body_file $body_file -delete_body_file=$delete_body_file_p -h..."
(procedure "::nsf::procs::util::http::request" line 11)
invoked from within
"util::http::request -url $url -method GET -headers $headers -timeout $timeout -max_depth $max_depth -..."
(procedure "::nsf::procs::util::http::get" line 3)
invoked from within
"util::http::get -url $repository_url"
("uplevel" body line 2)
invoked from within
"uplevel 1 [string map {"\\\r\n" " "} $script]"

[24/Nov/2018:23:05:48][22175.7fa2db22d700][-conn:iurix:0:162-] Notice: apm_load_any_changed_libraries: Reloading *-procs.tcl files in this interpreter...
[24/Nov/2018:23:05:48][22175.7fa2db22d700][-conn:iurix:0:162-] Notice: apm_load_any_changed_libraries: Reloading packages/acs-tcl/tcl/http-client-procs.tcl...
[24/Nov/2018:23:05:48][22175.7fa2db22d700][-conn:iurix:0:162-] Notice: ***** ns_http queue -timeout 30:0 -method GET -headers d5 https://openacs.org/repository/
[24/Nov/2018:23:05:48][22175.7fa2db22d700][-conn:iurix:0:162-] Notice: SockConnect: target host openacs.org has associated multiple IP addresses 137.208.116.31 2001:628:404:74::31
[24/Nov/2018:23:05:48][22175.7fa2db22d700][-conn:iurix:0:162-] Notice: async connect to 137.208.116.31 on sock 30 returned EINPROGRESS
[24/Nov/2018:23:05:48][22175.7fa2db22d700][-conn:iurix:0:162-] Notice: connect on sock 33 async 1 err 101 < Network is unreachable >
[24/Nov/2018:23:05:48][22175.7fa2db22d700][-conn:iurix:0:162-] Notice: close sock 33 due to error err 101 < Network is unreachable >
[24/Nov/2018:23:05:48][22175.7fa2db22d700][-conn:iurix:0:162-] Warning: SockConnect fails: SockAddr family AF_INET6, ip 2001:628:404:74::31, port 443
[24/Nov/2018:23:05:48][22175.7fa2db22d700][-conn:iurix:0:162-] Notice: checking entry < 127.0.0.1 > from host_node_map ->
[24/Nov/2018:23:05:48][22175.7fa2db22d700][-conn:iurix:0:162-] Warning: ignore untrusted host header field: '127.0.0.1:8443'
[24

2: Re: can't connect to openacs.org port 80: network is unreachable (response to 1)

Posted by Gustaf Neumann on 11/25/18 03:51 AM

The messages say that from your host, openacs.org can be reached via two IP-addresses, namely its IPv4 and IPv6 address. For unclear reasons, it fails on IPv6 via port 443 (https)

SockConnect fails: SockAddr family AF_INET6, ip 2001:628:404:74::31 port 443

Can you connect via browser with the URL?

https://[2001:628:404:74::31]/

I have currently no IPv6 connection to test.

3: Re: can't connect to openacs.org port 80: network is unreachable (response to 2)

Posted by Iuri Sampaio on 11/25/18 11:24 PM

Gustaf,

I believe the problem has another cause.

Hosting server doesn't support IPv6. That's why socketConnect fails on IPv6.

However, that wasn't supposed affect, once NS parameters are disabled since the beginning (i.e. within ns-config.tcl).

#set address_v6         ::1        ;# listen on loopback via IPv6                                                                                                                                           
#set address_v6         ::0        ;# listen on all IPv6 addresses

So, I can "wget" https://openacs.org/ but I can't wget https://[2001:628:404:74::31]/

Regarding your suspects on unclear reasons, it fails on IPv6 via port 443 (https)... I guess the hardware, which hosts our VMs, doesn't support IPv6. I believe Patrick can tell us why.

Thus, as expected IPv4 works and IPv6 doesn't.

If IPv4 works why does ns_http request fail?

root@iurix:~# wget https://[2001:628:404:74::31]/
--2018-11-25 22:13:31--  https://[2001:628:404:74::31]/
Connecting to [2001:628:404:74::31]:443... failed: Network is unreachable.
root@iurix:~#

4: Re: can't connect to openacs.org port 80: network is unreachable (response to 1)

Posted by Gustaf Neumann on 11/25/18 11:56 PM

Not sure, i under stand you correctly. openacs.org runs since many years here in Vienna on one of our research machines, it is not a VM. However, lately, there were several changes for DNS entries going on, since there were problems with recipients on gmail via IPv6, so maybe there was some collateral damage ...

5: Re: can't connect to openacs.org port 80: network is unreachable (response to 4)

Posted by Iuri Sampaio on 11/26/18 03:25 AM

Aside from the lack of support for IPv6 on my end, ad_proc ns_http works fine against several other endpoints. (ex. google REST APIs, etc).
Most of them, I've used ad_procs [ns_http], [util::http::get] and [util::http::post]

But when the URL is https://openacs.org/repository/ or https://openacs.org/repository/, they fail.

The source code is 5.9.2d1, https://openacs.org/repository/download/apm/acs-kernel-5.9.2d1.apm

Furthermore, in a different environment, different server and hosting provider, network is reachable and the page /acs-admin/install/install works fine.

I've written messages to the error.log, to confirm that URLs are the same.

[25/Nov/2018:20:43:37][762.7fd427d1a700][-conn:litli:0:304-] Notice: Running ad_proc util::http::native::request ...(****** https://openacs.org/repository/
[25/Nov/2018:20:43:37][762.7fd427d1a700][-conn:litli:0:304-] Notice: Running ad_proc util::http::native::request ...(****** https://openacs.org/repository/

traceroute command shows nothing unusual, tested from both servers.

1) From the server, which Network is reachable. 9 hoops.

2) From the server, which Network is unreachable. Although It has more hoops ( ie. 12), it does reach pa-outside.wu-wien.ac.at

litli@litli:~$ traceroute 137.208.116.31
traceroute to 137.208.116.31 (137.208.116.31), 30 hops max, 60 byte packets
1 * * *
2 te0-0-0-2.rcr11.b011027-3.yyz02.atlas.cogentco.com (38.122.101.121) 3.819 ms 3.888 ms 3.851 ms
3 te0-1-0-10.ccr31.yyz02.atlas.cogentco.com (154.54.6.237) 3.547 ms 3.554 ms 4.184 ms
4 level3.yyz02.atlas.cogentco.com (154.54.11.210) 3.972 ms 3.936 ms 3.878 ms
5 ae-2-2.bar4.Vienna1.Level3.net (4.69.148.154) 107.752 ms 107.725 ms 107.667 ms
6 ACONET.bar4.Vienna1.Level3.net (212.73.203.18) 107.459 ms 107.235 ms 107.195 ms
7 wien1.aco.net (193.171.13.17) 107.031 ms 106.971 ms 106.983 ms
8 ex-1.wu-wien.ac.at (193.171.13.18) 106.846 ms 106.889 ms 106.848 ms
9 pa-outside.wu-wien.ac.at (137.208.9.41) 107.682 ms 108.000 ms 107.866 ms
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *

iurix@iurix:/var/www/iurix$ traceroute 137.208.116.31
traceroute to 137.208.116.31 (137.208.116.31), 30 hops max, 60 byte packets
1 * * *
2 ae0-14.border1.denver4.handynetworks.com (68.71.128.69) 0.199 ms 0.225 ms 0.219 ms
3 10ge13-1.core1.den1.he.net (216.66.78.125) 0.440 ms 0.688 ms 0.801 ms
4 100ge14-1.core1.mci3.he.net (184.105.64.50) 10.771 ms 10.803 ms 10.844 ms
5 100ge15-1.core2.chi1.he.net (184.105.222.78) 31.333 ms 31.386 ms 100ge8-1.core2.chi1.he.net (184.105.81.210) 23.279 ms
6 100ge16-1.core1.nyc4.he.net (184.105.223.162) 39.866 ms 39.672 ms 39.653 ms
7 100ge4-1.core1.par2.he.net (184.105.81.78) 129.829 ms 129.707 ms 129.413 ms
8 100ge5-2.core1.vie1.he.net (184.105.65.6) 126.558 ms 126.575 ms 126.578 ms
9 wien1.aco.net (193.203.0.1) 126.930 ms 126.948 ms 126.871 ms
10 ex-1.wu-wien.ac.at (193.171.13.18) 126.795 ms 126.798 ms 126.774 ms
11 ex-1.wu-wien.ac.at (193.171.13.18) 126.984 ms pa-outside.wu-wien.ac.at (137.208.9.41) 127.344 ms ex-1.wu-wien.ac.at (193.171.13.18) 126.967 ms
12 * pa-outside.wu-wien.ac.at (137.208.9.41) 127.518 ms *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *

6: Re: can't connect to openacs.org port 80: network is unreachable (response to 1)

Posted by Gustaf Neumann on 11/28/18 07:43 AM

Once your DNS resolves openacs.org with an IPv6 address, you have at least minimal support for IPv6 on that machine.

I've just checked access to https://openacs.org via IPv6:

 ns_http run https:/openacs.org/repository/

returns as expected the full page.

 ns_http run {https://[2001:628:404:74::31]/repository/}

returns the redirect to the canonical URL

status 301 time 0:10582 headers d4 body {status 301 time 0:10582 headers d4 body {<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 4.01//EN">
<html>
<head>
<title>Redirection</title>
</head>
<body>
<h2>Redirection</h2>
<a href="https://openacs.org/repository/">The requested URL has moved permanently here.</a>
<p align='right'><small><i>NaviServer/4.99.17 on https://openacs.org</i></small></p>

</body></html>
} https {sslversion TLSv1.3 cipher TLS_AES_256_GCM_SHA384}

So, the IPv6 setup on openacs.org is correct. The current access log shows 23k IPv6 requests to openacs.org, about 2% of the requests.

Concernging your two machines: do both have the exactly the same versions of the OS and NaviServer and OpenACS installed?

7: Re: can't connect to openacs.org port 80: network is unreachable (response to 6)

Posted by Iuri Sampaio on 11/29/18 10:27 PM

Thanks Gustaf,
The cause was IPv6 being enabled, while the hardware of the server provides no support to it. Thus, every time ns_http tried to use IPv6, it returned a colateral error (i.e. network in unreachable).

We've have disabled IPv6, and now ns_http runs only on IPv4 only and the page "/admin/install" loads successfully.

Best wishes,