Forum OpenACS CMS: Problem installing Naviserver (Gentoo)

Request notifications

I know, this is not a OpenACS-specific issue, but I couldn't find a forum where to post my question.
My OpenACS install is quite old, and I wanted to update it with the latest build and switch from AOLServer to Naviserver on the way.

The problem is that I keep getting an error when running make for Naviserver. Running ./configure didn't give any errors.

OS: Gentoo Linux with Kernel 4.9.95
TCL 8.6.8 installed from sources
Postgresql 10 installed from the Gentoo repository (with the emerge package managing system)

Configure parameters used:

./configure --prefix=/usr/local/ns --with-tcl=/usr/local/tcl8.6.8/lib --enable-symbols

The error I get when running gcc make:
tclcrypto.c:283:34: error: unknown type name "ECDSA_SIG"
static void ECDSA_SIG_get0(const ECDSA_SIG*sig, const BIGNUM **pr, const BIGNUM **ps)
tclcrypto.c: In function "ECDSA_SIG_get0"

What am I doing wrong?

Thanks for your help!

Collapse
Posted by Gustaf Neumann on
From the error message I deduce, that you are building the (unreleased) tip version of NaviServer from bitbucket, and you are building against an unknown (unsupported) version of OpenSSL installed. If i see correctly, ECDSA_SIG was introduced to OpenSSL in 2016 in version 0.9.8. What version of OpenSSL are you using? Are you using some alternative version of OpenSSL?

The support of ECDSA (Elliptic Curve Digital Signature Algorithm) in NaviServer is currently extended, since the demand for crypto support in web servers is on the rise. Also, the OpenSSL crypto API is changing. Certainly, NaviServer should still build (with reduced functionality) on system like yours

all the best
-gn

Collapse
Posted by Michael Martí Menzel on
Thanks so much for your reply, Gustaf.
Yes, I'm installing the version from bitbucket. The released version from sourceforge gave me the same error.

My openssl is version is 1.0.2o-r3

Does this info help?

Collapse
Posted by Gustaf Neumann on

you wrote:

The released version from sourceforge gave me the same error.

no, this can't be, the whole source tree if 4.99.16 does not contain ECDSA_SIG

/usr/local/src% fgrep -r ECDSA_SIG naviserver-4.99.16
/usr/local/src% 

I have as well doubts that you are using OpenSSL 1.0.2o-r3 during the the compilation, since it defines ECDSA_SIG. One can figure out, what's happening in your configuration by asking the C-compiler to keep the result of the pre-processor

% cd naviserver
naviserver% make "CFLAGS_DEFAULT=-save-temps -DNDEBUG"  nsd/tclcrypto.o 

The command above produces a file named tclcrypto.i in the current directory, which we can check then, where it got the include files from, and what is the version of OpenSSL. Below are the results from my local notebook:

naviserver% fgrep evp.h  tclcrypto.i |head -1
# 1 "/usr/local//include/openssl/evp.h" 1 3

naviserver% fgrep opensslv.h  tclcrypto.i |head -1
# 1 "/usr/local//include/openssl/opensslv.h" 1 3

naviserver% fgrep OPENSSL_VERSION_NUMBER /usr/local//include/openssl/opensslv.h
# define OPENSSL_VERSION_NUMBER  0x10101005L

What do you get?

Collapse
Posted by Michael Martí Menzel on
Strange, Gustaf, all I get in tclcrypto.i is this:

# 1 "nsd/tclcrypto.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 1 "<command-line>" 2
# 1 "nsd/tclcrypto.c"
# 36 "nsd/tclcrypto.c"
# 1 "nsd/nsd.h" 1

Make ends with:

fatal error: ns.h: No such file or directory

but ns.h is there (in the include dir)

Collapse
Posted by Gustaf Neumann on

that is strange. It looks, like the include path is not found. Run the following command that will lead to the error above.

make  nsd/tclcrypto.o 

In the first line of the output of that command you see the compiler and flags. Add to this line -save-temps. For my machine, this looks like

% make "CFLAGS_DEFAULT=-save-temps -DNDEBUG"  nsd/tclcrypto.o 
gcc -save-temps -DNDEBUG -Wall -fno-common -pipe    -I../include -I"/usr/local/ns/include"  -DHAVE_CONFIG_H -I/usr/local//include   -c -o nsd/tclcrypto.o nsd/tclcrypto.c

If this does not help: In case i want to setup a machine with Gentoo for helping you here, what is the exact version of Gentoo i should be using? Are you using the packaged version of OpenSSL for Gentoo?

Collapse
Posted by Michael Martí Menzel on
Hello again, Gustaf.

When I enter and execute the command I get this:

gcc -save-temps -DNDEBUG -Wall -fPIC -pipe -m64 -I../include -I"/usr/local/tcl8.6.8/include" -DHAVE_CONFIG_H -c -o nsd/tclcrypto.o nsd/tclcrypto.c
gcc: warning: -pipe ignored because -save-temps specified
In file included from nsd/tclcrypto.c:36:0:
nsd/nsd.h:33:16: fatal error: ns.h: No such file or directory
#include "ns.h"
^
compilation terminated.
make: *** [: nsd/tclcrypto.o] Error 1

My gentoo release is 2.4.1 for Amd64.

Thanks again for your help!

Collapse
Posted by Gustaf Neumann on

so, during a meeting, i installed Gentoo Linux and did the following steps, but "unfortunately" everything worked fine for me.

 # cat /etc/gentoo-release 
 Gentoo Base System release 2.4.1

 # update gentoo and install libraries
 sudo bash
 emerge --sync
 emerge --ask  dev-vcs/mercurial
 emerge --ask  dev-libs/openssl

 # Install Tcl in /usr/local/ns
  wget https://prdownloads.sourceforge.net/tcl/tcl8.6.8-src.tar.gz
  tar zxvf tcl8.6.8-src.tar.gz
  cd tcl8.6.8/unix
  ./configure --prefix=/usr/local/ns
  make install
  cd ../..

  # Install NaviServer from Bitbucket
  hg clone https://gustafn@bitbucket.org/naviserver/naviserver
  cd naviserver
  sh autogen.sh
  ./configure
  make
  make install

  # as normal user (non-root)
  gentoo /home/vagrant # /usr/local/ns/bin/nsd -c
[15/Jun/2018:08:20:44][23783.7f66110e5740][-main-] Notice: OpenSSL 1.0.2o  27 Mar 2018 initialized
[15/Jun/2018:08:20:44][23783.7f66110e5740][-main-] Notice: binder: started
[15/Jun/2018:08:20:44][23783.7f66110e5740][-main-] Notice: nsmain: NaviServer/4.99.17d2 starting
[15/Jun/2018:08:20:44][23783.7f66110e5740][-main-] Notice: nsmain: security info: uid=1000, euid=1000, gid=1000, egid=1000
[15/Jun/2018:08:20:44][23783.7f66110e5740][-main-] Notice: nsmain: Tcl version: 8.6.8
...

So, what's different in your setup?

Collapse
Posted by Michael Martí Menzel on
Hello Gustaf,

many, many thank for researching this. I'm going through every step of your installation and will report back shortly.

Best,

Michael

Collapse
Posted by Michael Martí Menzel on
Hello again, Gustaf.

I've proceeded exactly the way you did yesterday, and I get again an error.

gcc -save-temps -DNDEBUG -Wall -fPIC -pipe -I../include -I"/usr/src/tcl8.6.6/generic" -DHAVE_CONFIG_H -c -o nsd/tclcrypto.o nsd/tclcrypto.c
gcc: warning: -pipe ignored because -save-temps specified
In file included from nsd/tclcrypto.c:36:0:
nsd/nsd.h:33:16: fatal error: ns.h: No such file or directory
#include "ns.h"
^
compilation terminated.
make: *** [: nsd/tclcrypto.o] Error 1

This is really strange, isn't it?

Best,

Michael

Collapse
Posted by Gustaf Neumann on

Does it mean, that you get still compilation error with "ECDSA_SIG" (which i did not get)?

In case, you got an error in the steps in posting (4), we can try to debug where the version of OpenSSL is taken from by checking the .i file. To produce the .i file from the "naviserver" directory, use:

gcc -save-temps -DNDEBUG -Wall -fPIC -pipe -I./include -I"/usr/src/tcl8.6.6/generic" -DHAVE_CONFIG_H -c -o nsd/tclcrypto.o nsd/tclcrypto.c

Note, one "." removed in the first "-I" flag

In case, the steps in posting (4) did not produce an error, NaviServer is installed already.

Collapse
Posted by Michael Martí Menzel on
Thanks again  for your help, Gustaf. This is the output of the "ggc --save-temps..." command.

# 1 "nsd/tclcrypto.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 1 "<command-line>" 2
# 1 "nsd/tclcrypto.c"
# 36 "nsd/tclcrypto.c"
# 1 "nsd/nsd.h" 1
# 33 "nsd/nsd.h"
# 1 "./include/ns.h" 1
# 40 "./include/ns.h"
# 1 "./include/nsversion.h" 1
# 41 "./include/ns.h" 2
# 1 "./include/nsthread.h" 1
# 45 "./include/nsthread.h"
# 1 "./include/nsconfig.h" 1
# 46 "./include/nsthread.h" 2

... (shortened by webmaster) ...

void ERR_load_OBJ_strings(void);
# 95 "/usr/include/openssl/evp.h" 2 3 4
# 129 "/usr/include/openssl/evp.h" 3 4
struct evp_pkey_st {
    int type;
    int save_type;
    int references;
    const EVP_PKEY_ASN1_METHOD *ameth;
    ENGINE *engine;
    union {
        char *ptr;

        struct rsa_st *rsa;


        struct dsa_st *dsa;


        struct dh_st *dh;




    } pkey;
    int save_parameters;
    struct stack_st_X509_ATTRIBUTE *attributes;
} ;

... (shortened by webmaster) ...
===================================================
Kind regards,

Michael

Collapse
Posted by Gustaf Neumann on

Dear Michael,

i think, i have found the problem. It seems to me, that for whatever reason, you have probably a version of OpenSSL installed, which has EC support turned off. The first clue came from your .i file, where the definition of evppkeyst does not contain eckeyst.

struct evp_pkey_st {
    int type;
    int save_type;
    int references;
    const EVP_PKEY_ASN1_METHOD *ameth;
    ENGINE *engine;
    union {
        char *ptr;
# ifndef OPENSSL_NO_RSA
        struct rsa_st *rsa;     /* RSA */
# endif
# ifndef OPENSSL_NO_DSA
        struct dsa_st *dsa;     /* DSA */
# endif
# ifndef OPENSSL_NO_DH
        struct dh_st *dh;       /* DH */
# endif
# ifndef OPENSSL_NO_EC
        struct ec_key_st *ec;   /* ECC */
# endif
    } pkey;
    int save_parameters;
    STACK_OF(X509_ATTRIBUTE) *attributes; /* [ 0 ] */
} /* EVP_PKEY */ ;

The second clue came from googling around, finding errors from other projects on Gentoo, where the missing EC support in OpenSSL was as well the cause.

It is not clear to me, why my build of OpenSSL on the same version of Gentoo was compiled with EC support, but not on yours. There is probably on Gentoo some magic place where some options for the compilation are stored. .. or maybe some version dependencies from other installed packages, etc. But i have no Gentoo experience, please find a Gentoo expert for more hints on this.

I have modified the NaviServer sources such it compiles now also against OpenSSL versions without OpenSSL support. However, I would recommend that you enable EC support also in OpenSSL on your machine for full functionality.

-g

PS: I'll check, how to shrink your posting with more than 28k lines above, which is probably in that detail not of interest for the normal reader.

Collapse
Posted by Michael Martí Menzel on
Hello Gustaf,

again many thanks for your invaluable help.
You are absolutely right! The problem is due to an use flag set by defult in Gentoo.

https://packages.gentoo.org/useflags/bindist

I've reemerged openssl with this flag unset and I could compile and install naviserver!!

I'm facing now some other problems when running it, that I guess are minor.

Extracts from my error log say:

[xxxxxx...xxxxx] Error: pidfile: failed to open pid file 'nsd.pid': Permission denied

I've created a blank pid file under logs and changed its owner to the non-root user under which I run nsd.

[xxxxxx...xxxxx] Warning: modload: could not load /usr/local/ns/bin/nscp.so: /usr/local/ns/bin/nscp.so: undefined symbol: Ns_ConfigCreateSection

I'd be very grateful for any hint regarding this error and warning.

Best,

Michael

Collapse
Posted by Gustaf Neumann on

i wonder, why the version i installed on a fresh Gentoo installation was built with EC enabled... but that is not important.

Concerning permissions: typically, after installation the following should happen:

  • create a special user for naviserver (such as nsadmin),
  • change permissions on naviserver log file accordingly,
  • start naviserver with the flags "-u nsadmin -g nsadmin"

The path of the pid-file might come from the configuration, so make sure, that the path is correct in the config-file you are using. I just noticed that the sample config file "nsd-config.tcl" has actually an incorrect pidfile entry, in case you are using this, just comment that line.

Here is what i get, ... up to the the loading of nscp on Genoo

gentoo # groupadd nsadmin
gentoo # useradd -g nsadmin nsadmin
gentoo # chown -R nsadmin:nsadmin /usr/local/ns/logs
gentoo # /usr/local/ns/bin/nsd -u nsadmin -t /usr/local/ns/conf/nsd-config.tcl -f
[18/Jun/2018:04:36:01][6324.7fb689055740][-main-] Notice: OpenSSL 1.0.2o  27 Mar 2018 initialized
[18/Jun/2018:04:36:01][6324.7fb689055740][-main-] Notice: binder: started
[18/Jun/2018:04:36:01][6324.7fb689055740][-main-] Notice: nsmain: enable progress statistics for uploads >= 1048576 bytes
[18/Jun/2018:04:36:01][6324.7fb689055740][-main-] Notice: nsmain: NaviServer/4.99.17d2 starting
[18/Jun/2018:04:36:01][6324.7fb689055740][-main-] Notice: nsmain: security info: uid=1001, euid=1001, gid=1001, egid=1001
[18/Jun/2018:04:36:01][6324.7fb689055740][-main-] Notice: nsmain: Tcl version: 8.6.8
[18/Jun/2018:04:36:01][6324.7fb689055740][-main-] Notice: nsmain: max files: soft limit 4096, hard limit 4096
[18/Jun/2018:04:36:01][6324.7fb689055740][-main-] Warning: nsmain: rl_cur (4096) > FD_SETSIZE (1024), select() calls should not be used
[18/Jun/2018:04:36:01][6324.7fb689055740][-main-] Notice: pool default: queueLength 0 low water 0 high water 0
[18/Jun/2018:04:36:01][6324.7fb689055740][-main-] Notice: nsd/init.tcl[default]: booting virtual server:  tcl system encoding: "utf-8"
[18/Jun/2018:04:36:01][6324.7fb689055740][-main-] Notice: modload: loading module nscp from file nscp.so
[18/Jun/2018:04:36:01][6324.7fb689055740][-main-] Notice: nscp[default]: listening on [0.0.0.0]:4080
[18/Jun/2018:04:36:01][6324.7fb689055740][-main-] Notice: nscp[default]: added user: ""
...

So, i see no problems with the nscp module (see last two lines). Can it be that you have a leftover of nscp from earlier attempts?

Make sure to run "make install" from the naviserver source directory to install all modules.

As you are installing manually, i would as well recommend to install nsf

gentoo #  emerge --ask  dev-vcs/git
gentoo # cd /usr/local/src
gentoo # git clone git://alice.wu-wien.ac.at/nsf
gentoo #  cd nsf
gentoo # ./configure --prefix=/usr/local/ns
gentoo # make install
Collapse
Posted by Michael Martí Menzel on
Hello again, Gustaf.

The problem with the pid-file is gone. BTW I had to create a "log" subdirectory instead of "logs". I couldn't find a setting for the log directory in nsd-config.tcl. I could find it in the sample config-file though. I used this file as a template for my config-file, and "logs" is being used by the server now.

I keep getting the "Could not load module" error. This time it involves nssock.so. This is due to the fact that the loading order is different in my new config-file from what t was in nsd-config.tcl. So, the server process is unable to load any module as per now.

Collapse
Posted by Gustaf Neumann on

NaviServer (and probably aolsever as well) uses since ages "/usr/local/ns/logs" as log directory. If you provide a path for the pidfile with "/log/" instead of "/logs/" in you config file, you have to create the directory, NaviServer does not create it automatically.

Do you get the error with any of the sample config files (nsd-config.tcl or sample-config.tcl) or just with your modified config file? For me, in both cases, loading works correctly on the Gentoo box. What do you mean that the loading order is different?

What do you get from the following two commands:

gentoo # ls -ltr /usr/local/ns/bin/*
gentoo # ls -ldtr /usr/local/ns/lib/*.so
Collapse
Posted by Michael Martí Menzel on
The problem was that I couldn't get NaviServer to use the "logs" directory. It kept complaining that it could not write the pid-file into the directory called "log". As I could not find any setting for the log file in nsd-config.tcl, I assumed it was a default. In sample-config.tcl I found the way to set the log directory.

- With nsd-config.tcl and sample-config.tcl I get the module loading error. It is always the first module it tries to load the one that gives the error

- With simple-config.tcl I get the following error:
Fatal: nsmain: missing: [ns/parameters]home

- With openacs-config.tcl I get the following error:
Error: Tcl exception: bad option "severity": must be hold, count, get, peek, flush, release, or truncate
while executing "ns_logctl severity "Debug(ns:driver)" $debug"
Fatal: config error

Directory listings:
gentoo # ls -ltr /usr/local/ns/bin/*
-rwxr-xr-x 1 root root 24328 Jun 18 14:09 nsthreadtest
-rwxr-xr-x 1 root root 17315 Jun 18 14:09 init.tcl
-rwxr-xr-x 1 root root 8144 Jun 18 14:09 nsd
-rwxr-xr-x 1 root root 12816 Jun 18 14:09 nssock.so
-rwxr-xr-x 1 root root 28584 Jun 18 14:09 nscgi.so
-rwxr-xr-x 1 root root 18952 Jun 18 14:09 nscp.so
-rwxr-xr-x 1 root root 28584 Jun 18 14:09 nslog.so
-rwxr-xr-x 1 root root 27752 Jun 18 14:09 nsperm.so
-rwxr-xr-x 1 root root 8000 Jun 18 14:09 nsdb.so
-rwxr-xr-x 1 root root 8216 Jun 18 14:09 nsdbtest.so
-rwxr-xr-x 1 root root 24992 Jun 18 14:09 nsssl.so
-rwxr-xr-x 1 root root 8080 Jun 18 14:09 nsproxy.so
-rwxr-xr-x 1 root root 8152 Jun 18 14:09 nsproxy
-rwxr-xr-x 1 root root 13868 Jun 18 14:09 install-sh
-rwxr-xr-x 1 root root 673 Jun 18 14:59 nxsh
-rwxr-xr-x 1 root root 712 Jun 18 14:59 nxwish
-rwxr-xr-x 1 root root 775 Jun 18 14:59 xotclsh
-rwxr-xr-x 1 root root 794 Jun 18 14:59 xowish

gentoo # ls -ldtr /usr/local/ns/lib/*.so
-rwxr-xr-x 1 root root 44728 Jun 18 14:09 /usr/local/ns/lib/libnsthread.so
-rwxr-xr-x 1 root root 920544 Jun 18 14:09 /usr/local/ns/lib/libnsd.so
-rwxr-xr-x 1 root root 52024 Jun 18 14:09 /usr/local/ns/lib/libnsdb.so
-rwxr-xr-x 1 root root 47744 Jun 18 14:09 /usr/local/ns/lib/libnsproxy.so
lrwxrwxrwx 1 root root 41 Jun 18 14:59 /usr/local/ns/lib/libnsf2.1.0.so -> /usr/local/ns/lib/nsf2.1.0/libnsf2.1.0.so

Collapse
Posted by Gustaf Neumann on

The symptoms sound like you are starting with an nsd (or libnsd), which is from an very old version of naviserver or aolserver.

Are you usre, you are starting the server with the binary /usr/local/ns/bin/nsd?

What is the output of ldd /usr/local/ns/bin/nsd ?

Collapse
Posted by Michael Martí Menzel on
I think I've installed the one from bitbucket (the latest)!

The output I get from ldd is:

linux-vdso.so.1 (0x00007ffed47e6000)
libnsd.so => /usr/local/aolserver/lib/libnsd.so (0x00007ff16d735000)
libz.so.1 => /lib64/libz.so.1 (0x00007ff16d51e000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007ff16d2e6000)
libnsthread.so => /usr/local/aolserver/lib/libnsthread.so (0x00007ff16d0db000)
libtcl8.6.so => /usr/lib64/libtcl8.6.so (0x00007ff16cd29000)
libgcc_s.so.1 => /usr/lib/gcc/x86_64-pc-linux-gnu/6.4.0/libgcc_s.so.1 (0x00007ff16cb13000)
libm.so.6 => /lib64/libm.so.6 (0x00007ff16c7c5000)
libssl.so.1.0.0 => /usr/lib64/libssl.so.1.0.0 (0x00007ff16c557000)
libcrypto.so.1.0.0 => /usr/lib64/libcrypto.so.1.0.0 (0x00007ff16c120000)
libc.so.6 => /lib64/libc.so.6 (0x00007ff16bd5d000)
libdl.so.2 => /lib/libdl.so.2 (0x00007ff16bb59000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00007ff16b939000)
/lib64/ld-linux-x86-64.so.2 (0x00007ff16dbc3000)

Collapse
Posted by Gustaf Neumann on

There is the problem: For whatever reason, the nsd you are using is linked against /usr/local/aolserver/lib/

libnsd.so => /usr/local/aolserver/lib/libnsd.so (0x00007ff16d735000)
libnsthread.so => /usr/local/aolserver/lib/libnsthread.so (0x00007ff16d0db000)

A correct output looks like:

gentoo # ldd /usr/local/ns/bin/nsd
    linux-vdso.so.1 (0x00007ffdb4f49000)
    libnsd.so => /usr/local/ns/lib/libnsd.so (0x00007fda05726000)
    libz.so.1 => /lib64/libz.so.1 (0x00007fda0550f000)
    libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007fda052d7000)
    libnsthread.so => /usr/local/ns/lib/libnsthread.so (0x00007fda050ce000)
    libtcl8.6.so => /usr/local/ns/lib/libtcl8.6.so (0x00007fda04d15000)
    libgcc_s.so.1 => /usr/lib/gcc/x86_64-pc-linux-gnu/6.4.0/libgcc_s.so.1 (0x00007fda04afe000)
    libm.so.6 => /lib64/libm.so.6 (0x00007fda047ac000)
    libssl.so.1.0.0 => /usr/lib64/libssl.so.1.0.0 (0x00007fda0453f000)
    libcrypto.so.1.0.0 => /usr/lib64/libcrypto.so.1.0.0 (0x00007fda0410c000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fda03d4a000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007fda03b46000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fda03926000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fda05bf4000)
Collapse
Posted by Michael Martí Menzel on
I hadn't noticed it!! I'm going to investigate it.

Thanks for your help!

Collapse
Posted by Michael Martí Menzel on
Hello again Gustag,

I've removed the paths to AOLServer (and old install) from LD_LIBRARY_PATH and it works now!

Thanks so much for your help!

Best,

Michael

Collapse
Posted by Gustaf Neumann on
arrgh. LD_LIBRARY_PATH is considered harmful [1].

Glad it works finally!
Cheers!

[1] https://gms.tf/ld_library_path-considered-harmful.html