Forum OpenACS Development: How to get gzip/brotli transfer compression working?

1: How to get gzip/brotli transfer compression working?

Posted by Gaven Eogan on 10/17/19 11:12 AM

Is there some up-to-date information about what exactly to do to get gzip or better yet brotli compression working in Naviserver/OpenACS for normal web content (HTML from TCL/ADP, CSS, JS)?

We are in the ridiculous situation right now of serving eg. 2.5Mb HTML sections of a TCL/ADP page as a 2.5MB uncompressed transfer. Instead of that being compressed down to a fraction of the size before transmission (https://tools.paulcalvano.com/compression.php)

The only mention of it I can find is an old post here from 2013 (https://openacs.org/forums/message-view?message_id=1205589) but it is a confusing because it appears to be about previous AOLServer setup/config and also the newer Naviserver is talking about large static files which we don't care about for now. I tried adding the various parameters to config file but it didn't work. Is the "preauth filter" method still the only way? I tried adding that but that code appears to be obsolete and non working too.

What is the current correct way to enable brotli transfer compression on Windows based Naviserver/OpenACS? (assume latest https://www.spazioit.com/pages_en/sol_inf_en/windows-openacs_en/) Any instructions/examples anywhere?

2: Re: How to get gzip/brotli transfer compression working? (response to 1)

Posted by Gustaf Neumann on 10/17/19 06:32 PM

Brotli compression is provided by NaviServer for static content. Activate and configure it in the "fastpath" section of naviserver [1], see also the sample config file at [2]

gzip compression is implemented in NaviServer for static content and dynamic content. Activating gzip compression for static content works exactly like for brotli static compression (see e.g. [2]), dynamic content is compressed, when "compressenable" is set as a parameter in the per server settings in the config file (see as well [2]).

For compression of static content, external programs are used (as configured in the config file). For dynamic content, NaviServer has to be compiled with gzip support (which is the case on all *nix platforms). Maurizios download page mentions "zlib.1,2,8), therefore i would think, that it contains gzip support.

Try it with a command like the following

curl -H "Accept-Encoding: gzip" -I https://www.amazon.com/

If you see in the result "Content-Encoding: gzip", then the result is sent gzipped (make sure to you a request without redirects)

Hope this helps
-gn

[1] https://naviserver.sourceforge.io/n/naviserver/files/ns_return.html
[2] https://bitbucket.org/naviserver/naviserver/src/default/openacs-config.tcl

3: Re: How to get gzip/brotli transfer compression working? (response to 2)

Posted by Gaven Eogan on 10/24/19 12:00 PM

Thanks for the information. It was more or less what I already had tried to do. But it appears there is some problem with the Windows version.

I installed the latest Windows naviserver to my local machine and added the "compressenable" parameter (so that the server should compress HTML files coming back to browser) but it doesn't do anything. HTML (from TCL/ADP pages) is still coming back uncompressed.

No matter what I do, I can't get it to do the dynamic compression (of TCL/ADP files). As far as I can tell, this is done by simply adding the extra "compressenable" parameter to config file. We are not so much interested in the static compression of other files right now, so don't need the fastpath stuff.

I can see in source code of naviserver in server.c where it uses this parameter and dumps an ns_log line warning of "init server %s: compress is enabled, but no zlib support built in" is there is some zlib problem, but this line doesn't appear in log so I guess there is no problem with zlib.

I would like to be able to add more log lines myself and recomplile to test, but I am not familiar enough with Visual Studio to do this.

Has anyone here used the Windows version of Naviserver and got any compression working?

4: Re: How to get gzip/brotli transfer compression working? (response to 3)

Posted by Gustaf Neumann on 10/24/19 01:33 PM

Concerning:

I can see in source code of NaviServer in server.c where it uses this parameter and dumps an ns_log line warning of "init server %s: compress is enabled, but no zlib support built in"

notice that the change [1] was added on Oct 17, while the last build of Maurizio is from Oct 12th. So, the absence of the message is in your case no indicator.

[1] https://bitbucket.org/naviserver/naviserver/commits/5c30b9aafc0b240d85627addd97277d417d3b246

5: Re: How to get gzip/brotli transfer compression working? (response to 4)

Posted by Gaven Eogan on 10/24/19 01:42 PM

Oh right... I understand. Maybe there could be more logging to show case of everything working as expected (compress enabled and zlib OK), instead of only on error.

So any idea why the compress parameter appears to be getting ignored? Its impossible for me to tell without adding extra logging myself which I don't have the ability to do.

6: Re: How to get gzip/brotli transfer compression working? (response to 5)

Posted by Gustaf Neumann on 10/24/19 02:19 PM

The build machinery on *nix/macOS systems is quite different to win* machines, and for these there are at least 3 way of building (mingw, nmake, native windows tool). For the latter variants, a maintainer has to add compile flags manually, which is not necessary in the *nix family. So, there is a certain possibility that these flags were never set in Maurizios windows build. Have you tried to contact him?

7: Re: How to get gzip/brotli transfer compression working? (response to 6)

Posted by Gaven Eogan on 10/25/19 11:40 AM

No I did not contact him. I don't really know best way to do that.

8: Re: How to get gzip/brotli transfer compression working? (response to 7)

Posted by Gustaf Neumann on 10/25/19 01:11 PM

Why are you not contacting him? You get probably faster a working solution than performing some guessing work here.

9: Re: How to get gzip/brotli transfer compression working? (response to 5)

Posted by Maurizio Martignano on 10/31/19 04:19 PM

Dear Gaven,
the previous versions of my distribution (up to 4.4.9) did not support compression.
I'm currently uploading (it will be available in few hours) version 4.5.0 with compression support included (via zlib 1.2.11).
The thing compiles and links properly and at startup does not print any message of the type "no zlib support built in".
But I did not have the time to do proper testing on this functionality.

I hope it helps, even though I personally believe that compression support is not going to improve dramatically performances.
Maurizio

10: Re: How to get gzip/brotli transfer compression working? (response to 9)

Posted by Gaven Eogan on 11/04/19 12:07 PM

Thanks Maurizio... I will be trying this out today to see if it works... we will still need to do a custom version of this for our system though... can't use your default build in our product.
Yes it may not speed it up much, but its better than none I think and is really an industry standard to to transfer compression of HTML and other easily compressible formats.

ps. the "no zlib support built in" message is something Gustaf added in later source version as he mentioned above. It is probably not in the source you used.

11: Re: How to get gzip/brotli transfer compression working? (response to 10)

Posted by Gustaf Neumann on 11/04/19 12:32 PM

The version built by Maurizio is based on the source code of "31st of October 2019" (he takes always the tip version of NaviServer).

A few sentences above confuse me: Why can't you use the version of Maurizio in your product? You are mentioning a "custom built" the first time: How did you usually build the custom version in your product in the past? Why not redoing your custom build process with recent versions of NaviServer?

13: Re: How to get gzip/brotli transfer compression working? (response to 11)

Posted by Gaven Eogan on 11/04/19 04:37 PM

At the time there were a few bugs we needed that were not in the standard version so we had to fix them in custom version. I think at the time the log rolling was not working. Not sure what else. I wasn't involved in this part of system at that time. There was also other things added. We worked with Maurizio who gave us the build environment. I heard that we didn't need to do a newer build as there were not many changes. But now we do (need to update a lot of things). Yes we will be rebuilding latest version into our own custom (if still needed - have to check this)

12: Re: How to get gzip/brotli transfer compression working? (response to 10)

Posted by Maurizio Martignano on 11/04/19 02:44 PM

Dear Gaven,
1. As Gustaf already mentioned, I always include in my distribution the tip version the Naviserver.
2. I would not have written what I wrote without having first checked that the statement printing "no zlib support built in" is indeed in the codebase.

Best regards,
Maurizio

14: Re: How to get gzip/brotli transfer compression working? (response to 12)

Posted by Maurizio Martignano on 11/16/19 01:00 PM

Dear all,
this is just to mention that brotli executable is now (from version 4.5.1) included in my distribution.
Thank you,
Maurizio

15: Re: How to get gzip/brotli transfer compression working? (response to 14)

Posted by Gaven Eogan on 11/19/19 05:45 PM

I managed to get dynamic compression (ADP/TCL HTML pages) working using the latest version of your Naviserver on our system.
To do this I had to remove one piece of custom code from our code to get it to startup without error.
I also had to remove the section of config that enabled SSL (module/nsssl). We normally run systems with NSSSL enabled.
If I don't do this, I get error during startup:

"Warning: modules nsssl requires a version of NaviServer built with OpenSSL"
Error: modload: c:/naviserver/bin/nsssl.dll: Ns_ModuleInit returned:
Fatal: modload: failed to load module 'c:/naviserver/bin/nsssl.dl

Is there some reason why your version of Naviserver does not have OpenSSL enabled?

Also I can't static compression to do anything even though it is turned on (turning on settings in Fastpath section to enable Brotli or gzip)

ns_param gzip_static true ;# check for static gzip; default: false
ns_param gzip_refresh true ;# refresh stale .gz files on the fly using ::ns_gzipfile
ns_param gzip_cmd "c:/naviserver/bin/gzip -9" ;# use for re-compressing

ns_param brotli_static true ;# check for static brotli files; default: false
ns_param brotli_refresh true ;# refresh stale .br files on the fly using ::ns_brotlifile
ns_param brotli_cmd "c:/naviserver/bin/brotli -f -Z" ;# use for re-compressing

Unless i am doing something wrong... none of the CSS/JS files are ever compressed on server and sent to browser.

I really don't understand why there are so many problems and things not working with your version of Naviserver. Are they all intentionally removed?

16: Re: How to get gzip/brotli transfer compression working? (response to 15)

Posted by Maurizio Martignano on 11/22/19 01:14 AM

Dear Gaven,
1. NSSL support is not turned on by default. You can adjust the configuration in the provided solution file to turn it on.
2. I never used the compression before. The testing is all up to you (did you verify on the first place that commands like "c:/naviserver/bin/brotli" actually work in your environment? did you verify if you have to use the *.exe extension?)
You get something for free, you want it to exactly meet your requirements and then you complain if it doesn't...
😊 😊 😊
In version 4.5.2 I believe I have activated the NSSL support, but for the compression part you're on your own...

Thank you,
Maurizio