Forum OpenACS Development: How to get gzip/brotli transfer compression working?

Is there some up-to-date information about what exactly to do to get gzip or better yet brotli compression working in Naviserver/OpenACS for normal web content (HTML from TCL/ADP, CSS, JS)?

We are in the ridiculous situation right now of serving eg. 2.5Mb HTML sections of a TCL/ADP page as a 2.5MB uncompressed transfer. Instead of that being compressed down to a fraction of the size before transmission (https://tools.paulcalvano.com/compression.php)

The only mention of it I can find is an old post here from 2013 (https://openacs.org/forums/message-view?message_id=1205589) but it is a confusing because it appears to be about previous AOLServer setup/config and also the newer Naviserver is talking about large static files which we don't care about for now. I tried adding the various parameters to config file but it didn't work. Is the "preauth filter" method still the only way? I tried adding that but that code appears to be obsolete and non working too.

What is the current correct way to enable brotli transfer compression on Windows based Naviserver/OpenACS? (assume latest https://www.spazioit.com/pages_en/sol_inf_en/windows-openacs_en/) Any instructions/examples anywhere?

Brotli compression is provided by NaviServer for static content. Activate and configure it in the "fastpath" section of naviserver [1], see also the sample config file at [2]

gzip compression is implemented in NaviServer for static content and dynamic content. Activating gzip compression for static content works exactly like for brotli static compression (see e.g. [2]), dynamic content is compressed, when "compressenable" is set as a parameter in the per server settings in the config file (see as well [2]).

For compression of static content, external programs are used (as configured in the config file). For dynamic content, NaviServer has to be compiled with gzip support (which is the case on all *nix platforms). Maurizios download page mentions "zlib.1,2,8), therefore i would think, that it contains gzip support.

Try it with a command like the following

curl -H "Accept-Encoding: gzip" -I https://www.amazon.com/
If you see in the result "Content-Encoding: gzip", then the result is sent gzipped (make sure to you a request without redirects)

Hope this helps
-gn

[1] https://naviserver.sourceforge.io/n/naviserver/files/ns_return.html
[2] https://bitbucket.org/naviserver/naviserver/src/default/openacs-config.tcl

Collapse
Posted by Gaven Eogan on
Thanks for the information. It was more or less what I already had tried to do. But it appears there is some problem with the Windows version.

I installed the latest Windows naviserver to my local machine and added the "compressenable" parameter (so that the server should compress HTML files coming back to browser) but it doesn't do anything. HTML (from TCL/ADP pages) is still coming back uncompressed.

No matter what I do, I can't get it to do the dynamic compression (of TCL/ADP files). As far as I can tell, this is done by simply adding the extra "compressenable" parameter to config file. We are not so much interested in the static compression of other files right now, so don't need the fastpath stuff.

I can see in source code of naviserver in server.c where it uses this parameter and dumps an ns_log line warning of "init server %s: compress is enabled, but no zlib support built in" is there is some zlib problem, but this line doesn't appear in log so I guess there is no problem with zlib.

I would like to be able to add more log lines myself and recomplile to test, but I am not familiar enough with Visual Studio to do this.

Has anyone here used the Windows version of Naviserver and got any compression working?

Concerning:
I can see in source code of NaviServer in server.c where it uses this parameter and dumps an ns_log line warning of "init server %s: compress is enabled, but no zlib support built in"
notice that the change [1] was added on Oct 17, while the last build of Maurizio is from Oct 12th. So, the absence of the message is in your case no indicator.

[1] https://bitbucket.org/naviserver/naviserver/commits/5c30b9aafc0b240d85627addd97277d417d3b246

Collapse
Posted by Gaven Eogan on
Oh right... I understand. Maybe there could be more logging to show case of everything working as expected (compress enabled and zlib OK), instead of only on error.

So any idea why the compress parameter appears to be getting ignored? Its impossible for me to tell without adding extra logging myself which I don't have the ability to do.

The build machinery on *nix/macOS systems is quite different to win* machines, and for these there are at least 3 way of building (mingw, nmake, native windows tool). For the latter variants, a maintainer has to add compile flags manually, which is not necessary in the *nix family. So, there is a certain possibility that these flags were never set in Maurizios windows build. Have you tried to contact him?
Collapse
Posted by Gaven Eogan on
No I did not contact him. I don't really know best way to do that.
Why are you not contacting him? You get probably faster a working solution than performing some guessing work here.
Dear Gaven,
the previous versions of my distribution (up to 4.4.9) did not support compression.
I'm currently uploading (it will be available in few hours) version 4.5.0 with compression support included (via zlib 1.2.11).
The thing compiles and links properly and at startup does not print any message of the type "no zlib support built in".
But I did not have the time to do proper testing on this functionality.

I hope it helps, even though I personally believe that compression support is not going to improve dramatically performances.
Maurizio

Collapse
Posted by Gaven Eogan on
Thanks Maurizio... I will be trying this out today to see if it works... we will still need to do a custom version of this for our system though... can't use your default build in our product.
Yes it may not speed it up much, but its better than none I think and is really an industry standard to to transfer compression of HTML and other easily compressible formats.

ps. the "no zlib support built in" message is something Gustaf added in later source version as he mentioned above. It is probably not in the source you used.

The version built by Maurizio is based on the source code of "31st of October 2019" (he takes always the tip version of NaviServer).

A few sentences above confuse me: Why can't you use the version of Maurizio in your product? You are mentioning a "custom built" the first time: How did you usually build the custom version in your product in the past? Why not redoing your custom build process with recent versions of NaviServer?

Collapse
Posted by Gaven Eogan on
At the time there were a few bugs we needed that were not in the standard version so we had to fix them in custom version. I think at the time the log rolling was not working. Not sure what else. I wasn't involved in this part of system at that time. There was also other things added. We worked with Maurizio who gave us the build environment. I heard that we didn't need to do a newer build as there were not many changes. But now we do (need to update a lot of things). Yes we will be rebuilding latest version into our own custom (if still needed - have to check this)
Dear Gaven,
1. As Gustaf already mentioned, I always include in my distribution the tip version the Naviserver.
2. I would not have written what I wrote without having first checked that the statement printing "no zlib support built in" is indeed in the codebase.

Best regards,
Maurizio

Dear all,
this is just to mention that brotli executable is now (from version 4.5.1) included in my distribution.
Thank you,
Maurizio
Collapse
Posted by Gaven Eogan on
I managed to get dynamic compression (ADP/TCL HTML pages) working using the latest version of your Naviserver on our system.
To do this I had to remove one piece of custom code from our code to get it to startup without error.
I also had to remove the section of config that enabled SSL (module/nsssl). We normally run systems with NSSSL enabled.
If I don't do this, I get error during startup:

"Warning: modules nsssl requires a version of NaviServer built with OpenSSL"
Error: modload: c:/naviserver/bin/nsssl.dll: Ns_ModuleInit returned:
Fatal: modload: failed to load module 'c:/naviserver/bin/nsssl.dl

Is there some reason why your version of Naviserver does not have OpenSSL enabled?

Also I can't static compression to do anything even though it is turned on (turning on settings in Fastpath section to enable Brotli or gzip)

ns_param gzip_static true ;# check for static gzip; default: false
ns_param gzip_refresh true ;# refresh stale .gz files on the fly using ::ns_gzipfile
ns_param gzip_cmd "c:/naviserver/bin/gzip -9" ;# use for re-compressing

ns_param brotli_static true ;# check for static brotli files; default: false
ns_param brotli_refresh true ;# refresh stale .br files on the fly using ::ns_brotlifile
ns_param brotli_cmd "c:/naviserver/bin/brotli -f -Z" ;# use for re-compressing

Unless i am doing something wrong... none of the CSS/JS files are ever compressed on server and sent to browser.

I really don't understand why there are so many problems and things not working with your version of Naviserver. Are they all intentionally removed?

Dear Gaven,
1. NSSL support is not turned on by default. You can adjust the configuration in the provided solution file to turn it on.
2. I never used the compression before. The testing is all up to you (did you verify on the first place that commands like "c:/naviserver/bin/brotli" actually work in your environment? did you verify if you have to use the *.exe extension?)
You get something for free, you want it to exactly meet your requirements and then you complain if it doesn't...
😊 😊 😊
In version 4.5.2 I believe I have activated the NSSL support, but for the compression part you're on your own...

Thank you,
Maurizio