Forum OpenACS Development: How to get gzip/brotli transfer compression working?

Request notifications

Is there some up-to-date information about what exactly to do to get gzip or better yet brotli compression working in Naviserver/OpenACS for normal web content (HTML from TCL/ADP, CSS, JS)?

We are in the ridiculous situation right now of serving eg. 2.5Mb HTML sections of a TCL/ADP page as a 2.5MB uncompressed transfer. Instead of that being compressed down to a fraction of the size before transmission (https://tools.paulcalvano.com/compression.php)

The only mention of it I can find is an old post here from 2013 (https://openacs.org/forums/message-view?message_id=1205589) but it is a confusing because it appears to be about previous AOLServer setup/config and also the newer Naviserver is talking about large static files which we don't care about for now. I tried adding the various parameters to config file but it didn't work. Is the "preauth filter" method still the only way? I tried adding that but that code appears to be obsolete and non working too.

What is the current correct way to enable brotli transfer compression on Windows based Naviserver/OpenACS? (assume latest https://www.spazioit.com/pages_en/sol_inf_en/windows-openacs_en/) Any instructions/examples anywhere?

Brotli compression is provided by NaviServer for static content. Activate and configure it in the "fastpath" section of naviserver [1], see also the sample config file at [2]

gzip compression is implemented in NaviServer for static content and dynamic content. Activating gzip compression for static content works exactly like for brotli static compression (see e.g. [2]), dynamic content is compressed, when "compressenable" is set as a parameter in the per server settings in the config file (see as well [2]).

For compression of static content, external programs are used (as configured in the config file). For dynamic content, NaviServer has to be compiled with gzip support (which is the case on all *nix platforms). Maurizios download page mentions "zlib.1,2,8), therefore i would think, that it contains gzip support.

Try it with a command like the following

curl -H "Accept-Encoding: gzip" -I https://www.amazon.com/
If you see in the result "Content-Encoding: gzip", then the result is sent gzipped (make sure to you a request without redirects)

Hope this helps
-gn

[1] https://naviserver.sourceforge.io/n/naviserver/files/ns_return.html
[2] https://bitbucket.org/naviserver/naviserver/src/default/openacs-config.tcl

Collapse
Posted by Gaven Eogan on
Thanks for the information. It was more or less what I already had tried to do. But it appears there is some problem with the Windows version.

I installed the latest Windows naviserver to my local machine and added the "compressenable" parameter (so that the server should compress HTML files coming back to browser) but it doesn't do anything. HTML (from TCL/ADP pages) is still coming back uncompressed.

No matter what I do, I can't get it to do the dynamic compression (of TCL/ADP files). As far as I can tell, this is done by simply adding the extra "compressenable" parameter to config file. We are not so much interested in the static compression of other files right now, so don't need the fastpath stuff.

I can see in source code of naviserver in server.c where it uses this parameter and dumps an ns_log line warning of "init server %s: compress is enabled, but no zlib support built in" is there is some zlib problem, but this line doesn't appear in log so I guess there is no problem with zlib.

I would like to be able to add more log lines myself and recomplile to test, but I am not familiar enough with Visual Studio to do this.

Has anyone here used the Windows version of Naviserver and got any compression working?

Concerning:
I can see in source code of NaviServer in server.c where it uses this parameter and dumps an ns_log line warning of "init server %s: compress is enabled, but no zlib support built in"
notice that the change [1] was added on Oct 17, while the last build of Maurizio is from Oct 12th. So, the absence of the message is in your case no indicator.

[1] https://bitbucket.org/naviserver/naviserver/commits/5c30b9aafc0b240d85627addd97277d417d3b246

Collapse
Posted by Gaven Eogan on
Oh right... I understand. Maybe there could be more logging to show case of everything working as expected (compress enabled and zlib OK), instead of only on error.

So any idea why the compress parameter appears to be getting ignored? Its impossible for me to tell without adding extra logging myself which I don't have the ability to do.

The build machinery on *nix/macOS systems is quite different to win* machines, and for these there are at least 3 way of building (mingw, nmake, native windows tool). For the latter variants, a maintainer has to add compile flags manually, which is not necessary in the *nix family. So, there is a certain possibility that these flags were never set in Maurizios windows build. Have you tried to contact him?
Collapse
Posted by Gaven Eogan on
No I did not contact him. I don't really know best way to do that.
Why are you not contacting him? You get probably faster a working solution than performing some guessing work here.
Dear Gaven,
the previous versions of my distribution (up to 4.4.9) did not support compression.
I'm currently uploading (it will be available in few hours) version 4.5.0 with compression support included (via zlib 1.2.11).
The thing compiles and links properly and at startup does not print any message of the type "no zlib support built in".
But I did not have the time to do proper testing on this functionality.

I hope it helps, even though I personally believe that compression support is not going to improve dramatically performances.
Maurizio

Collapse
Posted by Gaven Eogan on
Thanks Maurizio... I will be trying this out today to see if it works... we will still need to do a custom version of this for our system though... can't use your default build in our product.
Yes it may not speed it up much, but its better than none I think and is really an industry standard to to transfer compression of HTML and other easily compressible formats.

ps. the "no zlib support built in" message is something Gustaf added in later source version as he mentioned above. It is probably not in the source you used.

The version built by Maurizio is based on the source code of "31st of October 2019" (he takes always the tip version of NaviServer).

A few sentences above confuse me: Why can't you use the version of Maurizio in your product? You are mentioning a "custom built" the first time: How did you usually build the custom version in your product in the past? Why not redoing your custom build process with recent versions of NaviServer?

Collapse
Posted by Gaven Eogan on
At the time there were a few bugs we needed that were not in the standard version so we had to fix them in custom version. I think at the time the log rolling was not working. Not sure what else. I wasn't involved in this part of system at that time. There was also other things added. We worked with Maurizio who gave us the build environment. I heard that we didn't need to do a newer build as there were not many changes. But now we do (need to update a lot of things). Yes we will be rebuilding latest version into our own custom (if still needed - have to check this)
Dear Gaven,
1. As Gustaf already mentioned, I always include in my distribution the tip version the Naviserver.
2. I would not have written what I wrote without having first checked that the statement printing "no zlib support built in" is indeed in the codebase.

Best regards,
Maurizio