Forum OpenACS Q&A: Re: OpenACS clustering setup and how it relates to xotcl-core.

Hi Marty,

concerning your original question concerning clustering, I've done some work over the weekend. If one is using a recent version of NaviServer (at least 4.99.20), the parameter::* interface should be now cluster proof (added acs::clusterwide for cleaning and callbacks), and does no caching, when the cachingmode is set to "none" When the regression test is run with cachingmode set to "none". I've also cleaned up xo::clusterwide vs. acs::clusterwide (the former is gone). When running the regression test with cachingmode "none", some errors show up, but these are probably just due to the testing whether caching works (not sure about the client-property errors, but I am running right now out of time).

Concerning cluster modes: i see the following directions:

Classical OpenACS clustering: The basic idea is to setup intra-server talk to achieve cache-coherency. When some data is flushed in the cache of one server, it communicates this to the other servers in the cluster (via HTTP request). This is a multi-master mode, where every server talks to the same database. The NaviServers are symmetrical, one is responsible for scheduled procedures. When files are stored in the content repository, the content repository directory has to be shared between the instances. OpenACS 5.10 contains experimental support for storing content repository files in gridfs (which is horizontally scaling, based on mongodb). Some older packages bypassing the content-repository API have still to be adjusted to use the API.

However, the classical OpenACS clustering mode was never fully implemented, but might still work for certain applications. One could extend this clustering mode and make it "work as designed". Several cluster-flush operations are missing, probably some callbacks have to executed cluster-wide as well (e.g., change parameters), etc.

Known weaknesses: potential issue with scalability - when e.g., a single request requires e.g. 100 flush operations (not unusual), and e.g. 10 nodes are defined in the cluster, a single incoming request will cause 1000 additional HTTP requests for flushing. This can be probably improved via batching (bundle multiple flush requests in one HTTP request). Since the requests are HTTP requests, these will need very short timeouts, otherwise, these will block a while, when some nodes are down, slow, some connections might "hang" in the middle of a request, etc. After a network separation, full-cache-flush is required (but is not implemented).

Classical OpenACS clustering via UDP: This can reduce the clusterwide operation latency (no TCP setup times); one could consider multicast operations in the cluster to reduce the intra-server traffic substantially - the basic architecture will be the same.

Quick test for intra-server talk to a server on the same host (my notebook, localhost, mean value from 1000 requests)

   TCP 0.273ms
   UDP 0.171ms
   UDP 0.059ms (no reply)

So, flushing operations can be made faster by a factor of 4 or more by using UDP instead of TCP.

Clustering with no-caching: if nothing is cached (except maybe per-request caches), there will be no problem with cache-coherency, but there will be more load on the DB. The big advantage of no-caching is that the intra-server talk to preserve cache coherency is zero. For several types of applications, this will be the simplest setup.

Clustering via caching server: One could set up a system, where the caching is not done via a (central) caching server (e.g., REDIS; memcached, or a special purpose NaviServer) instead of NaviServer native caching. If there is a central cache, no intra-server talk is needed to achieve cache coherency, Disadvantages: NaviServer native caching is significantly more efficient than memcached/REDIS, and supports cache transactions (rollback, commit of cache settings). One could get this back via using one NaviServer instance as a purely caching instance ... but this requires deeper analysis. Some figures:

   NaviServer native cache performance: 0.33us == 0.00033ms
       time {ns_cache_eval ns:memoize 1 {set x 1}} 100000 

   REDIS
      https://k6.io/blog/benchmarking-redis-with-k6/

        iteration_duration...: avg=782.57µs min=67.35µs med=732.92µs max=15.86ms p(90)=1.1ms p(95)=1.3ms
        iterations...........: 506755 50660.636169/s
        redis_latency........: avg=764.8µs  min=0s      med=1ms     max=16ms    p(90)=1ms   p(95)=1ms

      100VUs
        iteration_duration...: avg=4.39ms min=46.8µs med=3.32ms max=87.24ms p(90)=9.5ms p(95)=12.51ms

When assuming 1ms latency for external cache, NaviServer's internal cache is about 3000x faster than going to an external cache, so, when intra-server talk is not overwhelming, one should get the best performance from OpenACS clustering via UDP - unless more options pop up.... but getting the majority of the OpenACS packages "cluster-safe" will take some time and debugging.