Forum OpenACS Q&A: Why is OpenACS becoming so slow?

Collapse
Posted by Rildo Pragana on
In my intranet, benchmarking OpenACS against Zope (with the Apache "ab" tool), gives a speed advantage to Zope about 200%.  The machine running both servers is the same, an AMD Duron 1.2 MHz, with 512MB RAM.

BTW, I have also let postgres use plenty of shmem (shared_buffers = 15200, sort_mem = 32168, and others setups as suggested in the installion guide)

Is OpenACS becoming so slow? Or, how do I fine tune it to become more responsible?

Collapse
Posted by Jade Rubick on
It's impossible to say without knowing more about your installation.

Which applications are you using?

Are you doing vacuum analyze on postgres?

Are you serving static or dynamic pages?

What permissions checks are going on?

Etc..

Collapse
Posted by Rildo Pragana on
Hi Jade,

Ok, here's is the site map:

/    Main Site    ACS Subsite
  (-) Rildo/    subsite    ACS Subsite
          1/    cal    Calendar
          2/    e-commerce    E-Commerce
          3/    news    News
          4/    survey    Survey
      (+) 5/    blog    Lars Blogger
          6/    photos    Photo Album
          7/    postcard    postcard
      acs-admin/    Site-Wide Administration    Site-Wide Administration
      acs-content-repository/    ACS Content Repository    ACS Content Repository
      acs-service-contract/    ACS Service Contract    ACS Service Contract
      api-doc/    ACS API Browser    ACS API Browser
      attachments/    Attachments    Attachments
      doc/    ACS Core Documents    ACS Core Documents
      forums/    forums    Forums
      notifications/    Notifications    Notifications
      rss-support/    RSS Support    RSS Support

Most pages are dynamic, but the same happens with Plone (Zope's CMS). The permissions are most what came after I installed. I have vacuumed the service0 database just before the measurement.

Thanks for your time answering my question!

best regards,
Rildo Pragana

Collapse
Posted by Jonathan Ellis on
are you comparing apples to apples?  what exactly are you using ab to benchmark with oacs and zope?
Collapse
Posted by Rildo Pragana on
The speed difference is very visible, no matter what content you try, but of course it is subjective. Most pages served from OACS take between 100 and 700 ms. When I turn on developer support, I may see half that time are spent in posgresql queries.

Maybe I expected too much from OACS. I used to program in tcl for the last 7 years, and that took me on the way to OACS. But python is not unfriendly too. (the only thing I really hate is Java!)

Well, for the project I have in mind, Apache+mod_rivet will do, both in speed and development time, so please don't bother with my previous comments.
Thanks for your prompt answers, I really apreciate!

Collapse
Posted by Jonathan Ellis on
I like python quite a bit.  If the zope sql api didn't suck rocks I'd be tempted to play with it some more.
Collapse
Posted by Peter Marklund on
Rildo,
I know a few of the OpenACS admin pages are quite slow. For example there are pages in the APM that perform hundreds and up to a thousand queries. However, you usually expect high load on the user pages only, so that's where performance tuning efforts should be focused. Can you maybe list the user pages that were particularly slow and the request times you got for these pages?

Thanks!

Collapse
Posted by Frank N. on
Rildo,

Some time ago, around Zope 2.3.x, I worked quite a bit with Zope, even deploying a few sites with it, one of which is still up. I also did a fair amount of benchmarking, and, given my recent experiences with OpenACS as well, I came to several conclusions when comparing the two:

*) If you only run a single HTTP request from ab at a time, then Zope will do nicely, and the speed will be OK for the hardware. However at least at the time when I used Zope, the HTTP server used the tried-but-inefficient One Giant Lock(TM) to achieve multithreading somewhere in its networking core. This resulted in an extremely bad scalability of the web server with dropped connections and unanswered requests as the result, plus impossibly long response times.

It would be interesting to see what results you would get if you try to run with, say, 5 simultaneous requests with a recent Zope to see if things has improved, ie. 'ab -c 5 -t 60 http://...'; or similar.

It is well known that AOLserver scales extremely well in this regard, something that is even more important than the maximum, fully serialized request speed that can be achieved. If you need to push the server due to many users of your site, then they will almost certainly not queue up their requests in a nice, orderly fashion. Just now I tried kicking my 2.5GHz/512MB P4 running FreeBSD 4.8/OpenACS HEAD. For a single, concurrent request of the main default page (excerpt):

Concurrency Level:      1
Time taken for tests:  60.126 seconds
Complete requests:      417
Failed requests:        0
Broken pipe errors:    0
Requests per second:    6.94 [#/sec] (mean)
Time per request:      144.19 [ms] (mean, across all concurrent requests)
Percentage of the requests served within a certain time (ms)
  50%    141
  98%    159
  99%    232
100%    280 (last request)

Not terribly impressive... Let's try that with 10 concurrent threads:

Concurrency Level:      10
Time taken for tests:  60.126 seconds
Complete requests:      418
Failed requests:        0
Broken pipe errors:    0
Requests per second:    6.95 [#/sec] (mean)
Time per request:      143.84 [ms] (mean, across all concurrent requests)
Percentage of the requests served within a certain time (ms)
  50%  1417
  98%  1805
  99%  1833
100%  4284 (last request)

Ie. at this concurrency level AOLserver/OpenACS scales linearily. OK, enough fooling around. Your site was just mentioned on slashdot + in prime time on national TV. 100 (one hundred) simultaneous requests:

Concurrency Level:      100
Time taken for tests:  60.036 seconds
Complete requests:      413
Failed requests:        0
Broken pipe errors:    0
Requests per second:    6.63 [#/sec] (mean)
Time per request:      145.37 [ms] (mean, across all concurrent requests)
Percentage of the requests served within a certain time (ms)
  50%  1599
  66%  1650
  75%  4584
  98%  7948
  99%  14272
100%  35882 (last request)

The server slows down a whopping 3.5%, two thirds of our visitors are still served within 1.7s and noone is left out in the cold.

How does Zope cope with 100 simultaneous requests these days? The last time I looked at the Plone tuning guide, it recommended proxying Zope behind Apache or Squid. This effectively circumvents the threading issue, as the requests are serialized from the point of view of Zope.

*) Updating the ZDB is very inefficient due to the undo buffer and the one-big-file DB organization. Try adding a hit counter to a page and hit it with a few simultaneous requests.

*) The database/users/permissioning model of Zope is basic, effectively a tree structure with inheritance. If your application calls for this very structure, like a library of articles, then Zope/Plone will do nicely.

On the other hand the database model of OpenACS is extremely complicated in comparison as it is prepared to deal with very complicated setups, like people organizing themselves in weird and wonderful ways.

Whenever Plone or OpenACS serves a page, they both have to hit the DB to check if this user may see the page. OpenACS is at an unfair disadvantage here for simple page requests, as it may have to do a comparatively more complex query for this basic task. It should be added here though, that on the tests I just ran, PG only consumed around 12-14% of the available CPU time, so this point is not the major reason for the speed difference.

It would be interesting to check how fast a Zope/Plone application with a core complexity of the scale of OpenACS is. No such application is available to the general public as far as I am aware though.

*) Plone has a policy of agressively caching bits of rendered pages in RAM. This speeds up the apparent render time quite a bit. The default pages of OpenACS doesn't do anything similar AFAIK (though the functionality to cache things in RAM is there), ie. by default every part of every page is parsed and rendered in full for every page request.

To sum up, then Zope/Plone will do nicely if whatever you want to do is compatible with the core object oriented database design, and you don't need very high concurrency at the level of the DB. Additionally it will be easy and simple to set up to boot.

But on the other hand if you will be attempting to do something non-trivial, like building a complex, database driven multi-user application and/or you really need to serve more than a trivial number of requests per second, then OpenACS is the way to go. I don't think there is anything comparable out there, and I did look. That is why I am here.

Frank.

Collapse
Posted by Frank N. on
I need to clarify a bit of what I wrote. In the '-c 100' test AOLserver does of course not serve more requests per second than the 6.something. What the numbers show is that it will, on the avarage, not slow down or crash even if hit by a load surge above what it can cope with from a pure requests-per-second point of view. Two thirds of 100*60 requests cannot be answered per minute, of course.

If you need to serve 100 pages per second sustained over longer periods of time, then you will still need to either use an external proxy, implement caching in RAM within AOLserver or add more AOLserver instances.