Forum OpenACS Development: Get number of visitors currently browsing the site

Is there something ready to use in aolserver or oacs to get nuber of visitors (registered users or not) currently visiting the web site?

there is /shared/whos-online to see how many peaple are currently logged-in but I havent' seen anything about visitors.

(Remember HTTP is stateless so there's really no such thing as "currently", there's only "in the last 10 minutes", or whatever time period you choose to define.)

For logged in users, yes definitely, see these threads from Oct. and March 2003. I think there was also some more recent discussion of that stuff just in the last week or two, but I can't find it right now.

For non-logged-in users, dunno, the above might track number of user_id 0 hits, or it might not. (Try it and see.) If not, what you want is to instead look in the AOLserver access log. It is easy to configure AOLserver and OpenACS to log the OpenACS user_id to the AOLserver access log, so to check unregistered users you can just grep for only user_id 0 hits in the log.

Keep in mind that the web pages showing the "who's online in the last 10 minutes" sort of info are intended for transient human use, not for any sort of load tracking over time. If you want to graph load or track which users have been clicking on what over time (clickstream data warehousing, etc.), then post-processing the AOLserver access log is the right way to go.

Collapse
Posted by Malte Sussdorff on
You could cheat your way by looking at the access log and get a list of unique IP addresses for the last x minutes. If grabbing the access log is too much of a burden, write a small enhancement to the request processor that stores the ip addresses on each request in an NSV and clean that up from time to time.
Collapse
Posted by Lars Pind on
Hi Jay

If you look at packages/acs-tcl/tcl/whos-online-procs.tcl, you'll find this comment:

# TODO: Record the IP address from [ad_conn peeraddr]

My plan was to use this to add exactly this feature, but I didn't have time.

If you'd implement this for us, that would be fantastic!

/Lars

Collapse
Posted by Tilmann Singer on
In my experience uniqueness of the ip address is very unreliable. There are providers where all accesses appear to come from the same ip address, others where the address of a single user changes from request to request. I wouldn't use it it for anything besides vague estimates.
Collapse
Posted by Lars Pind on
Yes, IP is unreliable. Any better suggestions?

Session_id, maybe?

/Lars

I guess that the problem with session_id is that it will also count the spiders, etc (I mean, everything accessing the site).

Something like:

set ten_min_ago [expr [ns_time] - (10*60)]
set sql_query "select count(distinct session_id) 
    as result from sec_session_properties where last_hit > :ten_min_ago"
db_1row select_sessions $sql_query
should give you some sessions. The only issue that I see is that, at least in OpenACS 4.6.3 (that's what we are using at the moment) you don't have an index on sec_session_properties(last_hit) so this query is quite inefficient (but I am not sure than adding an index is also a good solution, this table is touched a lot -for every hit, right?-).

Just my 2 cents,

/B

Collapse
Posted by Jade Rubick on
Perhaps we could cache the results every minute, and average them or something? The number doesn't have to be 100% accurate, but it should give a general idea of how many people are using the site.
I tried Bruno's query and find that I get 0 sessions even if there is an active member session.
Collapse
Posted by gustaf neumann on
I have improved our controling thread package described in
http://www.openacs.org/forums/message-view?message%5fid=203277
and among other things, it counts users by oacs-ids and by
their peer addresses.

It keeps these statistics for the last
10 minutes and for the last 24 hours, so one can get an idea
how many people (or peer addresses) used the site during the
last day as well. The IP-addresses are certainly not a
perfect measure, but still a good approaximization.

New in version 0.3 of the controlling thread package:
  - fixed a race condition during startup and heavy request traffic
    (could result in several controlling threads running)
  - keep peer address, timestamp and user activity
  - computes users in last 24 hours
  - adapted version of whos-online (shows activity and
    peer for sys admins, sort options)

You can see a sample screen shot of whos-online with
the user names blanked out here:
http://media.wu-wien.ac.at/download/whosonline.png

A sample display of the performance monitor is here:
http://media.wu-wien.ac.at/download/stat-2004-11-23a.htm

The updated version is in:
http://media.wu-wien.ac.at/download/throttle+stats0.3.tar.gz

best regards
-gustaf

Gustaf:

This is very interesting. Does the package require any changes to acs-core?

If not, would you consider uploading this code to openacs.org CVS?

I tried to make your code into an OpenACS package and stumbled upon some problems, which require changes to the ACS Core.

- The RP has to load .xotcl files as well
- thread::mutex does not work on my installation.

Maybe you can help me with the latter, so I could continue to make an openacs package out of it.

For thread::mutex to work, you presumably need to install the Tcl Threads Extension. The latest release supports AOLserver 4.x. Earlier releases support both AOLserver 3.x and 4.x.
Collapse
Posted by Gustaf Neumann on
Malte, the two xotcl files should not be loaded
by the RP, but during startup of the server.
if you make a "make install-aol" from the xotcl sources,
a file xotcl.tcl is installed that handles .xotcl files
in the modues/tcl/ directory.

yes, for the thread::mutex, Zoran's Tcl Thread Library
is needed (as indicated in the README file).
If you use the code on recent linux kernels,
make sure to use a recent tcl threads version (2.6),
earlier versions caused troubles at least with FC2.
i added the mutex lock just recently, it is just
needed for very busy sites, where the server receives
a high number of requests immediately after startup.
Before that we used the code for a long time without
noticing the problem.

We use the tcl thread library since a few years with
the aolserver, it never caused any troubles or
instabilities.

After providing the package, i added on a few more things:
- caching of the dns lookup. successful dns lookups are
already cached by the core, but the lookup for
non-resolveable names can be quite slow.
- we want to prevent user-id sharing to some degree.
Therefore i added some code to track whether a
uses accesses the site within a time window
simultanous from multiple sites. A single switch
happens frequently, multiple switches are suspicious.
- for measuring a users activity i added a exponential
fitted value. This might lead to a simpler and better
way to block overactive clients..
This stuff is not finished yet...

season greetings
-gustaf

Collapse
Posted by Orzenil Silva Junior on
Hi Gustaf,

I'm trying load aolserver4 with libthread2.6.so from tcl threads 2.6 and these errors are occurring (aolserver error log):

[30/Dec/2004:15:41:43][8945.1024][-main-] Notice: modload: loading '/usr/local/aolserver/lib/thread2.6/libthread2.6.so'
[30/Dec/2004:15:41:43][8945.1024][-main-] Warning: modload: could not find Ns_ModuleInit in /usr/local/aolserver/lib/thread2.6/libthread2.6.so
[30/Dec/2004:15:41:43][8945.1024][-main-] Fatal: modload: failed to load module '/usr/local/aolserver/lib/thread2.6/libthread2.6.so'

threads2.6 was installed as a loadable module for the AOLserver with this config options:

aoldir="/usr/local/aolserver"
../configure --enable-threads --with-aolserver=$aoldir \
--prefix=$aoldir --exec-prefix=$aoldir

In my aolserver config.tcl I have:

ns_param libthread ${homedir}/lib/thread2.6/libthread2.6.so

Maybe this error is related with this warning in threads CONFIG:

#
# AOLserver uses its own package loading mechanism.
# To load, just do "ns_eval package require Thread"
# at the AOLserver startup or later from any thread.
#

but when i add "ns_eval package require Thread" to aolserver4 config.tcl the server doesnt load and doesnt write to log file.

What am i missing?

Collapse
Posted by Gustaf Neumann on
Orzenil,

when a c module is loaded via the config-file,
it calls after loading Ns_ModuleInit. This was the only
way to load a c module for aolserver3. For aolserver4,
this is less important, not to say deprecated, since
one should be able to use a package require. So,
proceeding as malte suggests should be ok, as long the
package require happens "early engough".

I am still puzzeled about you messages, since
when i do the same configure, -DNS_AOLSERVER=1 is
added to the list of defines, and Ns_ModuleInit
is part of the .so file. (nm libthread2.6.so|grep Module)

-gustaf

Gustaf,

It is very strange! The only way i could have libthread2.6 compiled with Ns_ModuleInit was changing by hand generic/aolstub.cpp to

/* #ifdef NS_AOLSERVER */
#include "/usr/local/aolserver/include/ns.h"

althroug my CONFIG was passing parameter "--with-aolserver" the var NS_AOLSERVER was not setting. I do not know why.

This way i could have as a result from "nm libthread2.6.so|grep Module":

00001654 T Ns_ModuleInit
00011b6c D Ns_ModuleVersion

and libthread2.6.so module was loaded correct by aolserver when declaring "ns_param libthread ${homedir}/lib/thread2.6/libthread2.6.so" in modules section.

Problem now is with throttleThread :-(

When I put your modules/tcl/*xotcl files in my aolserver dir modules/tcl/ and reload aolserver thats ok. Server is up but every page i browse nothing happens and this message is written in aolserver log errors:

"throttleThread do ::throttle check 200.181.95.53 200.181.95.53"
("eval" body line 1)
invoked from within
"eval [my server] do [self] $args"
(procedure "do" line 3)
::throttle ::THREAD::Client->do
invoked from within
"my do check $requestor $pa"
(procedure "check" line 11)
::throttle->check
invoked from within
"my check"
(procedure "postauth" line 4)
::throttle->postauth
invoked from within
"throttle postauth"

(xotcl 1.3.4 is installed with make-aol; xotcl.tcl is in $aoldir/modules/tcl/ and *xotcl files are sourced because when i remove them the website pages display fine without any error):

Maybe errors in throttle_mod.xotcl script (as Malte observed)?

Thank you for your patience.

Collapse
Posted by Gustaf Neumann on
To shorten this thread, i hacked together my first apm
file
http://media.wu-wien.ac.at/download/xotcl-stuff-0.3.apm
that should work for some more people. It still assumes
that XOTcl and the Tcl Threads 2.6 are installed in place,
the two .xotcl files are changed into -procs.tcl files.

yes, there was a problem with the old version and the standard RP,
it was just working with plain AS and the
modified RP of learn@wu. I just installed on my notebook
a fresh version of OACS, and it seems to work fine....

the name is not polished, but it indicates that this
package contains different things (xotcl-thread support,
request monitor with throttleling support, value-added whos-online).
At least the first one should be moved into
a different package...

happy new year
-gustaf

Collapse
Posted by Jade Rubick on
Gustaf: are you willing to release this package in CVS?
Collapse
Posted by Gustaf Neumann on
Sure. First it should run without much tweaking,
then it should be splitted into two parts (xotcl
tread handling should not be part of a request
monitor), then adding some configuration options
and finally put it into cvs... and well, the docu
should be improved, and we have to find a way to
integrate the docs with the api browsers.

For the first part, Zoran has released a version
of libthread 2.6 that has the configuration
problems fixed (called 2.6.1). With that
the thread issues like the ones from Orzenil
should be resolved (got a positive feedback from
him).

http://prdownloads.sourceforge.net/tcl/

There is a version with improved functionality
(e.g. recording site switching, user tracking):

http://media.wu-wien.ac.at/download/xotcl-stuff-0.4.apm

Best regards
-gustaf

Collapse
Posted by Orzenil Silva Junior on
Jade: after some coordination with Gustaf he helps me to get xotcl-stuff package working. I think features included in xotcl-stuff-0.4 are very useful and Gustaf is working on it to get it better finding a way to document xotcl classes/objects/methods similar to ad_proc and add parameters to the package.

Thanks, Gustaf.

I am working now in a brief tutorial about how Gustaf help me to install requirements for xotcl-stuff and soon i will post it in this forum. Maybe other people could try it.

\Orzenil

Orzenil,

i am still puzzled about the -DNS_AOLSERVER.
it looks, as we are working with different versions.
my version is from sept 2004 or so from CVS, the
newest update is from aug. 2004. Can you check yours
as well?

thanks, -gustaf

localhost:~/import/thread2.6> find . -type f -exec fgrep RCS: \{} \;
...
* RCS: @(#) $Id: threadPoolCmd.c,v 1.26 2004/08/02 20:27:11 vasiljevic Exp $
...

Collapse
Posted by Orzenil Silva Junior on
Gustaf,

My threadPoolCmd.c version is newer than v 1.26

* RCS: @(#) $Id: threadPoolCmd.c,v 1.28 2004/11/27 06:11:02 vasiljevic Exp $

i downloaded threads2.6 from tcl.sf.net: http://prdownloads.sourceforge.net/tcl/thread2.6.tar.gz?download

following changes were introducing in v 1.28 over v 1.26 (directly from cvs):

http://cvs.sourceforge.net/viewcvs.py/tcl/thread/generic/threadPoolCmd.c?r1=1.28&r2=1.26&diff_format=u

More info about my environment:
debian gnu/linux ( linux kernel: 2.6.5-1-686)
tcl8.4.5
aolserver4.0.0r2

Unfortunately xotcl-stuff still doesnt work in my openacs-5-1-3 + dotlrn-2-1b install. Errors:

- http://servername/status/index

invalid command name "seconds"
while executing
"seconds set trend"
invoked from within
"thread::send $tid "$o $args""
(procedure "do" line 37)
::throttleThread ::THREAD->do
invoked from within
"throttleThread do $t set trend"
(procedure "counterTable" line 4)
invoked from within
"counterTable Views [list seconds Second minutes Minute hours Hour]"
invoked from within
"set views_trend [counterTable Views [list seconds Second minutes Minute hours Hour]]"
("uplevel" body line 102)
invoked from within
"uplevel {

# present usage statistics, active users, etc

...

- http://servername:8000/status/whos-online

invalid command name "Users"
while executing
"Users totalInfo"
invoked from within
"thread::send $tid "$o $args""
(procedure "do" line 37)
::throttleThread ::THREAD->do
invoked from within
"throttleThread do Users totalInfo"
invoked from within
"foreach element [throttleThread do Users totalInfo] {
foreach {user_id pa timestamp hits} $element break
if {[string first . $user_id] > 0} {
..."
("uplevel" body line 49)
invoked from within
"uplevel {
ad_page_contract {
Displays who's currently online

thanks,

Happy new year

.: My wish for 2005: learn xotcl ("exotickle" :-) )

Gustaf, the reason I talk about the RP is the fact that I want to take your code and make a decent OpenACS Package out of it, which can be installed using the APM (taking into account the prerequisits).

I too experience the problem of Orzenil, though I managed to get the thread::mutex command to work by just adding the package require in to the 0-acs-init.tcl. And this is the way it should be with AOLserver4 (using package require instead of loading library files manually).

Sadly though, .xotcl files are sourced before the .tcl files in /tcl, so I'm stuck again, but I managed to get this to work by adding the packagerequire to your thread_mod.xotcl right at the beginning.

So far so good, but now I'm stuck as you have a typo in your code due to the fact that you seem to have forgotten to open a paranthesis correctly "} persistent" is the culprit.

And now I got a command not found ::xotcl:: and after manually copying the code of xotcl.tcl over to my webserver I get "invalid command name throttle".

So, assuming that you have a new version anyway, I will give up at this point and look forward to see your code in the new year.

Malte, i see. look into modules/tcl/xotcl.tcl to understand,
what happens during loading. The .xotcl files are not
sourced after the .tcl files in general, since they are sourced by a .tcl file.
There is no big magic involved. If you have xotcl.tcl still
under modules, and you load libthread at some later time,
and you source thread_mod.xotcl before that, it won't work.

It should be possible to source xotcl.tcl in 0-acs-init.tcl
after the "package require Thread" (if this is the right
place, and the files there are ns_evaled). It should also
be possible to add the "package require Thread" to the
begin of thread_mod.xotcl (with the disadvantage that it
will cause problems with aolserver 3.*).

i can't diagnostic the other error messages. i have also
downgraded my notebook to the files of the 0.3-tar file,
but i did not get the parenthesis mismatch...

-gustaf

Collapse
Posted by Gustaf Neumann on
Hi folks,

Here are two .apm files for OpenACS
together with a patch to make the api-browser
xotcl-aware. I have tested it with a fresh
version of OpenACS 5.1.4, we have tested it with
dotlrn in our tailored configuration in learn@wu,
Orzenil got it runnig with dotlrn, neophythos uses
it with openacs. The biggest obstacles are now
howfully gone, interested people might try it.

If you have some earlier versions installed on
your system, please remove the *.xotcl files from
the aolserver/modules/tcl directory.

best regards
-gustaf neumann

PS: The how-to documentation below was written
mostly by Orzenil and explains the installation
from scratch.
===================================================
XOTcl Core 0.6

XOTcl-Core is a component containing core functionality for OpenACS
applications using XOTcl. It includes support for thread handling
(persistent and volatile threads) and provides an interface for
documenting XOTcl objects, classes and methods integrated with the
api-browser of OpenACS (supporting an XOTcl version of ad_proc for
object and class methods). The package provides as well an XOTcl
Object and Class browser.

XOTcl Request Monitor 0.10

This package contains a request monitor for OpenACS applications. It
computes performance summary information such as requests/views per
seconds, average response time, number of users connected,
etc. Furthermore it can block overactive users (e.g. automated
web-bots mirroring the site). It provides as well some user tracking
(such as whos-online) with activity measures, it tracks switching of
IP-addresses from users and provides request tracking per user for the
monitored time window. It contains as well overall url statistics with
performance measures. This package depends on XOTcl-Core.

INSTALLATION:

1. Install Thread Extension:

First of all, get tread2.6.1. There is a small bug in thread2.6.0
since it does not use the definitions from aolserver.m4 (see
Changelog for thread2.6.1 in
http://cvs.sourceforge.net/viewcvs.py/tcl/thread/ChangeLog?rev=1.144&only_with_tag=thread-2-6-1&view=markup).

a) download thread2.6.1 from http

http://sourceforge.net/project/showfiles.php?group_id=10894&package_id=10438&release_id=294044

from cvs

# cvs -z3 -d:pserver:mailto:anonymous@cvs.sourceforge.net:/cvsroot/tcl co -r thread-2-6-1 thread

untar it and go to you platform specific dir

Build for unix-like systems:

# cd unix
# ../configure --enable-threads \
--prefix=/usr/local/aolserver \
--exec-prefix=/usr/local/aolserver \
--with-aolserver=/usr/local/aolserver

use --prefix --exec-prefix --with-aolserver with path for
aolserver4 installed dir.
If you're using aolserver4 and tcl8.4 binaries from debian try this:
install aolserver4-dev:

#apt-get install aolserver4-dev

create symlink to includes:
# ln -s /usr/include/aolserver4 /usr/lib/aolserver4/include
use instead:

# ../configure --enable-threads \
--prefix=/usr/lib/aolserver4 \
--exec-prefix=/usr/lib/aolserver4 \
--with-aolserver=/usr/lib/aolserver4 \
--with-tcl=/usr/lib/tcl8.4

# make

check in output if there is following definition: -DNS_AOLSERVER=1

# make install

that's all. You should now have installed libthread2.6.1.so (check
for $aolserverdir/lib/thread2.6.1/libthread2.6.1.so)


b) adjusting config.tcl for libthread service

b1) open config.tcl (/var/lib/aolserver/${yourservice}/etc/config.tcl)
b2) look for modules section: ns_section ns/server/${server}/modules
b3) add libthread to modules section:

ns_param libthread ${homedir}/lib/thread2.6.1/libthread2.6.1.so

b4) restart the aolserver and check the error log, whether
libthread2.6.1 was loaded successfully

2. Get and install XOTcl:

a) Get xotcl from http://www.xotcl.org

The current version of XOTcl is 1.3.5. XOTcl (pronounced
exotickle) is a highly flexible objected oriented extension for
Tcl, which is now included in the major Tcl distributions
(ActiveTcl and Mac OSX Aqua). Read about XOTcl motivations and
feature from http://media.wu-wien.ac.at/whatIsXOTcl.html)

b) Install XOTcl:
Installing xotcl1.3.5 from source:

# wget http://media.wu-wien.ac.at/download/xotcl-1.3.5.tar.gz

The following commands can be used to compile XOTcl and install
it into the appropriate places of the specified aolserver:

# cd xotcl-1.3.5
# CC=gcc;export CC ./configure --enable-threads --enable-symbols \
--prefix=/usr/local/aolserver \
--with-tcl=/usr/src/tcl8.4.5/unix

Use appropriate paths for aolserver4 and your tcl version. If
you are running aolserver4 and tcl8.4 from debian binaries do
instead:

# CC=gcc; export CC ./configure --enable-threads \
--enable-symbols \
--prefix=/usr/lib/aolserver4 \
--with-tcl=/usr/lib/tcl8.4

# make
# make install-aol

After the "make install-aol" from the xotcl sources, a file
xotcl.tcl is installed that handles .xotcl files and XOTcl
serialization in the aolserver in the
${aolserverdir}/modules/tcl/ directory.

c) Restart your aolserver to load xotcl.

3. Install OpenACS-packages:

a) Get xotcl-core-0.6 and xotcl-request-monitor-0.10

# cd /var/tmp
# wget http://media.wu-wien.ac.at/download/xotcl-stuff-0.6.apm
# wget http://media.wu-wien.ac.at/download/xotcl-request-monitor-0.10.apm
# wget http://media.wu-wien.ac.at/download/acs-api-documentation-procs.patch

b) Extract apm files to your service packages directory

# cd /var/lib/aolserver/${yourservice}/packages/
# tar -xvzf /var/tmp/xotcl-core-0.6.apm
# tar -xvzf /var/tmp/xotcl-request-monitor-0.6.apm

c) Apply the patch to make the api-browser xotcl-aware

# patch -p 0 < /tmp/acs-api-documentation-procs.patch

d) Go to acs-admin/install (http://yourserver/acs-admin/install)
e) Choose install from local
f) Choose type Application
g) Install XOTcl Core and XOTcl Request Monitor (install from local)
XOTcl Core will be auto-mounted under /xotcl/
Request Monitor under /request-monitor/

h) Restart the server

# svc -t /service/${yourservice}

5) Now you can browse
http://yourserver/request-monitor
http://yourserver/xotcl

6) Enjoy it!

Collapse
Posted by gustaf neumann on
People notified me, that the first name for the file was
incorrect. xotcl-stuff should be replaced by xotcl-core.

The following is correct

# cd /var/tmp
# wget http://media.wu-wien.ac.at/download/xotcl-core-0.6.apm
# wget http://media.wu-wien.ac.at/download/xotcl-request-monitor-0.10.apm
# wget http://media.wu-wien.ac.at/download/acs-api-documentation-procs.patch

sorry for the confusion.

-gustaf