Forum OpenACS Development: Watching files, new solutions

Collapse
1: Watching files, new solutions
Posted by Malte Sussdorff on 11/30/03 02:05 PM
Carl opened a TIP (https://openacs.org/forums/message-view?message_id=147125) complaining about the inability to see, if the reload of file failed, when it is watched.

This brought me to think, that the whole idea of watching files is really not such a good thing as it is implemented now (have you ever watched the logfiles, with all the debug messages of reloaded files, that you do not really need, but can't turn off, as we are in debugging mode while developing.....).

Now, how about this approach (food for thought). We store the file system date of each file in the RP. Once per minute, the RP checks for all watched files if their date has changed. If it has, reload automatically (if this is the current behaviour, please ignore my ignorance. I do assume it just reloads all the files on a regular basis). If it breaks, deliver a page to *all* request (or, if this is too hard, all request to site-nodes that have an instance of the package installed), stating that the reloading of file xyz has failed and a developer should look at it. Along with a link to try reloading the file.

Basic idea behind this:

- I want to get the number of logfile messages down.
- If we break something, it will be immediatly shown. No way for an inconsistent system.
- You should never develop on a production site. Having a general "Your site is broken" might encourage people to not do this or fix it *really* quickly.

I know of enough developers that restart the server instead of watching files, as this feature is not working to the best advantage. Maybe everyone could raise their opinion here as well, stating what the feature *should* be able to do. We could tackle this issue in one of the OpenACS festivals coming up.

Collapse
2: Re: Watching files, new solutions (response to 1)
Posted by Randy O'Meara on 11/30/03 07:44 PM
Malte,

Some excellent ideas. Maybe you're talking about expanding the functionality of developer-support? There was a recent discussion (https://openacs.org/forums/message-view?message_id=137603) regarding log analysis that might fit in here also.

Randy

Collapse
3: Re: Watching files, new solutions (response to 1)
Posted by Don Baccus on 12/01/03 01:00 AM
The existing watch mechanism is efficient and simple (and is turned off if you run in production mode).

Checking all watched files once a minute would be more complex and would make me wait up to a minute for an edit to become effective, making me wonder, perhaps, if a bug fix is working or not.

As far as the vast number of debug messages ... well, I don't run with debug on normally for this very reason.  Now ... it would seem a per-package "debug" option would be extremely useful, because how often do you really need debug information for each and every package installed in your system?

Integrating watch logging with developer support would be reasonable.  Though I just watch the AOLserver logfile myself ...

Collapse
4: Re: Watching files, new solutions (response to 1)
Posted by Mark Aufflick on 12/01/03 02:23 AM
I agree with Don - it actually works really well now.

The only problem is the original one that was raised - if you're not wathcing the log (I don't always have enough screen real estate), you don't know if your file has failed to reload. It's always when you do something seemingly trivial that it fails too!

The developer support extention is a good idea, but I honestly think just putting a red X next to the package in the APM instead of a reload link should be enough of a reminder for when you forget.

Collapse
5: Re: Watching files, new solutions (response to 1)
Posted by Carl Coryell-Martin on 12/05/03 04:25 AM
After breaking the server, I like integrating it with developer support.  that way you know the page is broken when you load it.
Collapse
6: Re: Watching files, new solutions (response to 5)
Posted by Jeff Davis on 12/05/03 11:45 AM
Another thing we could do is to check the file is parsable before reloading.

Here is a useful tcl script that walks through all the tcl files and checks if they parse properly (in the [info complete] sense at least)... 

ad_page_contract {
    qa.tcl 
Check Tcl files for parsability

    Output streamed.

    @author Jeff Davis (davis@xarg.net)
    @creation-date 2003-12-01
    @cvs-id $Id$
} {
    {package:trim,optional {}}
}

ReturnHeaders

# couple of local helper procs 
proc ::tcl_p {file} { 
    return [expr [string match {*.tcl} $file] || [file isdirectory $file]]
}

proc bel {file} {
    return "<b>$file</b>(<a href=\"e?file=$file\">edit</a>)"
}

set startdir [acs_root_dir]/packages/$package

ns_write "Checks starting from $startdir<br />"

ns_write "Checking that the Tcl files parse...<br /><br />"

set checked 0
foreach file [ad_find_all_files -check_file_func ::tcl_p $startdir] { 
    incr checked 

    # Check that the file parses
    set fp [open $file "r"]
    set data [read $fp]
    close $fp

    if {![info complete $data]} {
        ns_write "<br />Parse Error: [bel $file]<br />"
    } else {
        #ns_write "$file "
    }
}
ns_write "Done checking files, did $checked<br /><br />"
Collapse
7: Re: Watching files, new solutions (response to 6)
Posted by Malte Sussdorff on 12/06/03 01:30 PM
Could we add scripts like yours to the monitoring package. Or developer support?
Collapse
8: Re: Watching files, new solutions (response to 7)
Posted by Jeff Davis on 12/06/03 03:48 PM
yes
Collapse
9: Re: Watching files, new solutions (response to 7)
Posted by Joel Aufrecht on 12/06/03 04:11 PM
I have added it to automated-testing in acs-tcl as acs_tcl__all_tcl_files_parse_p, in HEAD.  I dumbed it down a bit:

1) No support for testing only a sub-tree.  Maybe we should somehow make this a standard auto-test per-package instead of a single global?

2) Killed the link to the file since it didn't the original coded link didn't work for me and building a link to api-doc looked non-trivial.

3) I think we are going to need to get some conventions/support/documentation for differentiating between global tests like this, feature-specific regression tests, etc.  acs-automated-testing has some built-in categorization which we're not really using and we should tackle that in 5.1 as we add adp/tcl testing through tclwebtest.

Collapse
10: Re: Watching files, new solutions (response to 9)
Posted by Tilmann Singer on 12/09/03 05:31 PM
I never had troubles with the existing mechanism either. This script helps me find errors when tail'ing the error log by colorizing it and putting errors in bold red:

https://openacs.org/storage/file?file_id=151462

A few Shift-PGUP at most quickly reveal the source of any errors after a reload of the offending page.