How to handle "connection already closed" errors

Created by Gustaf Neumann, last modified by Gustaf Neumann 21 Aug 2020, at 12:47 PM

Newer versions of NaviServer produce by default an error, when a script tries to write a connection, which as already closed by the server (e.g. doing two ad_returndirect in a sequence). Such program code does not make sense and indicates an error that should be fixed.

In general, one can silence the behavior or fix it. To silence the behavior, on can alter the NaviServer config file to silently swallow these error conditions named "rejectalreadyclosedconn". See e.g. Sample configuration file

Recent version of OpenACS (oacs-5-10) have such error conditions already fixed in all the actively maintained packages in the oacs-5-10 branch. In order to fix custom packages, one has to watch out for the following situations:

• Request pages: all after every command that finishes a request a call to "ad_script_abort" is required. Such commands are essentially all "ns_return*" commands and the "ad_*" counterparts.
• Filters: in case, an application handles such commands in filters, the filter has to return with "filter_return" to stop the filter chain (i.e. not to call further filters). Notice that the documentation of ns_register states that the request page is as well handled to trigger e.g. the access log entry. So, in rare conditions, one has to check as well for additional closed cases by using "ns_conn isconnected".

Sometimes, it might be still tricky to find such occurrences. On recent versions of OpenACS one can use the xo* machinery to activate/deactivate Tcl command traces for such commands via ds/shell, like e.g. in:

xo::broadcast send {
  set traced_cmds [info commands ::ns_return*]
  foreach cmd $traced_cmds {trace add execution $cmd  enter {::tcltrace::before}}
}

"xo::broadcast" sends the command to every tcl based thread that executes the provided command. Tracing can be deactivated by replacing "trace add" by "trace remove" in this command.

Plain OpenACS (in oacs-5-10) has as well some support for this via acs-tcl/tcl/tcltrace-init.tcl.

xowf plugin for Monaco code editor

Created by Gustaf Neumann, last modified by Gustaf Neumann 11 Aug 2020, at 09:40 AM

Code metrics and Package Description

Package Specification Summary for Package: xowf-monaco-plugin

Bug Tracker Summary for Package: xowf-monaco-plugin

There is no package with the name "xowf-monaco-plugin" known to bug-tracker.

Code Metrics Summary for Package: xowf-monaco-plugin

Handling out of memory on "exec" calls

Created by Gustaf Neumann, last modified by Gustaf Neumann 13 Jul 2020, at 02:24 PM

On sites with high number of configured threads and high number of activated packages, the VM size of nsd might become large. One measures is to compile NaviServer and Tcl with the flag SYSTEM_MALLOC, and to run it with a memory efficient malloc library such as TCMalloc (using in the service file LD_PRELOAD=/usr/lib/libtcmalloc.so").

However, even then memory might become short, when "exec" is performed, even via the nsproxy module. The nsproxy module was developed to reduce memory consumption by running a separate process which communicates via pipes to nsd. The nsproxy module is designed to run multiple worker processes (the number can be configured), but when this number runs out (more such worker processes are needed) a new nsproxy worker process has to be created. This happens via a fork() system call under Linux, which can result in an out-of-memory message like the following:

 Error: exec: ns_fork() failed: Cannot allocate memory
 Error: exec failed: not enough memory

What should one do in such cases? In general, the first rule is to reduce the "exec" calls as far as possible, since these are relative slow and resource intensive, and NaviServer/Tcl provide a large number of built-ins or library functions, which should be used if possible.

Secondly, it is preferable to start many nsproxy workers rather soon in the live-time of nsd (e.g. at startup, when it has a small footprint) and ensure that the nsproxy module keeps these worker process alive a long time (by setting "idletimeout" to a high value)

ns_section ns/server/${server}/module/nsproxy {
     # ns_param recvtimeout        5000
     # ns_param waittimeout        1000
     # ns_param idletimeout        300000
     ns_param idletimeout  700000000
 }

When all configured nsproxy worker processes are running all the time, there is not need to fork later, and the error above will not occur anymore. The following snippet shows, how to start all nsproxy worker processes by creatin an ns_job queue with sufficient threads, and start a simple command executed by via nsproxy asynchronously (using the "-detached" flag) in parallel.

set concurrency [ns_proxy configure ExecPool -maxslaves]
if {"q1" ni [ns_job queues]} {
    ns_job create q1 $concurrency
}
#
# Queue a sufficient number of jobs to be executed in parallel
#
time {ns_job queue -detached q1 {exec sleep 1}} [expr {$concurrency * 2}]

Additionally, versions of NaviServer beyond 4.99.20 show via

ns_proxy stats ExecPool

the number of running worker processes of the ExecPool (also included in the process page of nsstats).

Site Wide Image Upload Widget

Created by Dave Bauer, last modified by Gustaf Neumann 01 May 2020, at 04:50 PM

MGH and Solution Grove are collaborating on a new image upload widget that will hopefully be easier to use and more easily integrated into any Xinha text element. 

Here is how it will work. The user will be editing the content of an object using a rich text editor (XINHA). If they decide an image or reference to a file is needed in the content, they can click the attach-file or attach-image icon.  A popup window appears allowing the user to choose a file to upload. The file is uploaded and stored in the content repository. If it is an image a thumbnail will be generated and linked to the full size image. When the content of the object is saved, the content is scanned for references to files or images, and a link is stored in the database that the particular object is "using" a file or image. This is the simplest case. Some more complex cases include allowing the user to search or browse images they have previously uploaded in a popup window to choose and image to insert.
 

The use case we are trying to solve:

• In assessment, right now an admin can associate one file or image with a question. The admin can already enter HTML for the question description. We need to allow image upload from Xinha so the admin can add an arbitraty number of images to one question description. We need this functionality in any richtext widget within the assessment creation process. 

 Requirements:

• upload image without thinking about where it will go, just entry optional title, description and choose a file from your Desktop
• image will have parent_id=package_id where its created.   if an object is used in multiple pages, we will keep track of that and let the owner see where it is used. We'll need to write a parser to extract /image/ url links. (DONE uses application_data_link feature from acs-tcl)
• image will inherit permissions depedning on the setting of the user. In general it'll inherit from the package_id or the subsite where its created.
  • To avoid namespace conflicts we can do someting like this for the cr_items.name
    • • cr_items.name = '{$item_id} {$filename}' or automatically rename the item filename(1) (2)...  (DONE, we strip the item id on display)
  • Image may be "private" which means it does not inherit permissions from any other object. This means the only person who can see it is the owner. If the image is uploaded within the context of another object, ie Xowiki page, a link is created between the page and the image. If someone has permission to view he page, they can also see the image. We make this work by creating a special URL /image/${image_id}/private/${xowiki_page_id}/${filename} This generated a unique URL that includes the "viewing context". When this URL is accessed 1) the system is checked to make sure a data link exists, that is, that the image is used "in" the viewing context object, and that the viewing user has permission to see the viewing context object.
  •  
• User uploading image will be granted direct "admin" permission over the image  DONE
• images will have a centralized URL for delivery, ie: an acs-subsite/www/images.vuh similar to o.vuh so urls will appear to be /images/${item_id}/image-filename.jpg (for example)  (Thanks to gustaf to notice the obvious namespace conflict with just using filename)  DONE
• images are stored as image type in the content repository (stored with "file" storage type) DONE
• thumbnails will be generated (propose to use the image-magick package, I have some code to contribute that will process an image upload and generate a thumbnail (this will be disabled if image-magick package is not installed) 
  • A new image thumbnail API is DONE

Once we have this in place, we have a simple, site-wide solution to upload image attachments to any object. This can be used in places like forums etc, and solves the problem of figuring out how to configure each Xinha widget in every package to find a place to store images.  By attaching the image directly to the object we solve the issue of finding a place to store images.

Future Ideas

These ideas are not planned for an initial implementation, but definitely are on our TODO list

• Add a UI to browse images you uploaded
• Add a UI to allow users to browse any images they can read?
• Add a way to search metadata on images when choosing an image so when you choose the add image button in Xinha, it will allow you to search existing images you have uploaded
• Add a recently viewed/uploaded feature to the image picker, so you can choose an image you recently viewed or uploaded without searching for it
  • Here is an example of what that could look like (actual working code exists for this now)
  • 
• Add a clipboard feature to the image picker, so you can use the optional clipboard package to mark an image in your clipboard while viewing it, then choose it from the clipboard to insert a link into another object. 

2006 November 3rd and November 4th (Training and Hacking Days)

Created by Carl Robert Blesius, last modified by Gustaf Neumann 01 May 2020, at 04:49 PM

Training and Hacking Days

The training and hacking days are part of the Fall_Conference_2006 and will be held in parallel at the Boston Museum of Science (details to follow).

Newbie Training for OpenACS/.LRN

This will be held in a beautiful classroom at the Musuem of Science with about 20 computers and taught by Tracy Adams.  This is an overview course designed to give you cover just enough of the right topics to lay the foundation for more advanced work.  The objective of the day will be to

1) Get hand-on introduction to the very basics
2) Get a high-level introduction to the more advanced topics and know where to go for more.

Agenda

9:30-10 - Preparation time, download and explore your virtual server image 

10-11 - The Basics

• TCL Basics - set, list
• ADP Basics - tags
• Templating Basics
• Hands on exercise

11-12 - Package Management, Site Map, Request Processor

• Package Manager, Installation Example
• Site Map Tour - Mounting
• Hands on Exercise - Modifying Site Map

12-1 - Basic Apis

• Procedure Documentation - api_doc
• URL Parameters - ad_page_contract
• Request Parameters - ad_conn
• Protecting pages to members
• Logging  ns_log 
• Hands on example

1-2 - Lunch at Museum of Science - There is no food or drink in the Museum of Science classrooms.

2-3 - Creating Packages

• Package Info File
• Directory Structure
• Creating Data Model
• Installing Package on System
• Creating Upgrade Script
• Upgrading
• Adding to Site Map
• Hands on Exercise

3-4 Procedures, Forms, Mail

• Forms - ad_form and components
• Mail - acs_mail_lite
• Procedures and self documenting
• Hands on Exercise

4-5 Database API, Displaying Multirow Database Results

• Database API
• Multirow/Multiple
• Hands on Exercise

5-6 Advance Topics

• ACS Objects
• Permission System

Hacking Days (a.k.a. Bug Bash)

more soon

OpenACS Core Packages (oacs-core)

Created by Benjamin Brink, last modified by Gustaf Neumann 01 May 2020, at 04:48 PM

Core packages are defined in CVS. See Aliases_at_CVS

Chart showing core package dependencies according to package specification files for oacs-5-9:

Chart is available in 2 versions:

• PNG format
• SVG format https://openacs.org/file/5370155/openacs-core-package-dependencies-by-pkg-spec.svg.svg

2006 OpenACS/.LRN Fall Conference

Created by Carl Robert Blesius, last modified by Gustaf Neumann 01 May 2020, at 11:39 AM

• Location
• Dates and Schedule
• Registration (and Fees)
• Travel
• Lodging
• Visa Information
• Public Attendance List
• Social Program
• Sponsors

Location 

Boston, Massachusetts, USA

Harvard Conference Center 

Map (includes location, close hotels, and public transportation stops) 

Dates and Schedule

• Keynotes and Schedule
• November 1st 2006 - International Workshop on Community Based E-Learning Systems (Educational/Community Web Applications Focus - e.g. .LRN) (Auditorium in the morning, 6 breakout sessions in the afternoon)
• November 2nd 2006 (General Web Applications Focus - OpenACS)
• November 3rd and November 4th (Training and Hacking Days): Location Boston Museum of Science

Registration and Fees

• Registration is open. Please let us know if you need any documents beforehand for visa applications or reimbursement
• Fees: Main conference fee is $95 (if you can not afford the cost of the conference please let us know - we will have a discounted rate and financial assistance may be available). Please see the registration page for additional details.

Travel

Weather

Average from Weather Underground

Local Airports

Logan International Airport is the airport in Boston. It is the closest airport to the conference center. Logan's airport code is BOS. International flights can land at Logan.

The Manchester Airport in Manchester, New Hampshire, and T.F. Green in Providence, Rhode Island, are airports within about an hour's drive of Boston and Cambridge. Many inexpensive airlines serve Providence and Manchester. It is possible to get from downtown Providence to Boston via commuter rail or bus. Hanscom Field in Bedford, Massachusetts, about 25 minutes away from Boston, has flights to and from Trenton, New Jersey.

Local Train Stations

Amtrak, the major American rail company, serves several stations in the Boston area. South Station is one of the area's major transit hubs. It is also on the Red Line (subway line). Amtrak also arrives at North Station, which is on the Green Line, and Back Bay station on the Orange Line.

Bus Lines and Subway

Lodging

Hotels

Best Western Inn at Longwood (SOLDOUT)
342 Longwood Ave, Boston, MA 02115
(617) 731-4700 or 1-(800)-GOT BEST
$159/night - "PARTNERS HEALTHCARE" rate
(try to reserve before October 16th to ensure availability)
0.5 miles from conference center

Longwood Guest Suites
63 Parker Hill Avenue, Boston, MA 02120
1-800-246-8357
$132/night - 1 person occupancy
1 mile from conference center

Longwood Inn
123 Longwood Ave, Brookline, MA 02446
1-800-754-6835
$119/night standard rate
0.8 miles from conference center

Looking for Roommates

(please add your name here if you are looking for someone to share a hotel room with) 

Crash Space

 (please add to these lists with a link to your user profile)

Crash Space Available:

• 4 spots (all taken) Carl Blesius

Crash Space Needed:

• Malte Sussdorff - cognovís - Germany

Who needs a visa to enter the US?

If you are not a resident of the United States, you may need a visa to enter. If you already have a visa and need to enter the US, make sure it will not expire before you use it to apply for admission (entry) at the port. Citizens of the following 27 countries do not need a visa to enter the US for tourism or business for stays of 90 days or less [1]:

[1] Providing you meet the visa waiver criteria:

a) You must have a machine-readable passport (this means it should have a line of chevrons (e.g., <<< ) valid for six months past your expected stay in the U.S.

b) Any passport issued after October 26, 2005 must include a digital photo.

c) You must have a ticket going out of the US. I.e. only a one-way ticket into the US is not sufficient.

Information for Canadians

Canadian citizens generally do not need either visa or passport to enter the US. All travelers should bring evidence of their identity (e.g., a government photo-id) and citizenship (e.g., a passport; or a birth or citizenship certificate). Fiancees of US citizens, or spouses of US permanent residents, may need a visa even if they are Canadian citizens — especially if they are coming to the US to await final immigration status. .

Getting a visa

To get a visa, contact the nearest US Embassy, Consulate, or other authorized institute to find out how long the process will take (generally anywhere from 3 to 45 days).

Some guidelines for painless visa processing:

• Ask us for a letter of invitation.
• Start applying for a visa early. The basic visa process will not take this long, but you will want time to resubmit the application if necessary, and to buy your tickets after the visa has been granted.
• Prepare for your visa interview/application. You should have
  1. Your entire travel itinerary, from when you leave your country to when you return

     Note that your travel plans depend upon early approval of the visa application

  2. Your invitation, and printed information about the conference

     If you are getting financial support to attend the conference, make sure you have printed documentation of this as well.

  3. Proof of association with OpenACS and/or .LRN (information about you as a researcher, developer, .LRN user, etc.)
  4. Evidence that you will return home -- that is, of "binding or sufficient" ties to your home country (normally your country of residence). Useful examples include:
     • evidence of family ties in your home country
     • evidence of property ownership
     • evidence/statements of bank accounts
     • an employment contract or letter from an employer demonstrating you have employment beyond the end of your trip
     • evidence of attending school, or a letter from a school official demonstrating you will be a student there beyond the end of your trip

Getting visa help

A letter of invitation alone does not guarantee you will be issued a visa. If you have followed the above steps, and your visa application is rejected, let us know immediately. Immigration officials in Massachusetts will contact your embassy to try to overturn the rejection.

If you cannot afford the cost of visas or related fees, let us know. Financial assistance may be available.

Interest in Attending

Enter your name, and any topics you are interested in here Fall Conference 2006 Interest in Attending

Social Program

Dinner  on November 1st will take place at

Jasper White's Summer Shack
Starts at 8:00PM
50 Dalton Street in the Back Bay, Boston, MA
across from the Sheraton Hotel entrance and the Hynes Auditorium. Upstairs from Kings Bowling Alley.
Closest Subway T Station: Green Line - Hynes Auditorium.

The social program on subsequent days will be informal and will be announced at the event

Sponsors

File Storage

Created by Robert Taylor, last modified by Gustaf Neumann 01 May 2020, at 11:37 AM

I added views support to file storage folder-chunk as can be seen here: 

Furthermore I added the ability to download checked files (instead of having to download the whole folder). THe name of the zipfile will be the name of the folder you are looking at (so the same as if you download the whole folder).

As the file is zipped, this effectively lets you download a compressed ZIP file, so you don't have to download that much data.

just playing around

Created by Robert Taylor, last modified by Gustaf Neumann 01 May 2020, at 11:36 AM

OpenACS Proposal - Sample

Created by Ryan Gallimore, last modified by Gustaf Neumann 01 May 2020, at 11:36 AM

OpenACS – Open Source Software Initiative

OpenACS has a proven suite of collaboration tools that are in current use by nonprofits, educational institutions and commercial companies worldwide.  Internationalization (I18n) is currently being completed by Heidelberg University.  Building on the strength of OpenACS, this project can focus on making the tools work effectively for NGOs rather than tool creation.  New features or feature enhancement will be developed when needed, but emphasis will be placed on using existing code and resources.

This project will customize the user interface and information architecture of existing OpenACS functionality to support multi-national grassroots communication.  It will also create an easy to install version of OpenACS based on these customizations.  Combined with enhanced and specific documentation it will make a very sophisticated multi-lingual community system available and accessible to XXX and other NGOs alike.  Finally, the project will also create educational marketing material aimed at multi-national grassroots NGOs to make them aware of this new resource.

Unified Toolset

On-line collaboration typically uses a variety of tools.  A well-constructed application uses them efficiently.  The figure below was created for marketing purpose to educational institutions; nonetheless it is a good image for understanding existing and planned OpenACS functionalities.

Modularity is wonderful for quickly assembling exactly the application you need.  Maintainability is enhanced in OpenACS when similar modules share the same basic functionality.  Thus, from a more technical perspective OpenACS is a layered architecture with many packages that provide diverse functionality to end users; it relies on common underlying structural packages such as the content repository, notifications, date-time space, authentication, and context search.  This structured approach allows the easy creation  of specialized, yet unified, new functionality with a minimum of separate and new code. 

Mature Broad Based Developer Community

OpenACS traces its roots back to Philip Greenspun, a Computer Science Professor at MIT, and his work on Photo.net starting in 1995 and Philip and Alex’s Guide to Web Publishing in 1998.  The code base has always emphasized collaboration and management of geographically diverse online communities.  The underlying engineering was supported by millions of dollars of venture capital spent on hiring PhDs in Computer Science from MIT, CalTech and other major universities across the Atlantic.

OpenACS has proven its durability and utility by surviving the death of its parent company (ArsDigita) to grow into a vibrant grassroots collection of independent consultants and small companies implementing diverse and complex web solutions around the globe for fun, philanthropy, and profit.

Example OpenACS Implementation - dotLRN

dotLRN is an application built on top of OpenACS specifically targeted for universities.  A consortium lead by the Sloan School of Management at MIT and Heidelberg University has significantly improved enhanced OpenACS.  In many ways this project will be philosophically modeled after dotLRN.  It will create an application built on top of OpenACS that is specifically targeted to NGOs needing web based community tools in a multi-lingual environment.

This project will also build directly upon functionality implemented by dotLRN. Especially important for this project is the strong emphasis on groups, subgroups and distributed administration.  In Learning Management Systems (LMS) it is very important that instructors have extensive control over membership, content and presentation.  Designated XXX leaders will reuse this functionality extensively to allow the project to have self-managing geographically diverse and common-interest groups and subgroups that can be managed locally without extensive time commitment from the parent organization.

One goal of this project is to deliver a code base that has significant overlap with dotLRN.  This is desirable from a sustainability and maintenance point of view because enhancements and bug fixes made to the dotLRN community will be available to the NGO community as well.  OpenACS’ layered package system is designed to support this type of code reuse and make it possible to have specialized code easily segregated from customized code for enhanced maintainability.

Case studies on dotLRN can be found at: https://dotlrn.org/users/cases

Active Development

OpenACS is continually being improved, often in ways that will be useful to nonprofits and NGOs. Currently there is active development of the following features.

• E-Commerce G2
• Xowiki
• dotLrn

Although many of these features will not be immediately used in the XXX site, they will be vital for the future sustainability and expandability of this project, giving the project the ability to be quickly extended to meet the needs of a multitude of NGOs.

Service Providers Available Globally

The product produced will be a custom installation of OpenACS that any OpenACS company or developer would be able to support.  Currently, there are consultants and companies that provide support for OpenACS in the following countries: Australia, Belgium, Brazil, Canada Denmark, Finland, Germany, Guatemala, India, Israel, Italy, Netherlands, Norway, Philippines, Spain, Sweden, Switzerland, UK, US,.

Standards Compliance and Interoperability

OpenACS has strong and continuing support of standards that will enable this project and future extensions to share content and interoperate with other systems.

The Simple Object Access Protocol (SOAP) web services – a way to create widely distributed, complex applications that run over the Internet, has already been incorporated into some OpenACS packages and research is underway at the University of Sydney into further uses of Web Services.

OpenACS also includes a RSS Support Design Module to support the RDF Site Summary (RSS), an XML-based, lightweight multipurpose extensible metadata description and syndication format.  RSS support positions OpenACS for future functionality requiring accepting and distributing syndicated news feeds and other RSS-compliant content.

The OpenACS WAP module supports the wireless access protocol (WAP), thus allowing easy deliver of content to WAP-enabled devices such as cell phones, PDAs, etc., as well as regular desktop browsers.  In fact, the simplicity of the OpenACS interface means it is well suited for use on mobile clients. 

OpenACS and its dotLRN consortium are committed to implementing the Sharable Content Object Reference Model (SCORM), Instructional Management System (IMS) and Open Knowledge Initiative (OKI) standards, all among the most widely used standards for learning object interoperability.