i18n-requirements.xml
Delivered as text/xml
[ hide source ] | [ make this the default ]
File Contents
<?xml version='1.0' ?>
<!DOCTYPE sect1 PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd" [
<!ENTITY % myvars SYSTEM "../variables.ent">
%myvars;
]>
<sect1 id="i18n-requirements" xreflabel="OpenACS &version; Internationalization Requirements">
<title>OpenACS Internationalization Requirements</title>
<authorblurb>
<para>by Henry Minsky,
<ulink url="mailto:yon@openforce.net">Yon Feldman</ulink>,
<ulink url="mailto:lars@collaboraid.biz">Lars Pind</ulink>,
<ulink url="mailto:peter@collaboraid.biz">Peter Marklund</ulink>,
<ulink url="mailto:christian@collaboraid.biz">Christian Hvid</ulink>,
and others.</para>
</authorblurb>
<sect2 id="i18n-requirements-introduction">
<title>Introduction</title>
<para>
This document describes the requirements for functionality in
the OpenACS platform to support globalization of the core and optional
modules. The goal is to make it possible to support delivery of
applications which work properly in multiple locales with the
lowest development and maintenance cost.
</para>
</sect2>
<sect2 id="i18n-requirements-definitions">
<title>Definitions</title>
<variablelist>
<varlistentry>
<term>internationalization (i18n)</term>
<listitem>
<para>
The provision within a computer program of the
capability of making itself adaptable to the requirements of different
native languages, local customs and coded character sets.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>locale</term>
<listitem>
<para>
The definition of the subset of a user's environment that depends on
language and cultural conventions.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>localization (L10n)</term>
<listitem>
<para>
The process of establishing information within a computer system
specific to the operation of particular native languages, local
customs and coded character sets.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>globalization</term>
<listitem>
<para>
A product development approach which ensures that software products
are usable in the worldwide markets through a combination of
internationalization and localization.
</para>
</listitem>
</varlistentry>
</variablelist>
</sect2>
<sect2 id="II._Vision_Statement">
<title>Vision Statement</title>
<para>The Mozilla project suggests keeping two catchy phrases in
mind when thinking about globalization:</para>
<itemizedlist>
<listitem>
<para>One code base for the world</para>
</listitem>
<listitem>
<para>English is just another language</para>
</listitem>
</itemizedlist>
<para>Building an application often involves making a number of
assumptions on the part of the developers which depend on their own
culture. These include constant strings in the user interface and
system error messages, names of countries, cities, order of given
and family names for people, syntax of numeric and date strings and
collation order of strings.</para>
<para>The OpenACS should be able to operate in languages and regions
beyond US English. The goal of OpenACS Globalization is to provide a
clean and efficient way to factor out the locale dependent
functionality from our applications, in order to be able to easily
swap in alternate localizations.</para>
<para>This in turn will reduce redundant, costly, and error prone
rework when targeting the toolkit or applications built with the
toolkit to another locale.</para>
<para>The cost of porting the OpenACS to another locale without some
kind of globalization support would be large and ongoing, since
without a mechanism to incorporate the locale-specific changes
cleanly back into the code base, it would require making a new fork
of the source code for each locale.</para>
</sect2>
<sect2 id="system-application-overview">
<title>System/Application Overview</title>
<para>A globalized application will perform some or all of the
following steps to handle a page request for a specific
locale:</para>
<orderedlist>
<listitem>
<para>Decide what the target locale is for an incoming page
request</para>
</listitem>
<listitem>
<para>Decide which character set encoding the output should be
delivered in</para>
</listitem>
<listitem>
<para>If a script file to handle the request needs to be loaded
from disk, determine if a character set conversion needs to be
performed when loading the script</para>
</listitem>
<listitem>
<para>If needed, locale-specific resources are fetched. These can
include text, graphics, or other resources that would vary with the
target locale.</para>
</listitem>
<listitem>
<para>If content data is fetched from the database, check for
locale-specific versions of the data (e.g. country names).</para>
</listitem>
<listitem>
<para>Source code should use a message catalog API to translate
constant strings in the code to the target locale</para>
</listitem>
<listitem>
<para>Perform locale-specific linguistic sorting on data if
needed</para>
</listitem>
<listitem>
<para>If the user submitted form input data, decide what character
set encoding conversion if any is needed. Parse locale-specific
quantities if needed (number formats, date formats).</para>
</listitem>
<listitem>
<para>If templating is being used, select correct locale-specific
template to merge with content</para>
</listitem>
<listitem>
<para>Format output data quantities in locale-specific manner
(date, time, numeric, currency). If templating is being used, this
may be done either before and/or after merging the data with a
template.</para>
</listitem>
</orderedlist>
<para>Since the internationalization APIs may potentially be used
on every page in an application, the overhead for adding
internationalization to a module or application must not cause a
significant time delay in handling page requests.</para>
<para>In many cases there are facilities in Oracle to perform
various localization functions, and also there are facilities in
Java which we will want to move to. So the design to meet the
requirements will tend to rely on these capabilities, or close
approximations to them where possible, in order to make it easier
to maintain Tcl and Java OpenACS versions.</para>
</sect2><sect2 id="IV._Use-cases_and_User-scenarios"><title>Use-cases and User-scenarios</title>
<para>Here are the cases that we need to be able to handle
efficiently:</para>
<orderedlist>
<listitem>
<para>A developer needs to author a web site/application in a
language besides English, and possibly a character set besides
ISO-8859-1. This includes the operation of the OpenACS itself, i.e.,
navigation, admin pages for modules, error messages, as well as
additional modules or content supplied by the web site
developer.</para>
<para>What do they need to modify to make this work? Can their
localization work be easily folded in to future releases of
OpenACS?</para>
</listitem>
<listitem>
<para>A developer needs to author a web site which operates in
multiple languages simultaneously. For example, www.un.org with
content and navigation in multiple languages.</para>
<para>The site would have an end-user visible UI to support these
languages, and the content management system must allow articles to
be posted in these languages. In some cases it may be necessary to
make the modules' admin UI's operate in more than one
supported language, while in other cases the backend admin
interface can operate in a single language.</para>
</listitem>
<listitem>
<para>A developer is writing a new module, and wants to make it
easy for someone to localize it. There should be a clear path to
author the module so that future developers can easily add support
for other locales. This would include support for creating
resources such as message catalogs, non-text assets such as
graphics, and use of templates which help to separate application
logic from presentation.</para>
</listitem>
</orderedlist>
</sect2><sect2 id="Competitive_Analysis"><title>Competitive
Analysis</title>
<para>Other application servers: ATG Dyanmo, Broadvision, Vignette,
... ? Anyone know how they deal with i18n ?</para>
</sect2><sect2 id="V._Related_Links"><title>Related
Links</title>
<itemizedlist>
<listitem>
<para><emphasis>System/Package "coversheet" - where all
documentation for this software is linked off of</emphasis></para>
</listitem>
<listitem>
<para><emphasis><link linkend="i18n-design">Design document</link></emphasis></para>
</listitem>
<listitem>
<para><emphasis><link linkend="i18n">Developer's guide</link></emphasis></para>
</listitem>
<listitem>
<para><emphasis>User's guide</emphasis></para>
</listitem>
<listitem>
<para><emphasis>Other-cool-system-related-to-this-one
document</emphasis></para>
<para><ulink url="http://www.li18nux.net/">LI18NUX
2000 Globalization Specification:
http://www.li18nux.net/</ulink></para>
<para><ulink url="http://www.mozilla.org/docs/refList/i18n/l12yGuidelines.html">Mozilla
i18N Guidelines:
http://www.mozilla.org/docs/refList/i18n/l12yGuidelines.html</ulink></para>
<para><ulink url="https://www.iso.org/standard/4766.html">ISO
639:1988 Code for the representation of names of languages
</ulink></para>
<para><ulink url="https://www.iso.org/standard/24591.html">ISO 3166-1:1997
Codes for the representation of names of countries and their
subdivisions Part 1: Country codes
</ulink></para>
<para><ulink url="http://www.isi.edu/in-notes/iana/assignments/character-sets">IANA
Registry of Character Sets</ulink></para>
</listitem>
<listitem>
<para><emphasis>Test plan</emphasis></para>
</listitem>
<listitem>
<para><emphasis>Competitive system(s)</emphasis></para>
</listitem>
</itemizedlist>
</sect2><sect2 id="VI_Requirements"><title>Requirements</title>
<para>Because the requirements for globalization affect many areas
of the system, we will break up the requirements into phases, with
a base required set of features, and then stages of increasing
functionality.</para>
</sect2><sect2 id="VI.A_Locales"><title>Locales</title>
<para><emphasis>10.0</emphasis></para>
<para>A standard representation of locale will be used throughout
the system. A locale refers to a language and territory, and is
uniquely identified by a combination of ISO language and ISO
country abbreviations.</para>
<blockquote>
<para>See
<ulink url="/doc/acs-content-repository/requirements.html#100-20">Content
Repository Requirement 100.20</ulink></para>
<para><emphasis>10.10</emphasis> Provide a consistent
representation and API for creating and referencing a locale</para>
<para><emphasis>10.20</emphasis> There will be a Tcl library of
locale-aware formatting and parsing functions for numbers, dates
and times. <emphasis>Note that Java has built-in support for these
already</emphasis>.</para>
<para><emphasis>10.30</emphasis> For each locale there will be
default date, number and currency formats. <remark>Currency i18n is
NOT IMPLEMENTED for 5.0.0.</remark></para>
<para><emphasis>10.40</emphasis>Administrators can upgrade their
servers to use new locales via the APM. <remark>NOT IMPLEMENTED in
5.0.0; current workaround is to get an XML file and load it
manually.</remark></para>
</blockquote>
</sect2><sect2 id="VI.B_Associating_a_Locale_with_a_Request"><title>Associating a Locale with a Request</title>
<para><emphasis>20.0</emphasis></para>
<para>The request processor must have a mechanism for associating a
locale with each request. This locale is then used to select the
appropriate template for a request, and will also be passed as the
locale argument to the message catalog or locale-specific
formatting functions.</para>
<blockquote>
<para><emphasis>20.10</emphasis> The locale for a request should be
computed by the following method, in descending order of
priority:</para>
<itemizedlist>
<listitem>
<para>get locale associated with subsite or package id</para>
</listitem>
<listitem>
<para>get locale from user preference</para>
</listitem>
<listitem>
<para>get locale from site wide default</para>
<para><emphasis>20.20</emphasis> An API will be provided for
getting the current request locale from the
<literal>ad_conn</literal> structure.</para>
</listitem>
</itemizedlist>
</blockquote>
</sect2><sect2 id="resource-bundles">
<title>Resource Bundles / Content Repository</title>
<para><emphasis>30.0</emphasis></para>
<para>A mechanism must be provided for a developer to group a set
of arbitrary content resources together, keyed by a unique
identifier and a locale.</para>
<para>For example, what approaches could be used to implement a
localizable nav-bar mechanism for a site? A navigation bar might be
made up of a set of text strings and graphics, where the graphics
themselves are locale-specific, such as images of English or
Japanese text (as on www.un.org). It should be easy to
specify alternate configurations of text and graphics to lay out
the page for different locales.</para>
<para>Design note: Alternative mechanisms to implement this
functionality might include using templates, Java ResourceBundles,
content-item containers in the Content Repository, or some
convention assigning a common prefix to key strings in the message
catalog.</para>
</sect2><sect2 id="VI.D_Message_Catalog_for_String_Translation"><title>Message Catalog for String Translation</title>
<para><emphasis>40.0</emphasis></para>
<para>A message catalog facility will provide a database of
translations for constant strings for multilingual applications. It
must support the following:</para>
<blockquote>
<para><emphasis>40.10</emphasis> Each message will referenced via
unique a key.</para>
<para><emphasis>40.20</emphasis> The key for a message will have
some hierarchical structure to it, so that sets of messages can be
grouped with respect to a module name or package path.</para>
<para><emphasis>40.30</emphasis> The API for lookup of a message
will take a locale and message key as arguments, and return the
appropriate translation of that message for the specified
locale.</para>
<para><emphasis>40.40</emphasis> The API for lookup of a message
will accept an optional default string which can be used if the
message key is not found in the catalog. This lets the developer
get code working and tested in a single language before having to
initialize or update a message catalog.</para>
<para><emphasis>40.50</emphasis> For use within templates, custom
tags which invoke the message lookup API will be provided.</para>
<para><emphasis>40.60</emphasis> Provide a method for importing and
exporting a flat file of translation strings, in order to make it
as easy as possible to create and modify message translations in
bulk without having to use a web interface.</para>
<para><emphasis>40.70</emphasis> Since translations may be in
different character sets, there must be provision for writing and
reading catalog files in different character sets. A mechanism must
exist for identifying the character set of a catalog file before
reading it.</para>
<para><emphasis>40.80</emphasis> There should be a mechanism for
tracking dependencies in the message catalog, so that if a string
is modified, the other translations of that string can be flagged
as needing update.</para>
<para><emphasis>40.90</emphasis> The message lookup must be as
efficient as possible so as not to slow down the delivery of
pages.</para>
</blockquote>
</sect2><sect2 id="VI.E_Character_Set_Encoding"><title>Character Set Encoding</title>
<para><emphasis>Character Sets</emphasis></para>
<para><emphasis>50.0</emphasis> A locale will have a primary
associated character set which is used to encode text in the
language. When given a locale, we can query the system for the
associated character set to use.</para>
<para>The assumption is that we are going to use Unicode in our
database to hold all text data. Our current programming
environments (Tcl/Oracle or Java/Oracle) operate on Unicode data
internally. However, since Unicode is not yet commonly used in
browsers and authoring tools, the system must be able to read and
write other character sets. In particular, conversions to and from
Unicode will need to be explicitly performed at the following
times:</para>
<itemizedlist>
<listitem>
<para>Loading source files (.tcl or .adp) or content files from the
filesystem</para>
</listitem>
<listitem>
<para>Accepting form input data from users</para>
</listitem>
<listitem>
<para>Delivering text output to a browser</para>
</listitem>
<listitem>
<para>Composing an email message</para>
</listitem>
<listitem>
<para>Writing data to the filesystem</para>
</listitem>
</itemizedlist>
<para>Acs-templating does the following.</para>
<itemizedlist>
<listitem>
<para>When the acs-templating package opens an ADP or Tcl
file, it assumes the file is iso-8859-1. If the output
charset (OutputCharset) in the AOLserver config file is set,
then acs-templating assumes it's that charset.
Writing Files</para>
</listitem>
<listitem>
<para>When the acs-templating package writes an ADP or
Tcl file, it assumes the file is iso-8859-1. If the output
charset (OutputCharset) in the AOLserver config file is set,
then acs-templating assumes it's that charset. </para>
</listitem>
</itemizedlist>
<sect3 id="Tcl_Source_File_Character_Set">
<title>Tcl Source File Character Set</title>
<blockquote>
<para>There are two classes of Tcl files loaded by the system;
library files loaded at server startup, and page script files,
which are run on each page request.</para>
<para><emphasis>Should we require all Tcl files be stored as UTF8?
That seems too much of a burden on developers.</emphasis></para>
<para><emphasis>50.10</emphasis> Tcl library files can be authored
in any character set. The system must have a way to determine the
character set before loading the files, probably from the
filename.</para>
<para><emphasis>50.20</emphasis> Tcl page script files can be
authored in any character set. The system must have a way to
determine the character set before loading the files, probably from
the filename.</para>
</blockquote>
</sect3>
<sect3 id="Submitted_Form_Data_Character_Set">
<title>Submitted Form Data Character Set</title>
<para><emphasis>50.30</emphasis> Data which is submitted with an
HTTP request using a GET or POST method may be in any character
set. The system must be able to determine the encoding of the form
data and convert it to Unicode on demand. </para>
<para><emphasis>50.35</emphasis> The developer must be able to
override the default system choice of character set when parsing
and validating user form data. <remark>INCOMPLETE - form
widgets in acs-templating/tcl/date-procs.tcl are not
internationalized. Also, acs-templating's UI needs to be
internationalized by replacing all user-visible strings with
message keys.</remark></para>
<para><emphasis>50.30.10</emphasis>In Japan and some
other Asian languages where there are multiple character set
encodings in common use, the server may need to attempt to do an
auto-detection of the character set, because buggy browsers may
submit form data in an unexpected alternate encoding.</para>
</sect3>
<sect3 id="Output_Character_Set">
<title>Output Character Set</title>
<blockquote>
<para><emphasis>50.40</emphasis> The output character set for a
page request will be determined by default by the locale associated
with the request (see requirement 20.0).</para>
<para><emphasis>50.50</emphasis> It must be possible for a
developer to manually override the output character set encoding
for a request using an API function.
</para>
</blockquote>
</sect3>
</sect2>
<sect2 id="VI.F_ACS_Kernel_Issues"><title>ACS Kernel Issues</title>
<blockquote>
<para><emphasis>60.10</emphasis> All OpenACS error messages must use
the message catalog and the request locale to generate error
message for the appropriate locale.<remark>NOT IMPLEMENTED for 5.0.0.</remark></para>
<para><emphasis>60.20</emphasis> Web server error messages such as
404, 500, etc must also be delivered in the appropriate
locale.</para>
<para><emphasis>60.30</emphasis> Where files are written or read
from disk, their filenames must use a character set and character
values which are safe for the underlying operating system.</para>
</blockquote>
</sect2><sect2 id="VI.G_Templates"><title>Templates</title>
<blockquote>
<para><emphasis>70.0</emphasis> For a given abstract URL, the
designer may create multiple locale-specific template files may be
created (one per locale or language)</para>
<para><emphasis>70.10</emphasis> For a given page request, the
system must be able to select an appropriate locale-specific
template file to use. The request locale is computed as per (see
requirement 20.0).</para>
<para><emphasis>70.20</emphasis>A template file may be created for
a partial locale (language only, without a territory), and the
request processor should be able to find the closest match for the
current request locale.</para>
<para><emphasis>70.30</emphasis> A template file may be created in
any character set. The system must have a way to know which
character set a template file contains, so it can properly process
it.</para>
</blockquote>
<sect3 id="Formatting_Datasource_Output_in_Templates"><title>Formatting
Datasource Output in Templates</title>
<para><emphasis>70.50</emphasis> The properties of a datasource
column may include a datatype so that the templating system can
format the output for the current locale. The datatype is defined
by a standard OpenACS datatype plus a format token or format string,
for example: a date column might be specified as
'current_date:date LONG,' or 'current_date:date
"YYYY-Mon-DD"'</para>
</sect3><sect3 id="Forms"><title>Forms</title>
<blockquote>
<para><emphasis>70.60</emphasis> The forms API must support
construction of locale-specific HTML form widgets, such as date
entry widgets, and form validation of user input data for
locale-specific data, such as dates or numbers. <emphasis>NOT
IMPLEMENTED in 5.0.0.</emphasis></para>
<para><emphasis>70.70</emphasis> For forms which allow users to
upload files, a standard method for a user to indicate the charset
of a text file being uploaded must be provided.</para>
<para><emphasis>Design note: this presumably applies to uploading
data to the content repository as well</emphasis></para>
</blockquote>
</sect3></sect2><sect2 id="VI.H_Sorting_and_Searching"><title>Sorting and Searching</title>
<blockquote>
<para><emphasis>80.10</emphasis> Support API for correct collation
(sorting order) on lists of strings in locale-dependent way.</para>
<para><emphasis>80.20</emphasis> For the Tcl API, we will say that
locale-dependent sorting will use Oracle SQL operations (i.e., we
won't provide a Tcl API for this). We require a Tcl API
function to return the correct incantation of NLS_SORT to use for a
given locale with <literal>ORDER BY</literal> clauses in
queries.</para>
<para><emphasis>80.40</emphasis> The system must handle full-text
search in any supported language.</para>
</blockquote>
</sect2><sect2 id="VI.G_Time_Zones"><title>Time Zones</title>
<blockquote>
<para><emphasis>90.10</emphasis> Provide API support for specifying
a timezone</para>
<para><emphasis>90.20</emphasis> Provide an API for computing time
and date operations which are aware of timezones. So for example a
calendar module can properly synchronize items inserted into a
calendar from users in different timezones using their own local
times.</para>
<para><emphasis>90.30</emphasis> Store all dates and times in
universal timezone, UTC.</para>
<para><emphasis>90.40</emphasis> For a registered users, a timezone preference should be stored.</para>
<para><emphasis>90.50</emphasis> For a non-registered user a timezone preference should be attached via a session or else UTC should
be used to display every date and time.</para>
<para><emphasis>90.60</emphasis> The default if we can't
determine a timezone is to display all dates and times in some
universal timezone such as GMT.</para>
</blockquote>
</sect2><sect2 id="VI.H_Database"><title>Database</title>
<blockquote>
<para><emphasis>100.10</emphasis> Since UTF8 strings can use up to
three (UCS2) or six (UCS4) bytes per character, make sure that
column size declarations in the schema are large enough to
accommodate required data (such as email addresses in
Japanese). <remark>Since 5.0.0, this is covered in the database
install instructions for both PostgreSQL and Oracle.</remark></para>
</blockquote>
</sect2>
<sect2 id="VI.I_Email_and_Messaging">
<title>Email and Messaging</title>
<para>When sending an email message, just as when delivering the
content in web page over an HTTP connection, it is necessary to be
able to specify what character set encoding to use, defaulting to UTF-8.</para>
<blockquote>
<para><emphasis>110.10</emphasis> The email message sending API
will allow for a character set encoding to be specified.</para>
<para><emphasis>110.20</emphasis> The email accepting API
allows for character set to be parsed correctly (the message has a MIME
character set content type header) </para>
</blockquote>
<para>Mail is not internationalized. The following issues must be addressed.</para>
<itemizedlist>
<listitem>
<para>
Many functions still call ns_sendmail. This
means that there are different end points for sending
mail. This should be changed to use the acs-mail-lite API instead.
</para>
</listitem>
<listitem>
<para>
Consumers of email services must do
the following: Determine the appropriate language or
languages to use for the message subject and message body
and localize them (as in notifications).
</para>
</listitem>
<listitem>
<para>Extreme Use case: Web site has a default language of Danish. A forum is set up for Swedes, so the forum has a package_id and a language setting of Swedish. A poster posts to the forum in Russian (is this possible?). A user is subscribed to the forum and has a language preference of Chinese. What should be in the message body and message subject?</para>
</listitem>
<listitem><para>Incoming mail should be localized.</para>
</listitem>
</itemizedlist>
</sect2>
<sect2 id="i18n-requirements-implementation-notes">
<title>Implementation Notes</title>
<para>
Because globalization touches many different parts of the system,
we want to reduce the implementation risk by breaking the
implementation into phases.
</para>
</sect2>
<sect2 id="i18n-requirements-revision-history">
<title>Revision History</title>
<informaltable>
<tgroup cols="4">
<tbody>
<row>
<entry><emphasis role="strong">Document Revision #</emphasis></entry>
<entry><emphasis role="strong">Action Taken, Notes</emphasis></entry>
<entry><emphasis role="strong">When?</emphasis></entry>
<entry><emphasis role="strong">By Whom?</emphasis></entry>
</row>
<row>
<entry>1</entry>
<entry>Updated with results of MIT-sponsored i18n work at Collaboraid.</entry>
<entry>14 Aug 2003</entry>
<entry>Joel Aufrecht</entry>
</row>
<row>
<entry>0.4</entry>
<entry>converting from HTML to DocBook and importing the document to the OpenACS
kernel documents. This was done as a part of the internationalization of
OpenACS and .LRN for the Heidelberg University in Germany</entry>
<entry>12 September 2002</entry>
<entry>Peter Marklund</entry>
</row>
<row>
<entry>0.3</entry>
<entry>comments from Christian</entry>
<entry>1/14/2000</entry>
<entry>Henry Minsky</entry>
</row>
<row>
<entry>0.2</entry>
<entry>Minor typos fixed, clarifications to wording</entry>
<entry>11/14/2000</entry>
<entry>Henry Minsky</entry>
</row>
<row>
<entry>0.1</entry>
<entry>Creation</entry>
<entry>11/08/2000</entry>
<entry>Henry Minsky</entry>
</row>
</tbody>
</tgroup>
</informaltable>
</sect2>
</sect1>
