Forum OpenACS Q&A: Globalization

One really import point for us in Europe is the support for multiple languages. Please note that most (if not all) major commercial learn platforms made the mistake of not being localizable from the start, and are paying for this now (figuratively and literally). I just wanted to mention this, because I doubt there are many foreign language courses at Sloan or Berklee, internationalization is easy to forget about when on a tight schedule, and working on it after the fact is a lot harder than when it is considered from the start (not to mention that making something support multiple languages at a later stage requires more coordination and communication than if it is done from the start).

This is important.

Finding out that OpenACS is the platform Greenpeace has chosen, eases my multilingual concerns on the OpenACS side (although it does not eliminate them). I just hope that the multilingual features of OpenACS and dotLRN will "dovetail" as so elegantly stated by Don. Obviously, the best solution would be one that is independent of dotLRN and built into OpenACS, making multilingual support a snap.

Can somebody that knows both systems and has experience in internationalization please comment on this? Who is working on this?

-- Carl Robert Blesius, May 9, 2002

i built the globalization infrastructure for ACS Java, so i have some experience with this stuff. i think we should use this thread to design and plan (logistically) the retrofit of globalization into OACS.

i am not familiar with the globalization infrastructure in OACS currently, i know of acs-reference, but not of its capabilities. why don't we start by cataloging everything that is in OACS currently and how we can use it.

Greenpeace is making heavy use of a derivative of acs-lang.  Bruno Mattarollo (currently on two days' holiday so he probably won't pop up here after all) has modified it to key messages to package and locale, rather than to language.  Keying to package allows for the generation of translator pages on a per-package basis (rather than burying them in a system-wide sea of messages with no organization).  Keying to package/page might be better in general but the current approach suits GP's needs just fine.  Keying to locale rather than language allows for the use of different dialects (American English, etc).

Also "translation missing" messages are turned into a hyperlink to an "enter translation" page if the user has privs.  I think this was part of the original acs-lang, but it's possible Bruno added it.

Only user-visible pages are being internationalized.  Admin pages specific to GP as well as standard ACS admin pages aren't internationalized.

Lars Pind added a "trn" tag to the template system and implemented a simple "#msg_key#" replacement syntax similar to the "@tcl_var@" construct.  This means that rather than tediously call "_" ( "[_ language_id msg_key]") and reference the result via @...@, you can just reference the key directly.  I modified his first "trn" implementation to allow variable substitution, so you can use it (for instance) with multirow queries that pull message keys from the db.

I've done a bunch of work to track current language and nationality via smart URLs and (if the user has them enabled) cookies, but this is more specific to GP's needs than the previous two items.

It all works quite well.  acs-lang caches a message-key/message pair, so it's fast.  The problem, of course, is that there's one more layer in the mix.  Tcl->Adp layout->message contents.  What was aD's solution for ACS Java, Yon?

When all is said and done, Bruno's anxious to get his modified acs-lang back in the mix.  It will take some work because to some degree it is specific to GP, and of course we would want some public discussion to see just how folks might want this to work in a more generalized context.  My guess is that it will be late summer before there's time to do this, but I can't really speak for Bruno.  Maybe he has Secret Plans to add more hours to the day and do it earlier :)

Lars's changes can go in at any time.  He already started a thread on a new form of the "include" tag, and was going to add it to the tree.    If this is a good time to discuss his "trn" and "#msg_key#" template extensions, let's do it.  They can go into the development branch of the tree at any time.

In the GP case, then, pages are designed as Tcl->Adp layout->message entities.  Standard OpenACS package pages aren't constructed this way.  Should they be?  Should the standard package Adp templates be simple Tcl->Adp entities as they are today, with alternative Adp templates made available as replacements?

You also need a useful discussion on configuration for multiple languages. There have been several threads concerned with this. Questions are:

• How to get your database to handle multiple languages?
• How to configure AOLserver to handle multiple languages?
• How to setup a testing environment to find bugs?

I have started to work on these issues because I have a site that uses Italian, and another that wishes to use Farsi, running from the same AOLserver instance.

In my case using the UTF-8 encoding of Unicode seems the only possibile solution. I didn't have any trouble configuring PostgreSQL to handle Unicode. I followed the advice on other threads for the configuration. Also, AOLserver was easily configured, I think. The main issue so far is setting up a testing environment to actually verify that these two big chunks of software are actually working.

To help, I have started to collect a few test pages. These are currently grouped at http://zmbh.com/utf-8/.

Here is a description of a few of these:

• http://zmbh.com/utf-8/UTF-8-test.txt tests a bunch of stuff related to the UTF-8 parser. You can use this to test if your display's parser has any bugs. So far I haven't found a web browser that doesn't have a few, but this doesn't necessarily mean that you cannot view correct UTF-8 characters, only that incorrect characters are not handled correctly. This could lead to security problems. The only display that I have working is an xterm in -u8 mode. You probably need something similar to RedHat 7.0+, then you need to install 10646-1 fonts and start xterm with a command similar to:

  LC_TYPE=en_US.UTF-8 xterm 
   -fn '-Misc-Fixed-Medium-R-Normal--15-140-74-75-C-90-ISO10646-1'
  

  The UTF-8 test file has the feature that if your parser is working, each line of the file is 79 chars plus a newline. The 79th char is '|', so you get a nice line down the right side that should line up.
• http://zmbh.com/utf-8/utf8.html has many languages on it and can be used to check if the font you are using supports the language you want. Not every font supports characters from every language. A correct parser will 'replace' characters it does not have a glyph for with it's replacement character. Sometimes this is a question mark that looks a little weird, or an upside down question mark, or in the case of xterm, an dotted outline of the boundary of the character. It is not a bug in your parser to show these replacement characters, it is a bug if other strange, obviously non-language characters show up. It means your parser did not correctly find the multibyte character.

  It might still be possible that something along the line mangled the file. What seems to work for me is to download the file with wget and cat the file with the correctly configured xterm.

• http://zmbh.com/utf-8/fconfigure.tcl opens the utf8.html file and configures the channel and reads the data into a string. It then uses ns_return to return the string. I used wget for this file as well and then used diff to figure out why the lengths were different. Everything was the same except the Vietnamese line. Maybe there is a bug in ns_return?

Tom mentions UTF-8 in an OpenACS site. I've been running a site with OpenACS 3.x in UTF-8 for the last 6-8 months (yad2yad.huji.ac.il); one of the requirements was that it work in Hebrew, Arabic, and English.

Getting PostgreSQL to work in Unicode wasn't hard at all; just pass the --encoding flag when you create the database. And once I put in the HackContentType flags described in someone's posting, the entire site worked just fine without modification. We're using the news, bboard, and chat modules in UTF-8, and everyone is pleased and impressed.

The few problems that I had were:

• Getting ns_sendmail to work correctly with UTF-8. I ended up modifying modules/tcl/sendmail.tcl to encode e-mail in UTF-8.
• Making sure that HTML forms would work for input in UTF-8. This normally happens if the encoding is set correctly, but testing this and double-checking that every page had the right content-type was tough.
• Here and there, people have been having weird problems with encoding that we can't easily duplicate. The data looks close enough to Windows-1255 (i.e., Hebrew and English) that I suspect user error, but that's a pretty lame excuse when your users are elementary school students.

I haven't yet had a chance to look into i18n and Unicode issues in OpenACS 4.x; that's one of next week's challenges. Now that I think about it, I wouldn't mind seeing a global parameter that sets the encoding for pages and for outgoing e-mail.

And of course, none of what I've written here is true i18n; it's just a matter of ensuring that the right text can potentially appear on the screen.

Thanks for an interesting piece of research. Clearly there's a need to standardize.

Would it make sense to do the pretty-formatting in the web server layer? That way your pretty-printer-proc will automatically have access to the user locale. And it's possibly easier to do more advanced stuff, such as being able to say "Today at 10:32" or "Yesterday", when that's when it was.

The DG provides one interesting feature that I don't think was mentioned above. We distinguish between publisher-added "navigational" text and user-added content (a couple of our custom packages support language tagging of content).

In a few cases, we bypass the TRN tags and use language-specific ADPs, switching the target of ad_return_template based on the user's setting. This is useful for pages containing HTML forms that we don't expect to change much.

This is tied to a GP-specific "homepage" package which is mounted for each of the national organizations that are participating. So it's not useful in the general context of OpenACS 4. However it's given me ideas for implementing similar functionality bsaed on acs-subsite mount-points. Currently it's easy to change the look-and-feel of a subsite via the master template, which lets you specify a CSS stylesheet, of course. But beyond that we have no mechanism to customize one subsite's look and feel, and this gives us one.

It's a bit off-topic but in Greenpeace's case the need is driven by the nature of the organization - pne umbrella international and individual national orgs each with their own opinion of what their website should look like. This is probably not unusual in the world of international NGOs so vaguely relates to "globalization".

Tnx