Forum .LRN Q&A: Has someone done a comparision of dotLRN with other commercial systems?

1: Has someone done a comparision of dotLRN with other commercial systems?

Posted by Mohan Pakkurti on 07/03/02 06:26 PM

hi!

I am wondering if any one in the community has done comparisions
of dotLRN with other commercial systems. It seems there are quite
a few commerical systems, and it would be nice to have a review
or comparision in terms of features, usability, PRICE, etc!

Does anyone already have this information? or bits of it.

Maybe I can compile and summarize the information to a document
if people can post some experiences and pointers.

Cheers.

2: Response to Has someone done a comparison of dotLRN with other commercial systems? (response to 1)

Posted by Alfred Essa on 07/04/02 02:53 AM

Dear Mohan:-

We have done some competitive analysis and plan to post it on the dotLRN web site in August. If you (or others) are willing to assist with this effort, please let me know. Our goal is to try to stay as objective as possible so that people can make the right choice. It would be particularly useful for us to agree on a taxonomy for doing the comparison.

Best,

Al Essa

3: Response to Has someone done a comparision of dotLRN with other commercial systems? (response to 1)

Posted by Ben Koot on 07/04/02 02:19 PM

Not a comparisson, but maybe interesting info re further chances for opensource initiatives...

Like a smart kid who lacks motivation, e-learning software stocks have yet to live up to their considerable potential. The corporate IT spending recession has saddled these companies with project deferrals, a drought in new deals, and reduced scope for existing deals. The result has been revenue shortfalls and deferred promises of break-even quarters. Most analysts have soured on the industry short-term, but many remain convinced that its long-term future remains bright.Full article

4: Response to Has someone done a comparision of dotLRN with other commercial systems? (response to 1)

Posted by Michael Feldstein on 07/04/02 04:35 PM

It's critically important to go deeper than high-level feature comparision lists and look at the quality of the implementations (from the user's perspective, not the programmer's). The game in e-learning infrastructure sales these days is to come up with a long bulleted list of features, even if they are so poorly implemented that they are unusable.

I recently wrote an article discussing this problem specifically for learning content management systems:

http://www.elearnmag.com/subpage/sub_page.cfm?section=4&list_item=4&page=1

We need to do similar analyses for all areas of core funcitonality in dotLRN.

5: Response to Has someone done a comparision of dotLRN with other commercial systems? (response to 1)

Posted by Alfred Essa on 07/06/02 12:30 AM

Functionality is not enough. Base architecture is also not enough. dotLRN will not get penetration in either higher education or the commercial sector unless it can fit into the enterprise. What do I mean? For example, authentication needs to be delegated so that it can fit into an LDAP, kerberos, PKI type infrastructure. Similarly, dotLRN will have to make it easy to hook into back-end administrative systems such as Registrar to get class lists, etc. I hope that OpenACS community will begin exploring these larger enterprise architecture issues....

6: Response to Has someone done a comparision of dotLRN with other commercial systems? (response to 1)

Posted by Michael Feldstein on 07/06/02 12:40 AM

Functionality is not enough. Base architecture is also not enough. dotLRN will not get penetration in either higher education or the commercial sector unless it can fit into the enterprise. What do I mean? For example, authentication needs to be delegated so that it can fit into an LDAP, kerberos, PKI type infrastructure. Similarly, dotLRN will have to make it easy to hook into back-end administrative systems such as Registrar to get class lists, etc. I hope that OpenACS community will begin exploring these larger enterprise architecture issues....

The LDAP/Kerberos stuff is definitely essential if any OpenACS-based app hopes to get traction in vertical markets of any size. The registrar integration, on the other hand, is a dotLRN-specific need (though a very real one) and should be handled using the IMS Enterprise Specification (http://www.imsproject.org/enterprise/index.html).

We have a huge amount of work to do on the standards and interoperability front.

7: Response to Has someone done a comparision of dotLRN with other commercial systems? (response to 1)

Posted by Malte Sussdorff on 07/06/02 12:55 AM

Quick question. Whats so difficult about LDAP, that was not solved in ACS 3.5 already? Using ns_ldap it is fairly easy to allow access through an LDAP server. It only starts to get interesting if you want to have single signon (any clues here?).

8: Response to Has someone done a comparision of dotLRN with other commercial systems? (response to 1)

Posted by Carl Robert Blesius on 07/06/02 04:24 AM

After talking with Lars and Don about the authentication problem it is obvious that this is something that is needed in the general toolkit. We need something that is independent of the authentication scheme. We need something that allows us to plug in different "authentication modules" as needed. Something like what has been done for the different payment gateways in the ecommerce module.

For our institution we need to authenticate against a different datasource if the primary authentication method should fail. For instance, if the user is a student in Heidelberg, he will be authenticated using RADIUS server because he already has an account in the computer center. If a users is a physician in a teaching hospital 50 miles away, he will not be in the RADIUS datasource, so the standard OpenACS method should be used.

Another reason we would need the modular scheme is the fact that we will be migrating to LDAP in a year or two. All we want to have to do is replace the RADIUS plug with the LDAP plug.

I am sure there are other institutions that would like have multi-tier fallback authentication (not just the two tiers described above).

9: Response to Has someone done a comparision of dotLRN with other commercial systems? (response to 1)

Posted by Carl Robert Blesius on 07/06/02 08:45 PM

Mohan,

Now I am going to try to address your original question: we are in the process of completing a comparison of various commercial and non-commercial systems. Our experience has shown that this is not a very easy process, because the results become stale very quickly and it is very hard to do much more than superficial testing on most systems.

More interesting are the criteria that one uses to make the choice as well as the quality of the implementation of the various functions from the user's perspective (as Michael mentions above... which we have learned the hard way). I will try to sort out some of the criteria we are using right now and will post them here for discussion as soon as I can. Obviously each entity considering dotLRN will place a different emphasis on the various criteria, which is why providing a framework to help decide is probably more useful than an actual comparison (which is useless in a couple of months anyway, due to the constant flux in the marketplace).

The fact that most of the documentation on dotLRN has not appeared yet makes it hard for someone that does not know the history behind dotLRN to make an informed choice. Here is the document that originally woke my interest in ACES many moons ago: http://web.archive.org/web/20010405233725/www.arsdigita.com/doc/overviews/education.html

I know it is very old and not all of the original ideas have been implemented, yet it does provide a good starting point to read about the original goals behind dotLRN. This is the kind of documentation that is missing right now for dotLRN (this is not just a problem that exists in the dotLRN sphere but on the OpenaACS site as a whole... we need something like the old asj). This kind of documentaion is critical for the future of the platform. Having clearly defined goals will help prevent dotLRN drift (which I feel is a major problem with a lot of opensource projects).

The major strengths of dotLRN seem to lie in the philosophy and people behind it, its modularity, easy extendibility, its proven solid architectural foundation, all the packages that are just waiting to get ported from ACS and OpenACS, and if something is not working the way it should you can have it fixed or fix it yourself. As soon as MIT, Openforce, and Berklee pull this off others will be able to take the stick and keep on running (while they continue to work as well), building on top of it without having to worry to much about scalability. The money that is saved on licensing can be put into development. It seems that Blackboard and WebCT both now realize the development potential their customers have and have reacted by creating sandboxes for their customers to play (Blackboard Buildingblocks and WebCT Vista's Software Development Kit).

10: We don't need no steenking enterprise system (response to 1)

Posted by Rich Graves on 07/08/02 04:04 AM

Kerberos and PKI and proper LDAP and ERP integration are all well and good for MIT and the few dozen schools of similar size with similarly professional IT staff and existing investment in enterprise systems, but I see a lot of value in software that an undergraduate programmer can install on their own Linux box and become productive with in less than a year.

WebCT is losing a lot of customers by going the enterprise route with Vista. There is no way we are going to pay $200K+ for unproven software. Based on customer reaction, WebCT recently announced that they were going to continue accepting on the neighborhood of $7K for their current Perl product after all. Now, we want to get away from WebCT because we don't think their abandonware is worth $7K, but the fact that they were forced to revoke their "Vista or the highway" decision tells you something.

I *do* agree that it is critical to interoperate with general campus identification/authentication systems and course registration systems in some well-defined way. Many Brandeis professors requested WebCT courses last semester merely because that was a (mostly) supported way of feeding dynamic course lists to them.

I don't know how important IMS is. A clear API and/or something as simple as CSV import/export could do just as well.

I do think that the central USERS table (acs 3) and PARTIES (acs 4) is a problem for integration. That's one of the reasons we're not rushing into dotLRN.

We will see how things go next semester, but I suspect that our most popular course management system this year might become Sympa (sympa.org), a mailing list manager with file upload (think Yahoo Groups) and LDAP/database integration features. The database integration is very simple-minded -- basically jut two denormalized tables -- but it gives us a clear way to authenticate users to LDAP and keep course lists in sync with our legacy student registration system.

Most professors don't want to mess with online quizzes and the like. They don't care about web design. They just want an easy-to-use place to share documents with their students. They don't want to have to add/drop people; that's the registrar's job. Yes online grading would be *nice*, especially in a few large lecture courses, but not really critical.

Having ACS delegate username/password or ever X509 cert authentication to another system is trivial -- just a few-lines change to register.tcl. On my.brandeis.edu we just fork off a perl script. Cheesy, but why bother with ns_ldap if you're only going to use it once per login. Now, if you're going to replace the entire USERS/PARTIES system so that you can expire/rename users based on enterprise data -- but that's not ns_ldap, that's a fundamental redesign of the data model and potentially thousands of lines of TCL.

11: Response to Has someone done a comparision of dotLRN with other commercial systems? (response to 1)

Posted by Ben Adida on 07/08/02 06:52 AM

Richard,

Any system that needs to identify users in a unique manner is
going to have a central users and/or parties table. I agree that
integrating with external authentication systems is a very useful
feature, but I disagree that a central users table stands in the
way of implementing such a feature.

Bottom line: there's nothing in dotLRN/OpenACS that prevents
external authentication as long as each user can be identified by
an email address (which is a small requirement to make for
someone who is online already). It's just one more feature to
implement.

12: Response to Has someone done a comparision of dotLRN with other commercial systems? (response to 1)

Posted by Gustaf Neumann on 07/09/02 01:11 PM

We have OACS running with here with kerberos authentication.
it consists of a small c-stub for the aol-server to delegate
oacs authentication to kerberos, and a modification (for our purposes)
of the oacs login screens. If there is enough interest, we can
provide a small package (depending on the policy, changes
will be necessary; we allow kerberos and non-kerberos authenticated
users...)

-gustaf neumann

13: Response to Has someone done a comparision of dotLRN with other commercial systems? (response to 1)

Posted by Lars Pind on 07/17/02 04:44 PM

I asked a friend of mine, who's using Blackboard at the school she's at, what their problems with it are. Her response totally confirms what Michael is saying above, that you have to dig deeper than the feature lists to judge the systems. Here's what she had to say:

There's no way to figure out what's happening without viewing 6-7 different pages (announcements, dicsussions, files, yadda...) in each of the courses you're following. So people rely on email instead of the online forums, for fear that nobody's going to hear what they're saying.
You can't see who's online
No web mail
The home page lists all courses she's ever taken -- no way to see only the courses she's taking this semester.
You have to enter personal data (phone number, address, picture) for each course you're following, so nobody does that.
You can't change your email address
You can't get your schedule into the calendar, mostly because you can't create recurring events, so you'll have to create every single lecture. So nobody's using the calendar
You can't save your chat room session, so nobody's using that for their team meetings.

In short, while it all looks slick and cool, it doesn't work so well in practice. Of course, these are things that they could potentially fix, but it also sounds like they're not really paying attention. Don't know.

Can anybody else help us substantiate the differences, or the deficiencies of other systems, and consequently dotLRN's selling points :)

/Lars

14: Response to Has someone done a comparision of dotLRN with other commercial systems? (response to 1)

Posted by Torben Brosten on 07/17/02 10:50 PM

I've used WebCT for a few years (as a student). It doesn't seem to scale well.

1. With more than just a few users, the system drags or posts are lost.

2. After each posting to a forum, all messages would be listed (unnecessarily putting a load on the server).

3. The UI for browsing multiple forums was awkward.

4. posted messages were viewed inconsistently by other browsers because of the limited use of html style tags permitted in the postings (due to poor use of css definitions).

The chat function was difficult to use... limited number of characters per post, messages scrolling off of the page were difficult to retrieve, no provided method to keep a transcript (though instructors get one), names were not changeable. Often long names were provided in the system, thereby causing extra linefeeds (and faster scrolling). Text sizes not changeable etc. ... perhaps this was because I was using a Mac???

Students could post info/notes/events to the calendar, but if one were to view the calander for printing (as a sequence of events), only the instructors notes/events appear!

Apprently, no wimpy-point for instructors to create presentations. Presentations mostly came from wordprocessors (such as MSWord) where the exported html was not web-compliant browswerfriendly.

Hmm.. I'll let you know if I think of anyting else.

15: Response to Has someone done a comparision of dotLRN with other commercial systems? (response to 1)

Posted by Torben Brosten on 07/17/02 11:27 PM

How could I forget?!?

Noone could refer to other messages or threads using urls! Only message numbers etc. which gets to be cumbersome about the time one wants to use them (threads get long).

The site urls could not be bookmarked --WebCT uses relative javascript-created urls and lots of pop-up windows (as well as frames). Javacript is required.

I don't recall having any option to use email instead of the web-based forum services. Perhaps this is a sysadmin option?

Had to enter the site everytime through the same homepage url ie. not a multi-doored site. Could not stop at a place, then come back later to it directly... had to always login via the homepage first... (or else certain unspecified data *useful to your grade* and instructors would not be tracked).

16: Response to Has someone done a comparision of dotLRN with other commercial systems? (response to 1)

Posted by Torben Brosten on 07/17/02 11:39 PM

One nice feature of webct boards... urls were automatically converted to links... (would be a nice feature here.. =)

17: Response to Has someone done a comparision of dotLRN with other commercial systems? (response to 1)

Posted by Michael Feldstein on 07/17/02 11:58 PM

When I evaluated Blackboard, WebCT, and Prometheus, I found all three to be pretty weak. Blackboard was the worst by far. WebCT had a huge number of features, but the seem to have been thrown in willy-nilly with very little intelligent integration. Prometheus was the best of the three in that the design seemed to value quality of implementation over number of features, but it had a lot of weaknesses too. It had relative links, making tie-ins to specific threads impossible, and no decent user profiling capabilities. (I seem to recall a problem with email alerts too, but it's been a while since I looked at it.)

Anyway, Prometheus was bought by Blackboard, so I don't know how much life it has left in it.

18: Response to Has someone done a comparision of dotLRN with other commercial systems? (response to 1)

Posted by Lars Pind on 07/18/02 11:40 AM

Torben,

One nice feature of webct boards... urls were automatically converted to links... (would be a nice feature here.. =)

I implemented this for ACS 3.4, and it's been part of ACS 4 from the start. I even offered to back-port it to OACS 3.2.5/openacs.org back in February, but it was deemed a waste of time, since the new OACS 4 site would be up soon, anyway :)

19: Response to Has someone done a comparision of dotLRN with other commercial systems? (response to 1)

Posted by Ben Koot on 07/18/02 01:59 PM

Lars,

It may be a waste of time from a development point of view, yet I have a feeling the majority of active sites at the moment are not ACS 4. I think untill there is a stable edition, with matching / better functionality and we have a simple way of migrating from 3-4, many folks will simply stick to their outdated edition. From that point of view something so basic, would realy be very helpfull I feel

20: Response to Has someone done a comparision of dotLRN with other commercial systems? (response to 1)

Posted by Lars Pind on 07/18/02 03:56 PM

I'd be willing to put in the time to dig it out and put it in for the sake of openacs.org. Now that a new 4-based openacs.org effort is seriously under way, I can't commit the time.

But if anybody working on an OACS 3.2.5-based site wants to implement it, it's really easy.

I managed to find an old ACS 3.x tar-ball, and in there you'll find:

packages/acs-core/text-html-procs.tcl

which contains all the procs. Then all you need to do is add calls to ad_convert_to_html at various places in www/bboard/*.tcl:

confirm.tcl:[ad_convert_to_html -html_p $html_p -- $message]
fetch-msg.tcl:[ad_convert_to_html -html_p $html_p -- $message]
grep: graphics: Is a directory
post-reply-top.tcl:[ad_convert_to_html -html_p $html_p -- $message]
q-and-a-fetch-msg.tcl:append whole_page "[ad_convert_to_html -html_p $html_p -- $message]
q-and-a-fetch-msg.tcl:    append this_response "[ad_convert_to_html -html_p $html_p " $message"]
grep: text: Is a directory
grep: unified: Is a directory
usgeospatial-fetch-msg.tcl:[ad_convert_to_html -html_p f $message]

I think that's it! If something doesn't work as expected, feel free to ask.

21: Response to Has someone done a comparision of dotLRN with other commercial systems? (response to 1)

Posted by Torben Brosten on 07/18/02 09:29 PM

Lars, Thank you for the correction. I'm looking forward to experiencing autoconversion of urls to links in board posts for openacs.org.

Clarification regarding lost posts in WebCT...

..occuring during heavy server loads. One could click the "post" message to board button, but the server might not respond for minutes (3, 5 or more). Clicking twice sometimes resulted in double (or more) posts. Using WebCT during busy periods could result in lost posts or multiple posts of the same message. The behavior was frequent enough for students to avoid busy times whenever possible --distributing server load through user-intervention.

22: Response to Has someone done a comparision of dotLRN with other commercial systems? (response to 1)

Posted by Michael Feldstein on 07/18/02 10:29 PM

Up until very recently, WebCT wasn't even built on a relational database. So it wouldn't surprise me to hear that it had/has ACID-compliance problems.