I read this question, plus the "Five 9s reliability, how would you do it?" thread, and just before that looked at JavaGroups - A Reliable Multicast Communication Toolkit for Java available at http://javagroups.sourceforge.net/
To me JavaGroups sounds pretty much like what you're looking for, it may not be AOLServer but it claims to give you features like atomicity when sending messages to multiple recipients. If you want an application server built on top of it, check out JBoss 3.0 (currently in alpha though, but they use JavaGroups for their clustering effort...)
