Forum OpenACS Q&A: Sysadmin Headaches

Collapse
31: Sysadmin Headaches (response to 1)
Posted by good bye on
David's point 3 in his post above needs emphasising - ease of hosting. My experience of 3 tier Java/web application environments is that they are a huge SysAdmin nightmare - a whole new world of potential memory leaks, strange crashes, uptime measured in days or weeks.

I've built and had to maintain systems using ATG Dynamo/ Apache/Oracle, ACS and OpenACS, Tomcat/Apache/Oracle, and old-school Informix/Netscape server (ugh)/Java. I've also acted as the "production sysadmin" for each of these systems. Steve's point about system administration headaches is something that hasn't really been stressed enough in comparing ACS to these other toolkits.

Winamp.com uses dynamo/apache/oracle. Winamp.com has about 20 million "hits" a month. Almost every hit is someone downloading the player, something that isn't logged in the database...just in the server logs. The database is actually used minimally, since the guys who initially wrote it tried to use dynamo's built in 'relationalviews' abstraction which made it impossible to use joins. The database is primarily only used by people searching for "skins and plugins" and uploading the same. The winamp.com forums, which have a lot of db-backed traffic...are actually running on a separate machine using PHP/ MySQL and VBulletin...software that costs $99 a copy and looks like it was written in 3 days by some highschool students. The forums machine is 1 machine running apache/php4/mysql. The Dynamo system, in comparison is 8 separate physical webservers, each with 1 gig of RAM. they point to a huge database server. Each webserver has between 3-7 apache+ dynamo instances running. For no apparent reason, the dynamos will start consuming more and more ram until they all die, due to lack of memory. The on-call OPS guy then will get paged (because this typically happens at 3 am on sunday) and then the next monday will berate the programming team. 20% of my job at winamp was writing perl scripts to more efficiently dump and restart dynamos when they started acting squirrely...and remember, those dynamos were not really even doing anything besides serving up pages.

Contrasting this to ACS, at aD I helped deploy actionnetwork.org. It was a pre-acs system, and was very badly written by the people before me, and badly improved upon by myself. However, even though there was a .CSV upload page that opened and closed multiple database handles about 17 times, the system didn't ever randomly die. It would HANG during a big csv upload...but would recover when the upload finished. Note that the problem here was myself and my predecessor's poor programming, not any inherent problem with AOLserver. Even with crappy applicationlevel programming, actionnetwork.org seemed to work with up to 200,000 users, hitting the database with every connection to the website, sending massive amounts of spam email...on a single e450 with a gig of ram, and a badly configured oracle. It actually worked relatively well for months on a old crummy ultra 10. It did far more with the database than winamp.com did, and when it seemed to break, all I had to do was "restart-aolserver."

some setup notes:

ACS is a pain to set up, especially the first time. OpenACS is the same way. However, when working with say, ATG Dynamo/ Apache/Oracle, you multiply the setup overhead by at least three times.

setting up OpenACS (or OracleACS) requires the following:

  • you have to make sure your database is set up correctly.

    with oracle, this can be awful, without the right wizard. with postgres..it is relatively easy.

  • you need to set up your aolserver right, and compile the oracle driver and put it in the right place
  • you need to edit your aolserver's config.tcl script, then the acs parameters/config.tcl script
  • you need to load datamodel and bootstrap
  • you might need to do some misc stuff, like set up your ld_library_path and edit /etc/inittab
  • for development, you just edit a .tcl page. if you edit a private .tcl page, you need to restart-aolserver to initialize it
with Dynamo/Apache/Oracle you need to:
  • get the proper JDK, and set it up
  • get the proper JDBC, and make sure it is set up
  • set up your database...same overhead as ACS
  • figure out how to recompile apache to support dynamo
  • recompile apache a few times using different options (since the documentation is incomplete and incorrect)
  • set up your classpaths correctly
  • make sure httpd.conf is set up right
  • make sure your apachectl is set up right
  • make sure your dynamo startup scripts are set up right
  • edit your jdbc .properties files
  • edit your master dynamo .properties files
  • edit a few more random.properties files
  • for each query you write, you need to edit a relationalviews file
  • for each relationalviews file, you need to create and edit another .properties file
  • you need to run "beanmaker" on the relationalviews file to create some relationalviews components
  • eventually give up and figure out how to turn off the relational views and just use straight up JDBC...since it is impossible to joins with relational views
  • for each class you write, you need to create and edit a .properties file
  • write a perl script that acts as a glorified "make" to compile all your classes together, since one isn't provided with Dynamo
  • to test if your class is working right, you have to compile your whole system, stop dynamo, then stop apache, then restart dynamo, then restart apache...then hope it works.
  • after all this, you get to start designing your "users table" ...
I could go on like this, but you get the point.

to fuel the flame, I get really nervous and sad when I see supposedly bright engineers claiming their java web toolkit is something other than a way to scam bigcorps (and in the past, small startups) out of their cash. If you are going to write a "real application" (which seems to mean a compiled system component) you still do it in C and C++. Financial trading systems are not written in java (at least the ones that work, aren't). Desktop apps are not written in java. Network components are not written in java. Not to say that java doesn't have its place, it seems to work very nicely in the area it was actually designed for...embedded systems...like the NTT cell phones in Japan or in smart cards.

Note that none of these complaints should affect you, the grungy independent developer. Just figure out the right way to turn each of these situations into cash. It is still relatively easy to make a mint as a enterprise java developer...so many banks and other corporations have hugely fucked java projects right now, that if you actually know what you are doing (and most java developers DON't)...you can charge huge hourly rates to fix easy problems.

aD actually has the right idea with their java toolkit if they follow the fuck-the-client business model. If they can convince a few deep-pocketed clients to use the new java version, they will be raking in the maintenance and hosting fees. Too bad by the time they finish the "product" the microsoft marketing juggernaut will have convinced the CTOs of the world to use .NET. Sigh...

However, most of us are not aD, and if your business is building reliable sites in a short amount of time, for smallish clients...(or even big clients) I cannot recommend using any of the java toolkits I've worked with, unless your business model is to fleece the client on maintenance fees until they are out of business.

I believe this exact thread was written about 3 years ago in the web/db forum back when it was still on photo.net....because I remember typing in comments sadly similar to the ones I just typed about three years ago. It sucks to see how little anything has changed since then.

Becoming reflective for a minute, although it was kind of fun wasting a 45 minutes of my day venting my frustrations with java, I don't think this thread is going to go anywhere. No one really cares which is better, if someone is going to use java, they are going to use java, and even if Ben's paper is well written and has some concrete examples, it ultimately won't sway anyone. The real way to make people use OpenACS more, is to improve the core functionality, write some killer modules (especially ones that are not found elsewhere), clean up the UI so it looks slick, and attack markets where speed of delivery, reduced maintenance headaches, and inexpensiveness are more important factors than buzzword compliancy.