Class membership quotas sound like a great idea.
Triggers on acs_rels, on the other hand, give me a bad headache reminiscent of ACS 3.x education package which had triggers all over the user table - which made modularity completely impossible in the end.
If you want to implement this using triggers, you should make sure the triggers are created and applied to tables within the same package. For example, you might add a trigger to dotlrn_student_rels. Or you can do some tricky locking using select for update when you're about to add a student. Either way, make sure it's something that is well contained to the dotLRN package itself.
