"is this broken, or is it my understanding of what should be happening here that's wrong?"
Unless you have spent at least two weeks looking at this code, while ignoring all other responsibilities, I would go with probably not broken, after that time, you will probably conclude the same thing, otherwise, you need to spend more time figuring it out. First you need a complete list of what you are trying to achieve via relationships (actually I make a pencil and paper drawing of objects and links/relationships). What users, what groups, what objects will attach, what privileges will apply. Most likely you will need to set this up with custom code. I don't know of any complex examples in core, so that makes it more difficult to develop.
One thing to note is that I don't think that a group-group relationship can use membership_rel, you need to create another rel_type based upon composition_rel. Don't expect to be able to use the simple membership hierarchies to achieve what you want, the data model is more generic than that and the code you are referencing is most likely a special case that doesn't apply in your situation. The only other option is that you are stuck with a fixed set of relationships, which would suck even more than having to figure it out to take advantage of it.
