I think this is an interesting idea, but I am not sure the composition model for groups is compatible and I would need to see some good research into these changes, and how they would affect existing permissions. I don't have a good explanation for why it was modeled that way, I am sure there is a mathematical explanation somewhere.
I think the problem you are having is that Group/composition cannot model a hierarchy. What you might want is a separate hierarchy that does not affect the permissions.
I have built this before. A simple mapping table with a tree_sortkey is all that is required to create a hierarchy of groups.
