I had this problem also. I use subsites to represent companies and I needed different contact info for a person by company.
It seems like the rels interface would be better for contacts. Then you could add a contact to any object and you could add a person object as an attribute on contact.