Dave,
the contacts package requires the packages you mention. So I know contacts builds on top of them.
It is not address and phone alone that I would need. But the main thing is to understand why the contact package should restrict itself to parties/persons? Do you know? The person itself is sometimes less relevant. Often, entering a contact person for a organizations or business is optional. So, why should an organization be a party (this has been discussed, I know) ?
Within openACS it is also about UI, but not UI alone.