I think the Package Developer initially determined which things needed to be ACS Objects. I don't think you can just decide that you don't want them to be, and delete them, or am I missing something here? Your performance is going to be more related to the number of permissions checks per page, rather than the number of objects.
Some packages over use permissions/privileges as an index to who can view what on a given page. These pages will be very slow, and potentially slow down the whole system.
