Assuming "out of office" messages always to go the "From" address and replies always go to "Reply To" it shouldn't be too hard to handle that particular issue.
Still the lack of any security in the smtp protocol and email makes putting words in someone else's mouth too easy. Perhaps the "Reply-To" address should have a special encoding that only works with messages sent "From" the address that message goes to.
