The error message isn't coming from ad_form, it's coming from your code in -new_data.
So ... use db_transaction {} on_error { do something to give a smart error message }. Select to see if the object already exists, for instance. I greatly prefer doing this select statement when there's an error rather than doing so before the insert as was suggested above, because you only incur the cost of the extra db operation when you need it, i.e. when things fail.
You can also just cheat and assume the most likely cause of the error was a dupe. Give an error message saying "perhaps you used the back arrow to resubmit the form, causing a duplicate entry - here's the error the database gave us, though". Depends on how user-friendly you want to be.
There's really no way for ad_form to run the -edit_data code in this case. It uses its internal form vars to figure out what's going on.
Jade's suggest works on Oracle if updating not inserting is what you want to do. There's no equivalent in PG.
If you want to allow dupe entries, don't use the "key" feature of ad_form, and us on_submit rather than new_data and allocate a new object_id each time the form's processed.
In this case, of course, you'd need separate add/edit pages.
In some sense, using the ad_form "key" feature and its support of new_data and edit_data makes it easier to keep these pages uniform (because they're the same form) but at the expense of perhaps slightly more confusing error messages.
