The commands look fine; i tested just now
set instance_id [ad_conn package_id]
set id [application_group::new -package_id $instance_id -group_name "test 1"]
which returns 1508. In the DB i see
oacs-5-9-1=# select * from application_groups;
group_id | package_id
----------+------------
-2 | 639
1502 |
1508 | 1110
(3 rows)
so, all is fine. If you look at the error message, you have posted, there is the following line:
ERROR: duplicate key value violates unique constraint "application_groups_package_id_un"
... which means, there is already an application group with the same package_id in your system - there can be only one application group per package_id; check in your DB...
all the best