In fact, there is no application group for my main site. In the failed attempts above, I was trying to create a new subsite instance and mount it on a node (4904) directly under the main site.
dev=# select * from application_groups;
group_id | package_id
----------+------------
3127 | 3221
3160 | 3254
(2 rows)
dev=# select node_id, name, parent_id, object_id from site_nodes;
node_id | name | parent_id | object_id
---------+------------------------+-----------+-----------
2603 | | | 2566
2604 | acs-admin | 2603 | 372
2605 | acs-service-contract | 2603 | 516
2606 | acs-content-repository | 2603 | 1068
2607 | acs-workflow | 2603 | 2547
2608 | doc | 2603 | 1356
2609 | api-doc | 2603 | 411
3218 | subsite1 | 2603 | 3221
3251 | subsite2 | 2603 | 3254
4904 | blah | 2603 |
(9 rows)
In case it's relevant, this instance was originally installed from code checked out around Feb 4, 2003 (I'm pretty sure it was right around 4.6.1 beta 1).