Malte, some of us have use cases for "real" nested transactions. Specifically, situations where the statement "After all, when I am in a db_transaction I want to reroll everything if it fails at one point and not each transaction separately." is false.
However, the Postgres branch of the db_api currently DOES NOT SUPPORT THESE. You can nest calls to db_transaction, but they don't actually do anything to the database. So it is not really clear what you are asking for with the "do not execute another db_transaction".
It seems you have an idea that using nested transactions are bad, something to be avoided. In my mind, they should be correctly implemented and used.
I am opposed to your idea of letting the caller decide whether he wants a transaction or not. This is an implementation decision, and shouldn't be exposed in the api.