In the newest generation of reference OpenACS configuration file [1], there is a configuration variable "smtprelay" that can be used.
You can always provide your own configuration file via a bind-mounting your own docker file by overlaying the docker-internal path (in the "volumes" section). This way, you have full freedom.
I am currently in the process of containerizing the openacs.org servers (including xotcl.org, dotlrn, former CVS viewer) into docker containers, including mail-relay and munin. When this is done, I will make it available as a reference. Updating the dockerhub docs is a pain, so it is currently not fully up to date, but keep watching, I have updated also today.
[1] https://openacs.org/forums/message-view?message_id=8728475
