As you know, to connect to a Postfix email server requires
Not necessarily. It depends on the setup of Postfix. In our university setup, we use postfix as a forwarder to the university mail infrastructure (where the server is whitelisted - I think). This is a common setup with very little configuration work.
my reference: https://bitbucket.org/naviserver/nssmtpd/src/main/
Don't use this. A few years ago, all NaviServer repositories were moved from Bitbucket to GitHub (https://github.com/naviserver-project/nssmtpd) which is also used from install-ns, when you install from the repository.
what pem file should we be using in the ns_param certificate?
You do not need to set this parameter. It is required, when you use nssmptd as a mail host for incoming mail requests performing StartTLS. The parameters "cafile" and "capath" are required for validating server certificates presented to the nssmtpd. These parameters are described in technical detail on the OpenSSL pages. You find a configuration description for NaviServer as used in ns_http requests in [1]. The upcoming release of NaviServer 5 will be secure by default, which means that for outgoing HTTP requests, certificate validation will be turned on by default, where these parameters will become more important in the future.
For the usual OpenACS setup, where mails are submitted via "ns_smtpd send", this is not relevant.
If time permits, I will look over the weekend into the nssmtpd module to figure out, what it means to add authentication for relay requests.
[1] https://naviserver.sourceforge.io/5.0/manual/files/admin-config.html#subsection13