If your mail server uses a self-signed certificate and you cannot change it, you can force Joomla to skip the certificate validity check.

Note: Joomla does not have a toggle in the dashboard for this, so it requires a minor file edit. This will be overwritten during Joomla core updates.

open the file: /libraries/vendor/phpmailer/phpmailer/src/PHPMailer.php

Locate the function where the SMTP connection is configured, or add a custom default configuration array for SMTPOptions. You want to inject this rule into the PHPMailer instance:

$this->SMTPOptions = array(
    'ssl' => array(
        'verify_peer' => false,
        'verify_peer_name' => false,
        'allow_self_signed' => true // not necessary unless you have a self signed certificate
    )
);

Since the mail server is on the same local network, sending unencrypted traffic within your private LAN is perfectly secure if your network perimeter is tight.

In Joomla's Mail Settings, change SMTP Security to None and switch the port to 25 (or the unencrypted port your mail server listens on).

Ensure your mail server is configured to accept authenticated SMTP traffic without TLS from internal LAN IP addresses.