If your mail server uses a self-signed certificate and you cannot change it, you can force Joomla to skip the certificate validity check.

Note: Joomla does not have a toggle in the dashboard for this, so it requires a minor file edit. This will be overwritten during Joomla core updates.

open the file: /libraries/vendor/phpmailer/phpmailer/src/PHPMailer.php

Locate the function where the SMTP connection is configured, or add a custom default configuration array for SMTPOptions. You want to inject this rule into the PHPMailer instance:

$this->SMTPOptions = array(
    'ssl' => array(
        'verify_peer' => false,
        'verify_peer_name' => false,
        'allow_self_signed' => true // not necessary unless you have a self signed certificate
    )
);