What does the `ssl_encryption` parameter to `mysql.rds_set_external_master` actually do?

We finally know the answer to this question. In short, the MySQL and RDS docs are quite confusing about this, and there's a difference in behavior between Aurora 2 and 3 that makes this even more confusing.

1. The ssl_encryption parameter to mysql.rds_set_external_source does 3 things under the hood:

   • It enables (and requires) a SSL connection to the source database. In other words, setting SOURCE_SSL=1 with MySQL's CHANGE REPLICATION SOURCE TO statement.
   • It enables verification of the source databases’s certificate. In other words, under the hood, this is threading ssl_ca rds “binlog rds ssl material” (as set by RDS's mysql.rds_import_binlog_ssl_material) through to the SOURCE_SSL_VERIFY_SERVER_CERT and SOURCE_SSL_CA* parameters for MySQL's CHANGE REPLICATION SOURCE TO statement.
   • It enables MySQL client certificate based authentication (as described in this MySQL blog post). This is a problem, as we do not have the power to generate a valid client certificate for RDS instances.
     • In other words, under the hood, this is threading the ssl_key and ssl_cert rds “binlog rds ssl material” through to the SOURCE_SSL_KEY, and SOURCE_SSL_CERT parameters for MySQL's CHANGE REPLICATION SOURCE TO statement.
     • Can we get a valid client cert? No. This SO post from 2018 says AWS doesn’t support it and it’s on the backlog. We filed a support ticket, and they said "RDS/Aurora does not support logging in via certificates".
       • We tried generating a cert ourselves (following the instructions here), and that unsurprisingly didn’t work. The source database rejects the connection because it doesn’t trust the cert being presented.
     • Does MySQL encrypted replication really require using client cert based authentication? No. We know this both from manual testing with MySQL 8 running on our laptops, and from the fact that we were able to do encrypted replication on Aurora 2 (perhaps unintentionally on AWS’s part). If we could just run the underlying CHANGE REPLICATION SOURCE TO command ourselves, we could configure this correctly. However, the the admin user on Aurora RDS doesn’t have permission to do this.
       • We even tried to get clever by doing this sequence of operations:
         • Call mysql.rds_import_binlog_ssl_material
         • Call mysql.rds_set_external_source(..., ssl_encryption=0)
         • Call mysql.rds_remove_binlog_ssl_material
           • This just results in a broken replication configuration (it’s pointing at certs that no longer exist on disk).
         • In January 2024, we asked AWS if there’s some way of enabling SSL without enabling client certs, and they said no. (Note that as of March 2024, this is now possible. Read on for more details).

2. On 2024-03-07, Aurora 3.06.0 was released, with a new mysql.rds_set_binlog_source_ssl setting: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/mysql-stored-proc-replicating.html#mysql_rds_set_binlog_source_ssl, which corresponds to just the SOURCE SSL setting for MySQL's CHANGE REPLICATION SOURCE TO statement.

   • This does exactly what we want it to: we call mysql.rds_set_external_source(..., ssl_encryption=0) and then we call mysql.rds_set_binlog_source_ssl(1), and we are able to do encrypted replication. Make sure you do it in this order, as mysql.rds_set_external_source will clobber the Source_SSL_Allowed setting. Since this is so stupid and easy to get wrong, just confirm that the replica is trying to connect with ssl by running this command:

     mysql> show replica status\G
                ...
                Source_SSL_Allowed: Yes
                ...
     1 row in set (0.01 sec)