By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Aurora/Mysql cross region replication failing due to SSL cert problem

0

We have an Aurora/Mysql database is in US-GOV-WEST, and was replicated using cross region replication to US-GOV-EAST. We recently had to do a blue/green deployment of the master in gov-west, so we promoted the us-east to be standalone, then deleted it.

Once we were done with the blue/green deployment we attempted to create a net gov-east cross region replica. It successfully creates the replica, but then the gov-east replica is unable to connect to the gov-west master. Show Replica status shows error connecting to master 'rdsrepladmin@10.8.15.0:3306' - retry-time: 60 retries: 53 message: SSL connection error: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed

which implies that the master in gov-west has a bad SSL cert somehow.

We've tried doing this several times, as well as restoring from a snapshot, but all snapshots after the blue/green deployment have the same issue.

This is done between 2 Aurora mysql, with RDS controlling the replication, so how can the SSL cert be bad?

Is there any way to either turn off SSL authentication between master and replica? Or can we reset it somehow?

Our last resort is to use DMS to copy just our tables to a new database, but that's more risky than somehow fixing the replication.

  • Steve_M EXPERT
    8 months ago

    It would be interesting to see the cert that the gov-west master is presenting. Is there an EC2 instance on which port 3306 is open between it and the master (and if there isn't then could you quickly and temporarily provision one?) on which you could run openssl s_client -showcerts -connect 10.8.15.0:3306

  • Andrew Barton
    8 months ago

    Steve_M - So I think that the 10.8.15.0 IP is an internal Aurora IP, all my replicas are saying that they're replicating off that, regardless of what server they're actually replicating. Maybe it's some intermediate server?

Topics
Database
Tags
Aurora MySQL
Language
English
Andrew Barton
asked 8 months ago380 views
1 Answer
0

There will be more risks to turn off the SSL authentication between your databases. I would suggest you to not do that. But you can check whether you have any security policies in place that require SSL connections ? if so, you need to update those policies.

Try these steps instead,

  • make sure your SSL Certificate is installed correctly on the client application
  • if the SSL certificate is valid and installed correctly and still you are getting the error . Try these steps Restart the client application, clear the SSL cache on the client application, update the application to latest version

Additional things to be remembered while troubleshooting SSL certificate issues:

  • SSL certificate must be installed in the correct location on the client application
  • firewall or proxy server between the client application and the Aurora MySQL cluster must not be blocking the SSL connections
Subbusainath Rengasamy
answered 8 months ago
  • Andrew Barton
    8 months ago

    This is NOT the SSL connection between a client application and the Aurora/Mysql server, but between two Aurora/mysql servers using cross region replication. Aurora/Mysql does not give any manual control over any of the SSL certificate. I don't know what security policies could break that, considering I have no control over the security policies, those are all controlled internally by Aurora/Mysql.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content