Accessing Running Container on AWS to my local machine

I would say that given your explanation, the way to do this is to have a EC2 instance running SSM and create a tunnel that will point localhost:3306 to your container|host:3306. If you are on ECS you could look at using execute-command but that'd be mostly to execute admin tasks remotely.

You could also consider using a web based admin tool, i.e. phpMyAdmin or stuff like that so that you don't require tunneling.

But for the long run, I'd recommend to not rely on that and not to run DB containers unless you really know what you're doing.

Instead, use RDS / RDS Aurora. If costs is an issue, consider Aurora Serverless. I have an instance running for 1y+ and never paid for it (it's there for API integrations, not for actual DB work). If need be you can still tunnel to it or use other ways to manage it.