Restrict AWS SSM connection to ec2 instance from VPN servers.

Attach the AmazonSSMManagedInstanceCore policy to the EC2

Attach the other policy to a user/group. you dont attach it to the EC2

What policy do you have assigned to your user? If your an administrator you will ALWAYS have access to the EC2. You will need to have a Deny instead applied to ALL users :-

I think this will only work too if you have the SSM VPC PrivateEndpoint and Private DNS Enabled along with your VPN pointing to the Route53 VPC resolver IP

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Deny",
            "Action": [
                "ssm:StartSession",
                "ssm:ResumeSession"
            ],
            "Resource": [
                "arn:aws:ec2:REGION:ACCOUNT_ID:instance/INSTANCE_ID"
            ],
            "Condition": {
                "NotIpAddress": {
                    "aws:SourceIp": [
                        "VPN_IP_1/32",
                    ]
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": "ssm:TerminateSession",
            "Resource": [
                "arn:aws:ssm:::session/${aws:userid}-*"
            ]
        }
    ]
}