Error on changing python runtime from 3.6 to 3.9- No module named 'psycopg2._psycopg

I am trying to change the python runtime for my Lambda job from 3.6 to 3.9 , but after changing, the Lambda function stopped working giving the following error. I have also updated the lambda layer for psycopg2 to include 2.9.9 (initially it was 2.8.3) via pip install and zip and upload like the folder structure mentioned in -https://docs.aws.amazon.com/lambda/latest/dg/packaging-layers.html..but still the function is giving the error. Any idea how to resolve?

Test Event Name
testLambda

Response
{
  "errorMessage": "Unable to import module 'xxxxxxxxxx': No module named 'psycopg2._psycopg'",
  "errorType": "Runtime.ImportModuleError",
  "requestId": "a2fdf345-8a46-4fb1-8a21-658c45b06308",
  "stackTrace": []
}

Function Logs
START RequestId: a2fdf345-8a46-4fb1-8a21-658c45b06308 Version: $LATEST
LAMBDA_WARNING: Unhandled exception. The most likely cause is an issue in the function code. However, in rare cases, a Lambda runtime update can cause unexpected function behavior. For functions using managed runtimes, runtime updates can be triggered by a function change, or can be applied automatically. To determine if the runtime has been updated, check the runtime version in the INIT_START log entry. If this error correlates with a change in the runtime version, you may be able to mitigate this error by temporarily rolling back to the previous runtime version. For more information, see https://docs.aws.amazon.com/lambda/latest/dg/runtimes-update.html
[ERROR] Runtime.ImportModuleError: Unable to import module 'av-oa-moa-hana-shortages': No module named 'psycopg2._psycopg'
Traceback (most recent call last):END RequestId: a2fdf345-8a46-4fb1-8a21-658c45b06308
REPORT RequestId: a2fdf345-8a46-4fb1-8a21-658c45b06308	Duration: 1.35 ms	Billed Duration: 2 ms	Memory Size: 512 MB	Max Memory Used: 56 MB

Request ID
a2fdf345-8a46-4fb1-8a21-658c45b06308