Set up WebSocket API with AWS_IAM auth using CloudFormation template

We are trying to set up a REST API and a WebSocket API on APi Gateway using CloudFormation template.

For the REST API, only the IAM user's access key and secret key would be required to instantiate a connection (using presignedURL with Signature v4). We would want the exact same authorization method for the WebSocket API.

The REST API template:

RestApi:
    Type: AWS::Serverless::Api
    Properties:
      Auth:
        DefaultAuthorizer: AWS_IAM
      (other properties)...

The closest thing we've found is to set the value of AuthorizationType inside the $connect route to AWS_IAM.

WebsocketConnectRoute:
    Type: AWS::ApiGatewayV2::Route
    Properties:
      RouteKey: $connect
      AuthorizationType: AWS_IAM
      (other properties)...

However the same authorization method does not seem to work with WebSocket API as it throws an error with message The security token included in the request is invalid.

Is there any way to set up authorization without the need of "security token"? Or, where can we gather this "security token" to pass the authorization?