Passing a custom message from Websocket API Lambda authroizer to the client

My websocket API has a custom authorizer, and I want to transmit a message to the WS client (ie. browser) when a validation is failed. For an example, if IP address is invalid, I want to send an object to the client's "onerror" or "onclose" event handlers. I have enabled route-responses for my $connect route.

The $connect route is integrated to a Lambda. I managed to transmit the error string from the authorizer to $connect handler using the context field in the returned deny-policy. And then I tried returning an object like follows from the handler, hoping it would reach the client's event handlers.

export const handler = async function (event, context) {
    try {
        if (event.requestContext.authorizer.rejectCode === 'INVALID_IP') {
            return {
                statusCode: 403,
                body: JSON.stringify({ message: "Connection accept failed. Invalid IP" })
            };
        }
        return { statusCode: 200, body: JSON.stringify({ message: "success" }) };
    } catch (error) {
        return { statusCode: 500, body: JSON.stringify({ message: "Connection failed " })  };
    }
}

But none of these returned values are reached to the client. I get an event object similar to follows on the client "onerror" and "onclose" events.

{
    isTrusted: true
    bubbles: false
    cancelBubble: false
    cancelable: false
    code: 1006
    composed: false
    currentTarget: WebSocket
    defaultPrevented: false
    eventPhase: 0
    path: []
    reason: ""
    returnValue: true
    srcElement: WebSocket
    target: WebSocket
    timeStamp: 40664.800000000745
    type: "close"    // or  "error"
    wasClean: false
}

I feel like I'm missing something basic. Any help is appreciated.