AWS SAM, unable to dynamically change the CodeUri based on Variable

I have different local directories for three different environments with differnet codes. I can map all other properties dynamically except the codeUri. The current code below shows up an error "'CodeUri' requires Bucket and Key properties to be specified."

  AuditEventsToDynamo:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: !Sub 'AuditEventsToDynamo${ENV}'
      CodeUri: !Sub 'AuditEventsToDynamo${ENV}/'
      Handler: !Join [ "", ["AuditEventsToDynamo" ,!Ref "ENV",".lambda_handler"]]
      Runtime: python3.9
      Environment:
        Variables:
          DB_TABLE: !Ref DBTABLE
      Architectures:
        - arm64

The code is local, if I replace the CodeUri with a literal string, it works. Is there a workaround?