Unable to connect redshift from emr

Hello,

"FATAL: user "IAM:yarn" does not exist"

Here EMR is considering 'yarn' as a User which in turn trying to authenticate as DBUser using GetClusterCredentials API and add as a suffix for "IAM: ". It might be the case that if the user might not be exist or not provided in your URL. As you get user doest exist, your connectivity seems to be working fine. However, just in case you have not configured already then, You need to update the security group attached to the Redshift cluster to allow incoming traffic from EMR cluster by adding below inbound rules for EMR Master and EMR Slave Security Group and test the connectivity,

telnet redshift-cluster-1.xxxxxxxxx.xxxxxxxx.redshift.amazonaws.com 5439

Then try the following connection/compare with your connection URL,

df1 = spark.read.format("jdbc").option("url", "jdbc:redshift://<redshift endpoint:5439/<Database_Name>?user=<Admin_Username>&password=<Admin_Password>").option("dbtable", "public.category").load()
df1.printSchema()
df1.show()

If a user named DbUser exists in the database, the temporary user credentials have the same permissions as the existing user. If DbUser doesn't exist in the database and AutoCreate is true, a new user named DbUser is created. -

https://docs.aws.amazon.com/redshift/latest/APIReference/API_GetClusterCredentials.html