Aim: Create an EMR cluster and attach to a workspace, to use with JupyerLab.
EMR cluster created with default options: see end of this post for full description.
Creating the studio:
aws emr create-studio \ --name "Studio_1" \ --service-role arn:aws:iam::1234567890:role/service-role/AmazonEMRStudio_ServiceRole_1735929246573 \ --vpc-id vpc-0fffffffffffffffffff \ --subnet-ids subnet-01111111111111 \ --auth-mode IAM \ --workspace-security-group-id sg-094b767de0d287eb7 \ --engine-security-group-id sg-00f32b765e6a2c117 \ --default-s3-location s3://aws-emr-studio-1234567890-us-east-1/1735929246573 \ --tags Key=Project,Value=EMRStudio
Note:
- sg-094b767de0d287eb7 - ElasticMapReduce-mast er - default workspace security group (unable to post
aws ec2 describe-security-groups - post is too long)
- sg-00f32b765e6a2c117 - ElasticMapReduce-sla ve - default engine security group (unable to post
aws ec2 describe-security-groups - post is too long)
The default security groups fail on attaching the EMR cluster j-2MXE9AR80RKTV to the workspace:
Cluster failed to attach to the Workspace. Reason: Attaching the workspace(notebook) failed. Notebook security group sg-094b767de0d287eb7 should not have any ingress rules. Please fix the security group or use the default option.
If I try to remove the ingress rules, they reappear again a few seconds later. I assume this security group is managed by AWS.
I created copies of the default security groups sg-094b767de0d287eb7 and sg-00f32b765e6a2c117 in order to be able to edit the rules
- sg-094b767de0d287eb7 (workspace security group) ----> sg-0742e9251454fcb2c (workspace security group copy)
- sg-00f32b765e6a2c117 (engine security group) ----> sg-01a100c7c938f0313 (engine security group copy)
I removed ingress rules from sg-0742e9251454fcb2c.
On creating a new studio with the new groups, I get a new error:
Cluster failed to attach to the Workspace. Reason: Attaching the workspace(notebook) failed. Notebook security group sg-0742e9251454fcb2c does not have an egress rule to connect with the mast er security group sg-01a100c7c938f0313. Please fix the security group or use the default option.
I added an egress rule from sg-0742e9251454fcb2c to sg-01a100c7c938f0313 (see later - it is definitely created, as far as I can see).
However, the workspace will still not attach the cluster, and still has the same complaint.
Are the security groups misconfigured? Could you give a quick command line template how to set things up?
**I have an assignment due soon (Tuesday) and I really need to have a working Pyspark session.
Will send a donation (10 euro) to a humanitarian charity of your choice.**
[cloudshell-user@ip-10-130-85-79 ~]$ **aws ec2 describe-security-groups --group-ids sg-0742e9251454fcb2c**
{
"SecurityGroups": [
{
"GroupId": "sg-0742e9251454fcb2c",
"IpPermissionsEgress": [
{
"IpProtocol": "-1",
"UserIdGroupPairs": [
{
"UserId": "1234567890",
"GroupId": "sg-01a100c7c938f0313"
}
],
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"Ipv6Ranges": [],
"PrefixListIds": []
}
],
"VpcId": "vpc-0fada9bb798d0af90",
"SecurityGroupArn": "arn:aws:ec2:us-east-1:1234567890:security-group/sg-0742e9251454fcb2c",
"OwnerId": "1234567890",
"GroupName": "New-Workspace-SG",
"Description": "New Workspace SG",
"IpPermissions": []
}
]
}
[cloudshell-user@ip-10-130-85-79 ~]$ **aws ec2 describe-security-groups --group-ids sg-01a100c7c938f0313**
{
"SecurityGroups": [
{
"GroupId": "sg-01a100c7c938f0313",
"IpPermissionsEgress": [
{
"IpProtocol": "-1",
"UserIdGroupPairs": [],
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"Ipv6Ranges": [],
"PrefixListIds": []
}
],
"VpcId": "vpc-0fada9bb798d0af90",
"SecurityGroupArn": "arn:aws:ec2:us-east-1:1234567890:security-group/sg-01a100c7c938f0313",
"OwnerId": "1234567890",
"GroupName": "New-Engine-SG",
"Description": "New Engine SG",
"IpPermissions": [
{
"IpProtocol": "tcp",
"FromPort": 0,
"ToPort": 65535,
"UserIdGroupPairs": [
{
"UserId": "1234567890",
"GroupId": "sg-00f32b765e6a2c117"
},
{
"UserId": "1234567890",
"GroupId": "sg-094b767de0d287eb7"
}
],
"IpRanges": [],
"Ipv6Ranges": [],
"PrefixListIds": []
},
{
"IpProtocol": "udp",
"FromPort": 0,
"ToPort": 65535,
"UserIdGroupPairs": [
{
"UserId": "1234567890",
"GroupId": "sg-00f32b765e6a2c117"
},
{
"UserId": "1234567890",
"GroupId": "sg-094b767de0d287eb7"
}
],
"IpRanges": [],
"Ipv6Ranges": [],
"PrefixListIds": []
},
{
"IpProtocol": "icmp",
"FromPort": -1,
"ToPort": -1,
"UserIdGroupPairs": [
{
"UserId": "1234567890",
"GroupId": "sg-094b767de0d287eb7"
},
{
"UserId": "1234567890",
"GroupId": "sg-00f32b765e6a2c117"
}
],
"IpRanges": [],
"Ipv6Ranges": [],
"PrefixListIds": []
}
]
}
]
}
**aws emr describe-cluster --cluster-id j-2MXE9AR80RKTV**
{
"Cluster": {
"Id": "j-2MXE9AR80RKTV",
"Name": "My cluster",
"Status": {
"State": "TERMINATING",
"StateChangeReason": {
"Code": "USER_REQUEST",
"Message": "Terminated according to the attached auto-termination policy after 3600 idle seconds"
},
"Timeline": {
"CreationDateTime": "2025-01-03T18:27:03.498000+00:00",
"ReadyDateTime": "2025-01-03T18:32:26.247000+00:00"
}
},
"Ec2InstanceAttributes": {
"Ec2KeyName": "Keypair7",
"Ec2SubnetId": "subnet-017c52ed302f6069c",
"RequestedEc2SubnetIds": [
"subnet-017c52ed302f6069c"
],
"Ec2AvailabilityZone": "us-east-1e",
"RequestedEc2AvailabilityZones": [],
"IamInstanceProfile": "EMR_EC2_DefaultRole",
"EmrManagedMast erSecurityGroup": "sg-094b767de0d287eb7",
"EmrManagedSla veSecurityGroup": "sg-00f32b765e6a2c117",
"AdditionalMast erSecurityGroups": [],
"AdditionalSla veSecurityGroups": []
},
"InstanceCollectionType": "INSTANCE_GROUP",
"LogUri": "s3n://aws-logs-1234567890-us-east-1/elasticmapreduce/",
"ReleaseLabel": "emr-7.6.0",
"AutoTerminate": false,
"TerminationProtected": false,
"UnhealthyNodeReplacement": true,
"VisibleToAllUsers": true,
"Applications": [
{
"Name": "Hadoop",
"Version": "3.4.0"
},
{
"Name": "Hive",
"Version": "3.1.3"
},
{
"Name": "JupyterEnterpriseGateway",
"Version": "2.6.0"
},
{
"Name": "Livy",
"Version": "0.8.0"
},
{
"Name": "Spark",
"Version": "3.5.3"
}
],
"Tags": [],
"ServiceRole": "arn:aws:iam::1234567890:role/EMR_DefaultRole",
"NormalizedInstanceHours": 96,
"Mast erPublicDnsName": "ec2-54-237-95-60.compute-1.amazonaws.com",
"Configurations": [],
"AutoScalingRole": "arn:aws:iam::1234567890:role/EMR_AutoScaling_DefaultRole",
"ScaleDownBehavior": "TERMINATE_AT_TASK_COMPLETION",
"KerberosAttributes": {},
"ClusterArn": "arn:aws:elasticmapreduce:us-east-1:1234567890:cluster/j-2MXE9AR80RKTV",
"StepConcurrencyLevel": 1,
"PlacementGroups": [],
"OSReleaseLabel": "2023.6.20241212.0",
"BootstrapActions": [],
"InstanceGroups": [
{
"Id": "ig-1CMCR8JPMEO59",
"Name": "Core",
"Market": "ON_DEMAND",
"InstanceGroupType": "CORE",
"InstanceType": "m4.xlarge",
"RequestedInstanceCount": 1,
"RunningInstanceCount": 1,
"Status": {
"State": "TERMINATING",
"StateChangeReason": {
"Code": "CLUSTER_TERMINATED",
"Message": "Job flow terminated"
},
"Timeline": {
"CreationDateTime": "2025-01-03T18:27:03.556000+00:00",
"ReadyDateTime": "2025-01-03T18:32:26.247000+00:00"
}
},
"Configurations": [],
"ConfigurationsVersion": 0,
"LastSuccessfullyAppliedConfigurations": [],
"LastSuccessfullyAppliedConfigurationsVersion": 0,
"EbsBlockDevices": [
{
"VolumeSpecification": {
"VolumeType": "gp2",
"SizeInGB": 32
},
"Device": "/dev/sdb"
},
{
"VolumeSpecification": {
"VolumeType": "gp2",
"SizeInGB": 32
},
"Device": "/dev/sdc"
}
],
"EbsOptimized": true,
"ShrinkPolicy": {}
},
{
"Id": "ig-EI9Y0PY5YGM0",
"Name": "Task - 1",
"Market": "ON_DEMAND",
"InstanceGroupType": "TASK",
"InstanceType": "m4.xlarge",
"RequestedInstanceCount": 1,
"RunningInstanceCount": 1,
"Status": {
"State": "TERMINATING",
"StateChangeReason": {
"Code": "CLUSTER_TERMINATED",
"Message": "Job flow terminated"
},
"Timeline": {
"CreationDateTime": "2025-01-03T18:27:03.556000+00:00",
"ReadyDateTime": "2025-01-03T18:32:27.774000+00:00"
}
},
"Configurations": [],
"ConfigurationsVersion": 0,
"LastSuccessfullyAppliedConfigurations": [],
"LastSuccessfullyAppliedConfigurationsVersion": 0,
"EbsBlockDevices": [
{
"VolumeSpecification": {
"VolumeType": "gp2",
"SizeInGB": 32
},
"Device": "/dev/sdb"
},
{
"VolumeSpecification": {
"VolumeType": "gp2",
"SizeInGB": 32
},
"Device": "/dev/sdc"
}
],
"EbsOptimized": true,
"ShrinkPolicy": {}
},
{
"Id": "ig-147XGW812JXRI",
"Name": "Primary",
"Market": "ON_DEMAND",
"InstanceGroupType": "MAST ER",
"InstanceType": "m4.4xlarge",
"RequestedInstanceCount": 1,
"RunningInstanceCount": 1,
"Status": {
"State": "TERMINATING",
"StateChangeReason": {
"Code": "CLUSTER_TERMINATED",
"Message": "Job flow terminated"
},
"Timeline": {
"CreationDateTime": "2025-01-03T18:27:03.555000+00:00",
"ReadyDateTime": "2025-01-03T18:31:54.130000+00:00"
}
},
"Configurations": [],
"ConfigurationsVersion": 0,
"LastSuccessfullyAppliedConfigurations": [],
"LastSuccessfullyAppliedConfigurationsVersion": 0,
"EbsBlockDevices": [
{
"VolumeSpecification": {
"VolumeType": "gp2",
"SizeInGB": 32
},
"Device": "/dev/sdb"
},
{
"VolumeSpecification": {
"VolumeType": "gp2",
"SizeInGB": 32
},
"Device": "/dev/sdc"
}
],
"EbsOptimized": true,
"ShrinkPolicy": {}
}
]
}
}
Default security groups do not seem to be working