EMR on EKS - Job Flow/Run Id

Hello,

As Amazon Elastic Kubernetes Service (EKS) is a Kubernetes platform, the JobMetadata would be in the Spark Driver’s Pod Spec

a. The Spark Job ID would be in the pod name

example: 000000042acm1kt2x1o is Spark Job id

> kubectl get pods
spark-000000042acm1kt2x1o-driver

b. All job meta data would be in Pod Spec.

This is a sample job, with all the information about the job

> kubectl get pod spark-000000042acm1kt2x1o-driver --output-yaml
...
...
metadata:
  annotations:
    kubernetes.io/psp: eks.privileged
  creationTimestamp: "2022-08-23T22:22:19Z"
  labels:
    emr-containers.amazonaws.com/component: driver
    emr-containers.amazonaws.com/job.id: 000000042acm1kt2x1o
    emr-containers.amazonaws.com/resource.type: job.run
    emr-containers.amazonaws.com/virtual-cluster-id: qqqreav12ops1qa7kngtk1oe0
    spark-app-selector: spark-000000042acm1kt2x1o
    spark-role: driver
  name: spark-000000042acm1kt2x1o-driver
  namespace: emr-ns-1
  ownerReferences:
  - apiVersion: batch/v1
    blockOwnerDeletion: true
    controller: true
    kind: ConfigMap
    name: 000000042acm1kt2x1o-spark-defaults
    uid: fead72159-24cb-2cef-11b0-ff421fc8e659
  resourceVersion: "13765828"
  uid: hgta8c41-362b-8u5e-b62c-28e6f6564788
...
...
...
...

To view Kubernetes resources using a command-line tool, use kubectl.

https://docs.aws.amazon.com/eks/latest/userguide/view-kubernetes-resources.html