EMR on EKS - Job Flow/Run Id

Question

EMR on EC2 provides job flow metadata via `/mnt/var/lib/info/job-flow.json` file on disk. However, this doesn't exist on EMR on EKS. Is there an equivalent metadata provider to get Id of the job run?

Answer

Hello,

As Amazon Elastic Kubernetes Service (EKS) is a Kubernetes platform, the JobMetadata would be in the Spark Driver’s Pod Spec

a. The Spark Job ID would be in the pod name 
```
example: 000000042acm1kt2x1o is Spark Job id

> kubectl get pods
spark-000000042acm1kt2x1o-driver
```

b. All job meta data would be in Pod Spec.

This is a sample job, with all the information about the job

```
> kubectl get pod spark-000000042acm1kt2x1o-driver --output-yaml
...
...
metadata:
  annotations:
    kubernetes.io/psp: eks.privileged
  creationTimestamp: "2022-08-23T22:22:19Z"
  labels:
    emr-containers.amazonaws.com/component: driver
    emr-containers.amazonaws.com/job.id: 000000042acm1kt2x1o
    emr-containers.amazonaws.com/resource.type: job.run
    emr-containers.amazonaws.com/virtual-cluster-id: qqqreav12ops1qa7kngtk1oe0
    spark-app-selector: spark-000000042acm1kt2x1o
    spark-role: driver
  name: spark-000000042acm1kt2x1o-driver
  namespace: emr-ns-1
  ownerReferences:
  - apiVersion: batch/v1
    blockOwnerDeletion: true
    controller: true
    kind: ConfigMap
    name: 000000042acm1kt2x1o-spark-defaults
    uid: fead72159-24cb-2cef-11b0-ff421fc8e659
  resourceVersion: "13765828"
  uid: hgta8c41-362b-8u5e-b62c-28e6f6564788
...
...
...
...
```

To view Kubernetes resources using a command-line tool, use kubectl. 
> https://docs.aws.amazon.com/eks/latest/userguide/view-kubernetes-resources.html

EMR on EKS - Job Flow/Run Id

Relevant content