1 Risposta
- Più recenti
- Maggior numero di voti
- Maggior numero di commenti
0
I am not aware of any inbuilt mechanism to achieve this use case.
But you can have some custom logic to see if there are any applications from user livy is running. User livy is being used by EMR notebook to submit jobs in EMR cluster.
[root@ip-172-31-42-13 ~]# yarn application -list |grep -i hadoop
24/05/08 11:27:15 INFO client.RMProxy: Connecting to ResourceManager at ip-172-31-42-13.ec2.internal/172.31.42.13:8032
24/05/08 11:27:15 INFO client.AHSProxy: Connecting to Application History server at ip-172-31-42-13.ec2.internal/172.31.42.13:10200
[root@ip-172-31-42-13 ~]# echo $?
1
[root@ip-172-31-42-13 ~]#
[root@ip-172-31-42-13 ~]#
[root@ip-172-31-42-13 ~]# yarn application -list |grep -i livy
24/05/08 11:27:22 INFO client.RMProxy: Connecting to ResourceManager at ip-172-31-42-13.ec2.internal/172.31.42.13:8032
24/05/08 11:27:22 INFO client.AHSProxy: Connecting to Application History server at ip-172-31-42-13.ec2.internal/172.31.42.13:10200
application_1715160855732_0002 livy-session-1 SPARK livy default RUNNING UNDEFINED 10% http://ip-172-31-42-13.ec2.internal:4041
[root@ip-172-31-42-13 ~]# echo $?
0
echo $?
determine the exit status of the command and if its 0
then it means the application is present and the session is active. But it doesn't necessarily mean that the notebook execution is going on or finished. It could also be a case where the kernel is staying idle and waiting for the command execution.
for knowing the job execution, you would need to track it using YARN with per the corresponding application id.
Contenuto pertinente
- AWS UFFICIALEAggiornata un anno fa
- AWS UFFICIALEAggiornata 2 anni fa
- AWS UFFICIALEAggiornata 2 anni fa
Hi ! Thanks for the reply. Yes, after a bit of research, I too came to Livy session usage. Basically each opened Jupyter notebook with spark session denotes a single Livy session on Yarn. Used this documentation to achieve my goal https://livy.incubator.apache.org/docs/latest/rest-api.html