Orchestrate Sqoop with Step Functions

0

Can we use Step Functions to orchestrate a Sqoop job? The goal is to create a transient cluster which load data with Sqoop first followed by transformation in Hive, but looks like Command Runner does not have such option.

If not, what are the alternatives?

AWS
gefragt vor 3 Jahren375 Aufrufe
1 Antwort
0
Akzeptierte Antwort

You can run any script in script runner mode.

https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hadoop-script.html

Basically from the step function invoke the syntax thats required by the script runner, for example:

       "StartAt":"Step 1",
       "States":{ 
          "Step_1":{ 
             "Type":"Task",
             "Resource":"arn:aws:states:::elasticmapreduce:addStep.sync",
             "Parameters":{ 
                "ClusterId.$":"$.ClusterId",
                "Step":{ 
                   "Name":"1 - Step 1",
                   "ActionOnFailure":"CONTINUE",
                   "HadoopJarStep":{ 
                      "Jar":"s3://elasticmapreduce/libs/script-runner/script-runner.jar",
                      "Args":[ 
                         "s3://xxx/scripts/step1.sh"
                      ]
                   }
                }
             },
             "End":true
          }
       }
    }

Put all the code in the step1.sh script, this script will execute on master node and do any task you want including your sqoop stuff

AWS
EXPERTE
Behram
beantwortet vor 3 Jahren

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen