Issue with DataBrew RecipeJob Destination S3 path

0

I have a bit of an issue with databrew job pushing result file into the incorrect location. The use case: I am working on the step function which is masking the data using Data Brew. I have a parquet input file in S3 with path: folder1/file.parquet in the Account A. My Databrew job is configured to mask the data in the file and push it to S3 in Account B under the same path as original file. The config is:

"Outputs": [
    {
      "Format": "PARQUET",
      "MaxOutputFiles": 1,
      "Overwrite": "TRUE",
      "Location": {
        "Bucket": "BucketInAccountB",
        "BucketOwner": "AccountBOwnerId",
        "Key.$": "States.Format('{}/{}/{}/{}/{}',States.ArrayGetItem(States.StringSplit($.detail.object.key, '/'), 0),States.ArrayGetItem(States.StringSplit($.detail.object.key, '/'), 1), States.ArrayGetItem(States.StringSplit($.detail.object.key, '/'), 2), States.ArrayGetItem(States.StringSplit($.detail.object.key, '/'), 3), $.Dataset.filename)"
      }
    }
  ],

The actual behaviour is that output file name is folder1/file.parquet/NameOfTheRecopeJob_time_part000.parquet

The crazy bit is that when I check the job in DataBrew console, in Job run history and open the Output, it actually shows the expected path - https://us-east-1.console.aws.amazon.com/s3/object/BucketInAccountB?region=ap-southeast-2&prefix=folder1/file.parquet

Denys
질문됨 일 년 전97회 조회
답변 없음

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠