Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Want to create AWS Datazone lineage by capturing lineage from AWS Glue tables?

0

I am trying to capture lineage from AWS Glue tables and reflect it in AWS Datazone asset lineage for this I am following the official blog https://aws.amazon.com/blogs/big-data/amazon-datazone-introduces-openlineage-compatible-data-lineage-visualization-in-preview/ following same steps as mentioned in the blog. In Job Details I have mentioned required jar(openlineage-spark_2.12-1.9.1.jar, openlineage-spark-1.8.0.jar) also set up the job parameters as mentioned and also using the mentioned glue version that is 3.Enter image description here. Using this earlier I was able to create lineage successfully, but since last update in AWS Datazone I am not able to create data lineage for aws glue tables. Also Using spark.sql to create lineage between glue tables

Steps I am following for creating

  1. Uploading python file extract_glue_spark_lineage.py in AWS CLI
  2. Creating glue job with required job jars and parameters
  3. setting up environment in aws cli
  4. Running the uploaded py file extract_glue_spark_lineage.py for required domain using python extract_glue_spark_lineage.py --region "us-west-2" --domain-identifier 'domainID' in aws cli environment
  5. Parallel Starting the glue job
  6. Lineage is build using error logs of glue job Since this steps were able to capture lineage of glue tables but since update I am not able to create to create with same jars, job parameters and mentioned steps and facing issue Enter image description here earlier state would go from START to COMPLETE successfully posting lineage on aws datazone for asset now its not working.

Please provide assistance to resolve this issue. Provide the open-lineage jar which is needed and also glue version we need or any other setting within aws datazone to capture aws glue table lineage

Topics
AnalyticsDeveloper Tools
Tags
AWS CloudShellAWS GlueAmazon DataZone
Language
English

asked 2 years ago2.7K views

1 Answer
0

To capture lineage from AWS Glue tables and reflect it in AWS DataZone asset lineage, you're on the right track with your approach. However, since you're experiencing issues after a recent update, there are a few things you can try to resolve the problem:

  1. OpenLineage JAR files: Ensure you're using the most up-to-date OpenLineage JAR files. The latest compatible versions for AWS Glue are:
  • openlineage-spark_2.12-1.9.1.jar
  • openlineage-spark-1.8.0.jar

  1. AWS Glue version: Use AWS Glue version 3.0 or later, as these versions are compatible with the OpenLineage integration.

  2. Job parameters: Double-check that you've set the required job parameters correctly. These typically include:

  1. AWS DataZone configuration: Ensure that data lineage is enabled for your AWS DataZone domain and that you have the necessary permissions to capture and view lineage data.

  2. Use Spark DataFrames: The OpenLineage Spark plugin may not be able to extract data lineage from AWS Glue Spark jobs that use AWS Glue DynamicFrames. Instead, use Spark SQL DataFrames in your Glue jobs.

  3. Check CloudWatch Logs: Review the Glue job's CloudWatch logs for any error messages or warnings that might indicate why the lineage capture is failing.

  4. Update extract_glue_spark_lineage.py: Ensure you're using the latest version of this script, as it may have been updated to work with the new AWS DataZone version.

  5. API compatibility: Verify that you're using the correct API endpoints and methods for the updated version of AWS DataZone.

If you're still encountering issues after trying these steps, you may need to contact AWS support for further assistance, as there could be specific changes in the latest update that require additional configuration or troubleshooting.

Remember to test your changes in a non-production environment first to ensure they work as expected before applying them to your production workflows.

Sources
Amazon DataZone introduces OpenLineage-compatible data lineage visualization in preview | AWS Big Data Blog
Data lineage in Amazon DataZone (Preview) - Amazon DataZone
Amazon DataZone introduces API-driven, OpenLineage-compatible data lineage visualization in preview - AWS

answered 2 years ago

  • Devagya Udgir
    2 years ago

    The above solution is not working there is issue with aws glue job error logs due to formatting lineage event is getting captured in two seperate error logs block due to which current provided python script provided by aws doesn't work, please provide updated script to extract_glue_spark_lineage.py file. Since the existing is not working and able to capture lineage from error logs

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content