Glue to use s3 to Dynamo Db

0

Can anyone share any script that can be used in my glue job to load files from s3 to dynamo db?

asked a year ago1184 views
2 Answers
1

An entire script looks like the following, this is for CSV but you can easily use any supported data format:

import sys
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue.utils import getResolvedOptions

args = getResolvedOptions(sys.argv, ["JOB_NAME"])
glue_context= GlueContext(SparkContext.getOrCreate())
job = Job(glue_context)
job.init(args["JOB_NAME"], args)

dyf = glue_context.create_dynamic_frame.from_options(
    connection_type="s3",
    connection_options = {
    "paths": ["s3://mybucketb/myprefix"]
    }, 
    format = "csv",
    format_options = {
        "withHeader": True
        }
)

glue_context.write_dynamic_frame_from_options(
    frame=dyf,
    connection_type="dynamodb",
    connection_options={"dynamodb.output.tableName": "MyTableName",
        "dynamodb.throughput.write.percent": "1.0"
    }
)

job.commit()
profile pictureAWS
EXPERT
answered a year ago
-1

Here is an example code:

ApplyMapping_node = ApplyMapping.apply( 
        frame=S3bucket_node, 
        mappings=[ 
        ("Item.pk.S", "string", "Item.pk.S", "string"), 
        ("Item.sk.S", "string", "Item.sk.S", "string") 
        ], 
          transformation_ctx="ApplyMapping_node"  
        )
            
      S3bucket_node_alt = glueContext.write_dynamic_frame.from_options( 
        frame=ApplyMapping_node, 
        connection_type="dynamodb", 
        connection_options={"dynamodb.output.tableName": "my-table"}
       }

You could as well take this link as reference: https://aws.amazon.com/blogs/database/implement-vertical-partitioning-in-amazon-dynamodb-using-aws-glue/

AWS
vtjean
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions