Glue to use s3 to Dynamo Db

0

Can anyone share any script that can be used in my glue job to load files from s3 to dynamo db?

gefragt vor einem Jahr1209 Aufrufe
2 Antworten
1

An entire script looks like the following, this is for CSV but you can easily use any supported data format:

import sys
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue.utils import getResolvedOptions

args = getResolvedOptions(sys.argv, ["JOB_NAME"])
glue_context= GlueContext(SparkContext.getOrCreate())
job = Job(glue_context)
job.init(args["JOB_NAME"], args)

dyf = glue_context.create_dynamic_frame.from_options(
    connection_type="s3",
    connection_options = {
    "paths": ["s3://mybucketb/myprefix"]
    }, 
    format = "csv",
    format_options = {
        "withHeader": True
        }
)

glue_context.write_dynamic_frame_from_options(
    frame=dyf,
    connection_type="dynamodb",
    connection_options={"dynamodb.output.tableName": "MyTableName",
        "dynamodb.throughput.write.percent": "1.0"
    }
)

job.commit()
profile pictureAWS
EXPERTE
beantwortet vor einem Jahr
-1

Here is an example code:

ApplyMapping_node = ApplyMapping.apply( 
        frame=S3bucket_node, 
        mappings=[ 
        ("Item.pk.S", "string", "Item.pk.S", "string"), 
        ("Item.sk.S", "string", "Item.sk.S", "string") 
        ], 
          transformation_ctx="ApplyMapping_node"  
        )
            
      S3bucket_node_alt = glueContext.write_dynamic_frame.from_options( 
        frame=ApplyMapping_node, 
        connection_type="dynamodb", 
        connection_options={"dynamodb.output.tableName": "my-table"}
       }

You could as well take this link as reference: https://aws.amazon.com/blogs/database/implement-vertical-partitioning-in-amazon-dynamodb-using-aws-glue/

AWS
vtjean
beantwortet vor einem Jahr

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen