2 Answers
- Newest
- Most votes
- Most comments
1
An entire script looks like the following, this is for CSV but you can easily use any supported data format:
import sys
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue.utils import getResolvedOptions
args = getResolvedOptions(sys.argv, ["JOB_NAME"])
glue_context= GlueContext(SparkContext.getOrCreate())
job = Job(glue_context)
job.init(args["JOB_NAME"], args)
dyf = glue_context.create_dynamic_frame.from_options(
connection_type="s3",
connection_options = {
"paths": ["s3://mybucketb/myprefix"]
},
format = "csv",
format_options = {
"withHeader": True
}
)
glue_context.write_dynamic_frame_from_options(
frame=dyf,
connection_type="dynamodb",
connection_options={"dynamodb.output.tableName": "MyTableName",
"dynamodb.throughput.write.percent": "1.0"
}
)
job.commit()
-1
Here is an example code:
ApplyMapping_node = ApplyMapping.apply(
frame=S3bucket_node,
mappings=[
("Item.pk.S", "string", "Item.pk.S", "string"),
("Item.sk.S", "string", "Item.sk.S", "string")
],
transformation_ctx="ApplyMapping_node"
)
S3bucket_node_alt = glueContext.write_dynamic_frame.from_options(
frame=ApplyMapping_node,
connection_type="dynamodb",
connection_options={"dynamodb.output.tableName": "my-table"}
}
You could as well take this link as reference: https://aws.amazon.com/blogs/database/implement-vertical-partitioning-in-amazon-dynamodb-using-aws-glue/
answered a year ago
Relevant content
- Accepted Answerasked 6 months ago
- asked 4 months ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated a year ago