Glue python repartition while retaining old partition column
0
I have data currently partitioned on a key (say cluster) and I'm repartitioning to a new key 'date'. So I do (in Python)
df = glueContext.create_dynamic_frame.from_options(...)
df = df.rename_field('cluster', 'old_cluster')
p_df = df.toDF().repartition(1, 'date')
p_d_df = DynamicFrame.fromDF(p_df, ...)
This works and I can get the 'date' value as a partition/column. However, I cannot see 'cluster' or 'old_cluster'. How can I retail the old cluster key? Thanks
asked 3 months ago156 views
1 Answer
- Newest
- Most votes
- Most comments
0
When you rename the old name no longer exists, to make a copy just declare a new column in DataFrame taking the value from the other column (while keeping it):
df = df.withColumn("cluster", df['old_cluster']) # notice df is a DataFrame
Also note that when you repartition there you are not creating a partition column, just reorganizing the data by that column (which with 1 partition is pointless)
Relevant content
- asked 9 months ago
AWS OFFICIALUpdated a year ago- AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated 2 months ago
Thanks, that did not work. Here is what I did:
And I get an error "Error Category: QUERY_ERROR; AnalysisException: Cannot resolve column name "old_cluster" among (<all columns except cluster or old_cluster)". Cluster is a partition column and so is not explicitly in the parquet object itself.
you are still doing the rename, so the old name is gone
Sorry, not following you. I could not find either old_cluster or new cluster. Both of these columns were not there in the error message. I also tried
and it said "Error Category: QUERY_ERROR; AnalysisException: Cannot resolve column name "cluster" among (<all columns except cluster>)"
you cannot create a column name the same of an existing column, not cannot reference a column that doesn't exist. To make the copy you need to pass on withColumn the name of the new column and in the value the reference to the column that you want to copy from
That doesn't work either. I get "Error Category: QUERY_ERROR; AnalysisException: Cannot resolve column name "cluster" among (<all the other columns>)".