1 Risposta
- Più recenti
- Maggior numero di voti
- Maggior numero di commenti
0
When you rename the old name no longer exists, to make a copy just declare a new column in DataFrame taking the value from the other column (while keeping it):
df = df.withColumn("cluster", df['old_cluster']) # notice df is a DataFrame
Also note that when you repartition there you are not creating a partition column, just reorganizing the data by that column (which with 1 partition is pointless)
Contenuto pertinente
- AWS UFFICIALEAggiornata un anno fa
- AWS UFFICIALEAggiornata 3 anni fa
- AWS UFFICIALEAggiornata un anno fa
Thanks, that did not work. Here is what I did:
And I get an error "Error Category: QUERY_ERROR; AnalysisException: Cannot resolve column name "old_cluster" among (<all columns except cluster or old_cluster)". Cluster is a partition column and so is not explicitly in the parquet object itself.
you are still doing the rename, so the old name is gone
Sorry, not following you. I could not find either old_cluster or new cluster. Both of these columns were not there in the error message. I also tried
and it said "Error Category: QUERY_ERROR; AnalysisException: Cannot resolve column name "cluster" among (<all columns except cluster>)"
you cannot create a column name the same of an existing column, not cannot reference a column that doesn't exist. To make the copy you need to pass on withColumn the name of the new column and in the value the reference to the column that you want to copy from
That doesn't work either. I get "Error Category: QUERY_ERROR; AnalysisException: Cannot resolve column name "cluster" among (<all the other columns>)".