How to remove Unnamed column while creating dynamic frame from catalog options

Question

Hi Team,

I have created a glue job which reading a table from glue data catalog and creating dynamic frame. But while running the job, it is throwing the below error. After checking the logs I saw it is not able create the dynamic frame due to the below error.

"Attribute name "Unnamed: 7" contains invalid character(s) among " ,;{}()
	=". Please use alias to rename it."

I am using the below statement to create dynamic frame. I want to avoid creating the unnamed columns during the creation of dynamic frame. Any help would be much appreciated. Thank you

demotable= glueContext.create_dynamic_frame.from_catalog(
    database="intraday",
    table_name="demo_table",
    push_down_predicate = "bus_dt = 20180117",
    transformation_ctx="demotable",
)

Answer

o remove the unnamed column while creating a dynamic frame from the catalog options, you can use the ApplyMapping class from the awsglue.transforms module. This allows you to selectively keep the columns you want and exclude the unnamed columns.

Here's an example of how you can do this:

```
from awsglue.transforms import ApplyMapping

# Read the data from the catalog
demotable = glueContext.create_dynamic_frame.from_catalog(
    database="intraday",
    table_name="demo_table",
    push_down_predicate="bus_dt = 20180117",
    transformation_ctx="demotable"
)

# Define the schema mapping, excluding the unnamed column
mapping = [
    ("column1", "string", "column1", "string"),
    ("column2", "string", "column2", "string"),
    # Add all other columns that you want to keep, but exclude "Unnamed: 7"
]

# Apply the mapping
demotable_transformed = ApplyMapping.apply(frame=demotable, mappings=mapping, transformation_ctx="demotable_transformed")

# Continue to the code ....
```

How to remove Unnamed column while creating dynamic frame from catalog options

Read the data from the catalog

관련 콘텐츠