Python Script has Value Error on EC2

0

The python script works well read/write file to S3 with my local machine by aws-vault authentication, then I wanted to deploy this script on EC2. However, this error pops up when I run the scripts on EC2 linux instance.

What the script does are

  1. Call the restAPI
  2. Data Transformation from Json to Pandas Dataframe
  3. Does some aggregation as report (I think this is where the error appears)
  4. Write back report back to S3
Traceback (most recent call last):
  File "/home/ec2-user/intl_salesOrder_query.py", line 146, in <module>
    orders_since_monthBeginning[float_columns] = orders_since_monthBeginning[float_columns].astype('float64').copy()
  File "/home/ec2-user/miniconda3/lib/python3.9/site-packages/pandas/core/frame.py", line 3643, in __setitem__
    self._setitem_array(key, value)
  File "/home/ec2-user/miniconda3/lib/python3.9/site-packages/pandas/core/frame.py", line 3687, in _setitem_array
    self[k1] = value[k2]
  File "/home/ec2-user/miniconda3/lib/python3.9/site-packages/pandas/core/frame.py", line 3645, in __setitem__
    self._set_item_frame_value(key, value)
  File "/home/ec2-user/miniconda3/lib/python3.9/site-packages/pandas/core/frame.py", line 3775, in _set_item_frame_value
    raise ValueError("Columns must be same length as key")
asked 2 years ago259 views
1 Answer
1

The error message your seeing is coming from pandas and it indicates to me that the data you're loading is somehow not in the format that is expected. It's not possible to provide more advice here without seeing at least part of the code and the input data that is causing the error.

profile pictureAWS
EXPERT
answered 2 years ago
  • I tried to change the data type from string to numeric so that I can do further calculation. Following are the code snippets cause error

    df[float_columns] = df[float_columns].apply(pd.to_numeric)
    df[float_columns] = df[float_columns].astype('float64')
    
  • I'm not sure that's the issue - the error message indicates that there may be a formatting/data issue: "Columns must be the same length as key".

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions