How to convert CUR billing data into CUR 2.0 data without raising a case for backfilling?

Question

I currently have an S3 bucket with billing data in legacy CUR format. I want to convert all the billing data in the bucket into CUR 2.0 format. Is it enough to just create a new data export with CUR 2.0 export data in the same bucket or do I have to do something else. I do not want to raise a customer ticket asking for backfilling of data.

AWS Support - Kraig · Accepted Answer

Hi Hermanth,

Just creating a new CUR 2.0 report definition pointing to the same bucket won't automatically convert your existing legacy CUR data. The new CUR 2.0 setup will only generate reports in the new format going forward from the time you set it up.

To handle historical data, you have a few options:

**1. Maintain both formats:**
* Keep your legacy CUR data as-is
* Set up new CUR 2.0 reports for future data
* Update your data processing pipelines to handle both formats

**2. Manual conversion:**
* Create a processing pipeline to transform the legacy CUR data into CUR 2.0 format
* This would require mapping the old columns to new CUR 2.0 schema
* Process and rewrite all historical files

**3. Fresh start:**
* If you don't absolutely need historical data in CUR 2.0 format
* Set up CUR 2.0 and accept that data before the switch will be in legacy format

Without raising a support ticket for backfilling, there's no official AWS-provided way to convert historical data to CUR 2.0 format. Most organizations either:
* Keep both formats and handle them separately in their analysis tools
* Only use CUR 2.0 going forward and maintain legacy format for historical analysis

The recommended approach depends on:
* How far back you need to analyze data
* Whether your analysis tools can handle both formats
* Resources available for building conversion tools

Here's the relevant documentation tying back to the mentioned key points

Creating a new CUR 2.0 report won't convert existing legacy data. This is documented in AWS's CUR documentation:
* [CUR 2.0 Overview](https://docs.aws.amazon.com/cur/latest/userguide/what-is-cur.html)
* [Creating Cost and Usage Reports](https://docs.aws.amazon.com/cur/latest/userguide/cur-create.html)

For handling historical data, AWS provides these resources:
* [Understanding CUR data delivery timeline](https://docs.aws.amazon.com/cur/latest/userguide/cur-delivery-overview.html)
* [CUR Data Dictionary](https://docs.aws.amazon.com/cur/latest/userguide/data-dictionary.html) - helpful for manual conversion

If you want to process the data yourself, you can refer to:
* [AWS Cost and Usage Report Schema](https://docs.aws.amazon.com/cur/latest/userguide/cur-data-overview.html)
* [Querying Cost and Usage Reports](https://docs.aws.amazon.com/cur/latest/userguide/cur-query.html)

For working with both formats:
* [Managing Cost and Usage Reports](https://docs.aws.amazon.com/cur/latest/userguide/managing-cur.html)
* [AWS CUR Best Practices](https://docs.aws.amazon.com/cur/latest/userguide/cur-best-practices.html)

You can also find community discussions about CUR migration on:
* [AWS re:Post](https://repost.aws/topics/TABRbH3RpkNqQrEFiQE_q0SQ/cost-and-usage-report)
* [AWS Cost Management Blog](https://aws.amazon.com/blogs/aws-cost-management/)

These resources should help you understand the implications and options for handling both legacy and CUR 2.0 formats.

\- Kraig E.

Harish · Answer

One of the key differences between CUR 2.0 and CUR:
CUR 2.0 contains a fixed set of columns, whereas the columns included for CUR can vary monthly depending on your usage of AWS services, cost categories, and resource tags

To be able to convert all legacy CUR data into a standard CUR 2.0 you could use MS Excel Power Query or Python's Pandas library.   For Excel, import your files into Power Query, and then use its features to combine the data, addressing any missing column issues.

Here are the steps:

1. Import your CSV files into Power Query by going to "Data" > "Get & Transform Data" > "From File" > "From Folder". 
2. In the Power Query Editor, you'll see a list of your files. You can choose to "Combine & Transform Data" or "Combine & Load". 
3. Within the Power Query Editor, you can select the columns you want to merge and specify how missing data should be handled. 
4. You can also use Power Query to clean and format your data before merging.

Using Python:
1. Install the Pandas library using pip install pandas. 
2. Read each CSV file into a Pandas DataFrame using pd.read_csv(). 
3. Use the pd.concat() function to combine the DataFrames. 
4. Handle missing columns by either:
Filling them with empty strings or NaN values using fillna(). 
Renaming them to a common name if appropriate. 
5. Save the merged DataFrame to a new CSV file using merged_df.to_csv("merged.csv", index=False).

How to convert CUR billing data into CUR 2.0 data without raising a case for backfilling?

Add your answer

Relevant content