How can I understand AWS Config billing by retrieving the number of configuration items recorded per month?

Resolution

To help you identify previous and current billing trends, use Amazon Athena to identify the number of configuration items (CI) per month for your account.

Verify that your S3 bucket contains configuration files

Note: Be sure that the AWS Config service is able to deliver configuration history files successfully to your designated Amazon Simple Storage Service (Amazon S3) bucket. Typically, CIs are delivered to the bucket as configuration history files every 6 hours. For more information, see Viewing configuration details.

1. Open the AWS Config console, and then choose Settings from the navigation pane.
2. In the Amazon S3 bucket section, note the Bucket name.
3. Open the S3 console. Then, for Bucket name, choose your S3 bucket. Verify that the S3 bucket you choose contains configuration files.
   Note: If there are no configuration files, then your role might be missing permissions. For more information, see Identity and access management in Amazon S3.

Create a table in Athena

1. Sign in to the Athena console, and then follow the instructions to create a table using the wizard.
2. Use the following table syntax:

CREATE EXTERNAL TABLE awsconfig (
         fileversion string,
         configSnapshotId string,
         configurationitems ARRAY < STRUCT < configurationItemVersion : STRING,
         configurationItemCaptureTime : STRING,
         configurationStateId : BIGINT,
         awsAccountId : STRING,
         configurationItemStatus : STRING,
         resourceType : STRING,
         resourceId : STRING,
         resourceName : STRING,
         ARN : STRING,
         awsRegion : STRING,
         availabilityZone : STRING,
         configurationStateMd5Hash : STRING,
         resourceCreationTime : STRING > > 
) 
ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe' LOCATION 's3://<BUCKET-NAME>/AWSLogs/<ACCOUNT-ID>/Config/<REGION>/';

For LOCATION, use the location and Region for the AWS Config items stored in your Amazon S3 bucket. For BUCKET-NAME, ACCOUNT-ID, and REGION, use your specific information.

Note: If you're using Athena engine version 2, then the maximum line length for text files is 100 MB. If you have a large number of resources, then Config items stored in the designated AWS Config S3 bucket might exceed this limit. For example, AWS Config also delivers configuration snapshot files in the same bucket LOCATION, and the configuration snapshot file could exceed this limit. If you exceed the limit, then when querying the AWS Config item, you receive an error that looks similar to the following:

HIVE_BAD_DATA: Line too long in text file: <s3_path_to_config_data_object>

In this case, use the following table syntax to point Amazon Athena to directly query the S3 path that stores the configuration history files instead:

CREATE EXTERNAL TABLE awsconfig (
         fileversion string,
         configSnapshotId string,
         configurationitems ARRAY < STRUCT < configurationItemVersion : STRING,
         configurationItemCaptureTime : STRING,
         configurationStateId : BIGINT,
         awsAccountId : STRING,
         configurationItemStatus : STRING,
         resourceType : STRING,
         resourceId : STRING,
         resourceName : STRING,
         ARN : STRING,
         awsRegion : STRING,
         availabilityZone : STRING,
         configurationStateMd5Hash : STRING,
         resourceCreationTime : STRING > > 
) 
PARTITIONED BY (`year` string,`month` string,`day` string)
ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'
LOCATION 's3://<BUCKET-NAME>/AWSLogs/<ACCOUNT-ID>/Config/<REGION>/'
TBLPROPERTIES (
  'projection.enabled'='true', 
  'projection.year.interval'='1', 
  'projection.year.range'='2021,2121', 
  'projection.year.type'='integer', 
  'projection.month.interval'='1', 
  'projection.month.range'='1,12', 
  'projection.month.type'='integer', 
  'projection.day.interval'='1', 
  'projection.day.range'='1,31', 
  'projection.day.type'='integer', 
  'storage.location.template'='s3://<BUCKET-NAME>/AWSLogs/<ACCOUNT-ID>/Config/<REGION>/${year}/${month}/${day}/ConfigHistory/')

For LOCATION, use the location and Region for the AWS Config items stored in your Amazon S3 bucket. For BUCKET-NAME, ACCOUNT-ID, and REGION, use your specific information. The following example table partitions the Athena table with partitions projection from /2021/1/1/ to /2121/12/31/. Customize this time period as needed.

Note: The AWS Config data S3 path date format isn't compatible with Athena partitions projection date type format.

Example Athena queries

The following example query retrieves the number of configuration items per day in February 2021:

SELECT result.configurationitemcapturetime,
         count(result.configurationitemcapturetime) AS NumberOfChanges
FROM 
    (SELECT regexp_replace(configurationItem.configurationItemCaptureTime,
         '(.+)(T.+)', '$1') AS configurationitemcapturetime
    FROM default.awsconfig
    CROSS JOIN UNNEST(configurationitems) AS t(configurationItem)
    WHERE "$path" LIKE '%ConfigHistory%'
            AND configurationItem.configurationItemCaptureTime >= '2021-02-01T%'
            AND configurationItem.configurationItemCaptureTime <= '2021-02-28T%') result
GROUP BY  result.configurationitemcapturetime
ORDER BY  result.configurationitemcapturetime

The results are similar to:

configurationitemcapturetime    NumberOfChanges
2021-02-02    7
2021-02-03    3
2021-02-07   11
...

The following example query retrieves the number of changes per resource in February 2021, sorted by most frequently changed:

SELECT configurationItem.resourceType,
         configurationItem.resourceId,
         COUNT(configurationItem.resourceId) AS NumberOfChanges
FROM default.awsconfig
CROSS JOIN UNNEST(configurationitems) AS t(configurationItem)
WHERE "$path" LIKE '%ConfigHistory%'
        AND configurationItem.configurationItemCaptureTime >= '2021-02-01T%'
        AND configurationItem.configurationItemCaptureTime <= '2021-02-28T%'
GROUP BY  configurationItem.resourceType, configurationItem.resourceId
ORDER BY  NumberOfChanges DESC

The results are similar to:

resourcetype              resourceid        NumberOfChanges
AWS::EC2::VPC             vpc-9ed00bfa        7
AWS::EC2::Subnet          subnet-4472e248     5
AWS::EC2::SecurityGroup   sg-450c6531         4

Note: When comparing the total number of CIs between Athena query results and AWS billing data for the same month and Region, a discrepancy can occur. The data queried by Athena can cross day boundaries and also include CIs billed in adjacent months. AWS Config CIs are metered based on when the time configurationItemCaptureTime was initiated.

As a best practice, increment the specified end day from end of month by one.

For example, change this query:

AND configurationItem.configurationItemCaptureTime <= '2021-02-28T%') result

To this query:

AND configurationItem.configurationItemCaptureTime <= '2021-03-01T%') result

Note: The last day of the month is incremented by one day.

Related information

AWS Config pricing

Amazon Athena pricing