How can I speed up the creation of a global secondary index for an Amazon DynamoDB table?

3 minuto de leitura
0

I want to create a global secondary index (GSI) for an Amazon DynamoDB table, but it's taking a long time.

Short description

When you add a new global secondary index to an existing table, then the IndexStatus is set to CREATING and Backfilling is true. Backfilling reads items from the table and determines whether they can be added to the index. When you backfill an index, DynamoDB uses the internal system capacity to read items from the table. This minimizes the effect of index creation and makes sure that the table doesn't run out of read capacity.

The time required for building a global secondary index depends on multiple factors:

  • The size of the base table
  • The number of items in the table that qualify for inclusion in the index
  • The number of attributes projected into the index
  • The provisioned write capacity of the index
  • Write activity on the base table during index creation
  • Data distribution across index partitions

To speed up the creation process, increase the number of write capacity units (WCUs) on the index.

Global secondary indexes inherit the read or write capacity mode from the base table. If your table is in on-demand mode, then DynamoDB also creates the index in on-demand mode. In this case, you can't increase the capacity on the index, because an on-demand DynamoDB table scales itself based on incoming traffic.

Resolution

Use the OnlineIndexPercentageProgress Amazon CloudWatch metric to monitor the index creation progress:

1.    Open the DynamoDB console.

2.    In the navigation pane, choose Tables, and then select your table from the list.

3.    Choose the Metrics tab.

4.    Choose View all CloudWatch metrics.

5.    In the search box, enter OnlineIndexPercentageProgress.
Note: If the search returns no results, wait a minute or so for metrics to populate. Then, try again.

6.    Choose the name of the index to see the progress.

Determine the number of additional WCUs that you need. To do this, divide the table size in kilobytes by your desired backfill time. See the following examples of this calculation.

Example 1

Suppose that you have a 1 GiB (1,074,000 KB) table. You want the backfilling process to complete in 10 minutes (600 seconds). Therefore, calculate the number of WCUs as follows:

1,074,000 / 600 = 1,790 WCUs

Example 2

Suppose that you want the index to be 2 GB in size, and you want the index creation to be completed in one hour. Therefore, calculate the number of WCUs as follows:

(2GB * 1024 * 1024) KB / 60 minutes / 60 second = ~583 WCU The required number of WCUs depends on the index size and the time that you estimate.

Note: This is only an estimate. Creation time depends on multiple factors, such as your key distribution, the items' size, and the number of attributes that are projected into the index.

To provision additional write capacity, do the following:

1.    Open the DynamoDB console.

2.    In the navigation pane, choose Tables, and then select your table from the list.

3.    Choose the Capacity tab.

4.    Increase the write capacity of the index, and then choose Save.

5.    After about a minute, check the OnlineIndexPercentageProgress metric to see if the creation speed is improved.

Note: You don't need to provision additional read capacity.


Related information

Improving data access with secondary indexes

Adding a global secondary index to an existing table

Managing global secondary indexes

AWS OFICIAL
AWS OFICIALAtualizada há um ano