How to dump from DocumentDB 5.0 and restore to DocumentDB 4.0/MongoDB/Mongo Atlas

Background

Amazon DocumentDB 5.0 introduced a new feature: collection-level document compression. This feature helps reduce storage and I/O costs by compressing documents in collections. When creating a collection in DocumentDB 5.0, you may notice a new field, storageEngine, which is set to "disabled" by default.

However, when using mongodump to dump collections from DocumentDB 5.0, the storageEngine field is included in the collection metadata. This field causes an issue when attempting to restore the dump into DocumentDB 4.0, MongoDB, or Mongo Atlas because the storageEngine field is not recognized in these platforms. You would see different error messages on different platforms:

• Mongo: documentDB is not a registered storage engine for this server

• DocumentDB 4.0: error running create command: Field 'storageEngine' is currently not supported

Solution

To resolve this, we need to remove the storageEngine field from the metadata files in the dump before restoring to DocumentDB 4.0, MongoDB, or Mongo Atlas. Here’s how to do it:

1. Remove the storageEngine Field from Metadata Files:

After dumping your collections from DocumentDB 5.0, we need to delete the filed "storageEngine" in the dump metadata files.

Use the following script to delete the storageEngine field from the metadata files. Make sure jq is installed and the jq shell script is put under dump/.

#!/bin/bash

for directory in *; do
    if [ -d "${directory}" ] ; then
        for metadata_file in $directory/*.metadata.json; do
            echo $metadata_file
            result=$(jq 'del(.options.storageEngine)' $metadata_file)
            echo $result > $metadata_file
        done
    fi
done

This script processes each metadata file, removes the storageEngine field, and updates the file with the corrected metadata.

2. Restore the Dump:

Once the storageEngine field has been removed from the metadata files, you can proceed to restore the dump to DocumentDB 4.0, MongoDB, or Mongo Atlas without errors using the following sample command:

mongorestore --drop -d local sample_xxx/

By following these steps, you can successfully migrate your data from DocumentDB 5.0 to older versions of DocumentDB, MongoDB, or Mongo Atlas without encountering issues related to the storageEngine field.