AWS glue from data catalog not working for mongodb

It seems you're encountering a few issues when trying to export data from MongoDB using AWS Glue. Let's address these problems step by step:

1. Missing collection configuration: When using AWS Glue Studio, you need to specify the database and collection names. Even though you've created a crawler and it successfully crawled your collection, this information might not be automatically passed to the Glue job. In the Data Catalog table properties or in your Glue Studio job configuration, ensure that you've specified both the database and collection names.

2. Authentication error: The "command aggregate requires authentication" error suggests that the job is not using the correct credentials to access MongoDB. This could happen for a few reasons:

a. The connection information might not be properly associated with your job. Make sure you've added the MongoDB connection as an "Additional network connection" in your AWS Glue job configuration.

b. If you're using AWS Secrets Manager to store your MongoDB credentials, ensure that the IAM role associated with your AWS Glue job has permission to read the secret.

c. Double-check that the connection details (URL, username, password) stored in your AWS Glue connection are correct and up-to-date.

To resolve these issues:

1. In AWS Glue Studio, when creating your MongoDB source node, make sure to specify the database and collection names in the node properties.

2. Verify that your Glue job's IAM role has the necessary permissions to access the MongoDB connection and any associated secrets.

3. Double-check your connection details in the AWS Glue connection configuration.

4. If you're using a custom script, ensure you're providing all necessary options, including the connection name, database, and collection. Your script should look something like this:

mongo_source = glueContext.create_dynamic_frame.from_catalog(
database="your_glue_database",
table_name="your_glue_table",
additional_options={
"database": "myCompany",
"collection": "myFavoriteCollection"
},
transformation_ctx="mongo_source"
)

5. If your MongoDB instance is within a VPC, ensure that your Glue job has the correct VPC configuration to access it.

By carefully reviewing and adjusting these settings, you should be able to successfully export your MongoDB collection using AWS Glue.
Sources
Using a MongoDB or MongoDB Atlas connection - AWS Glue
MongoDB connections - AWS Glue
Working with MongoDB connections in ETL jobs - AWS Glue
Connecting to MongoDB in AWS Glue Studio - AWS Glue

Hello Lieven,

I think you are encountering two separate issues when working with MongoDB and AWS Glue:

1. Missing Configuration for Collection:
   The error message (Missing configuration for: collection) occurs because AWS Glue expects you to provide the collection name explicitly when reading from MongoDB. The fix you applied to the script by including additional_options = {"database":"myCompany", "collection":"myFavoriteCollection"} seems to be the right approach, but the second error shows that there's an authentication issue.

2. Authentication Error (command aggregate requires authentication):
   This issue arises when the connection to MongoDB does not have the proper authentication or credentials to perform operations (like aggregation) on the collection.

Please try the below steps:

1. Verify Connection String and Authentication:

The crawler is able to access MongoDB because it is likely using valid credentials. Ensure that the same credentials are being used in your AWS Glue job. In your Glue job script, the connection string should look something like this:

additional_options = {
    "uri": "mongodb://<username>:<password>@<hostname>:<port>",
    "database": "myCompany",
    "collection": "myFavoriteCollection"
}

Double-check that:

• You are using the correct MongoDB URI, with valid username and password.
• The username provided has sufficient permissions to access the specific database and perform aggregation queries.

2. Glue Connection Configuration:

Make sure the Glue connection you’re using (which was likely created for the crawler) is correctly configured for your job:

• The connection should include the same authentication details and should point to the correct database.
• In the Glue job, ensure you are referencing the correct connection ID in the script:

glueContext.create_dynamic_frame.from_catalog(
    database="myCompany",
    table_name="myFavoriteCollection",
    additional_options={
        "uri": "mongodb://<username>:<password>@<hostname>:<port>",
        "database": "myCompany",
        "collection": "myFavoriteCollection"
    },
    connection_options={"connectionName": "mongodb-connection"}
)

3. Ensure Proper Role and Permissions:

Check if the AWS Glue job role has proper permissions to connect to the MongoDB instance. The IAM role assigned to the Glue job needs to have network access (e.g., through VPC peering or AWS Transit Gateway) and any necessary MongoDB permissions.

4. Verify MongoDB Authentication Mechanism:

Depending on how MongoDB is configured, it may require specific authentication mechanisms like SCRAM or X.509 certificates. Ensure that your Glue job is set up to use the correct mechanism. If using SCRAM, ensure the credentials passed are correct, and if using SSL/TLS, the Glue job might need additional certificates or options for secure connection.

5. Check MongoDB Logs:

You may want to review MongoDB logs to verify the reason for the failed authentication during the aggregation command. Sometimes this reveals misconfigurations in the credentials or access permissions for the user.

Please ensure that the credentials used in your Glue job match those successfully used by the crawler, and ensure that the MongoDB instance is properly configured to accept these credentials for read operations.