[Aws Glue DataBrew] - Does DataBrew has dataset limit size?
0
Hi!
In the DataBrew, I'm trying to import a S3 dataset with aprox. 5 millions rows - I selected a job profile to run on the full dataset. The job run successfully but the number of rows are less than the full dataset (aprox. 2.5millions)
Can you help me to understand if is there any size limitation? It's weird because there isn't any error during the process and in the Data profile overwiew is written: "Data profile was run on full dataset."
I selected the full dataset, so the job profile should run in the entire dataset. The problem is that the job run successfully but not on the entire dataset (only 2.5million rows were considered out of 5 millions).
AWS OFFICIALAktualisiert vor 2 Jahren
I think it depends on mode you selected. Did you provided value? https://docs.aws.amazon.com/databrew/latest/dg/API_JobSample.html
Hi! thanks for your comment.
I selected the full dataset, so the job profile should run in the entire dataset. The problem is that the job run successfully but not on the entire dataset (only 2.5million rows were considered out of 5 millions).