Hi,
I'm loading a set of files from S3 and filtering the dataset to files from the last hour.
See error in the Cloudwatch log below:
Processing parameterized path with the following parameters:
{
"LastModifiedDateCondition": {
"Expression": "relative_after :date1",
"ValuesMap": {
":date1": "-1H"
}
},
"Parameters": {
"fileDate": {
"CreateColumn": true,
"DatetimeOptions": {
"Format": "MMddyyyy"
},
"Name": "fileDate",
"Type": "Datetime"
},
...
ERROR - Failed to load dataset from 's3://<<bucket>/data/<folder>_P{fileDate}_R{fileEndDate}T{fileTime}_{suffixNum}/<filename>.csv'. Exception can't compare offset-naive and offset-aware datetime
This appears to be new as the jobs were running perfectly a few days ago. I also tested it by trying to load the project and that fails with the same error. Removing the filter from the dataset allows me to load the project.
Did some new update break this functionality?
AWS UFFICIALEAggiornata 2 anni fa
Maybe you have some new timezones in the data that aren't valid/recognised. eg '2023-04-12T00:00:00+20:45'
5376 - Thanks for the response. But does not have to do with data -- this is databrew's dataset filter function to simply pick up files that have modified times in S3 in the prior hour (in my case) from execution time of the job. Still, nothing in the data/file convention etc. has changed and this was tested by just removing the filter and allowing the job to run -- which it did.
@emencee we have started having this exact same issue too. It started occurring randomly on the 5th April.
AWS Team has identified this as a bug and as per support, they will push out a release this week.