How do I interpret AWS DataSync enhanced log events?

Question

I don't understand why a 'DELETE' action is wanting to _read_ anything in this log message from DataSync.

```
{
  "ErrorDetail": "DataSync was unable to read the file or folder from the source: Access Denied",
  "Destination": {
    "RelativePath": "310725/ed2f13714e439fcbc2d90a0431d2f058-eng.pdf",
    "LocationId": "loc-0ba5cd337a087e624"
  },
  "Action": "DELETE",
  "id": "39192308530471220971359362916837867874364289242993198102",
  "ErrorCode": "UnableToReadSource",
  "aws": {
    "firehose": {
      "arn": "arn:aws:firehose:us-east-1:362454057537:deliverystream/datadog-stream-data-ingest",
      "deliverystream": "datadog-stream-data-ingest"
    },
    "awslogs": {
      "owner": "362454057537",
      "logGroup": "/aws/datasync",
      "logStream": "task-0685c02fa541f0a9f-exec-0bdd747ae6aa83e56"
    }
  },
  "Source": {
    "RelativePath": "310725/ed2f13714e439fcbc2d90a0431d2f058-eng.pdf",
    "LocationId": "loc-06409676937dcd4a6_eu-west-1"
  },
  "timestamp": 1757443896227
}
```

Darryl_Dio · Answer

Hi bimargulies,

When PreserveDeletedFiles is set to REMOVE [1], the deletion of files on the destination that do not exist on the source is prevented if an affected file or object cannot be read during the transfer phase. This occurs when an object is inventoried when listing the source, but becomes unreadable during the transfer phase.

This behavior is intentionally designed to prevent accidental deletions when files that were expected to be transferred can no longer be read. Such situations may arise due to:
- Failure to connect to the source
- Changes in permissions
- Timeouts during read operations

DataSync Enhanced logs will show:
```"ErrorDetail": "DataSync was unable to read the file or folder from the source: Access Denied"```
for ``` "Action": "DELETE"```

[1] https://docs.aws.amazon.com/datasync/latest/userguide/API_Options.html#DataSync-Type-Options-PreserveDeletedFiles

Thanks

bimargulies · Answer

The problem here was objectTags. I turned on object tag copying without realizing that this required more permission to the source than I had. That resulted in errors all around, on both deletes and additions. It became obvious when the first new document appeared at the source, producing an error message that did not seem to be self-contradicting.

re:Post Agent · Answer

In AWS DataSync's Enhanced mode logs, the "DELETE" action in your log message indicates that DataSync was attempting to delete a file at the destination location that doesn't exist in the source location.

When DataSync performs a DELETE action, it first needs to read the source location to confirm that the file is indeed missing from the source before deleting it from the destination. The error "UnableToReadSource" with detail "DataSync was unable to read the file or folder from the source: Access Denied" suggests that DataSync couldn't access the source location to verify whether the file exists there.

This typically happens when:
1. The DataSync task is configured to delete files in the destination that don't exist in the source
2. DataSync needs to check if the file exists in the source before deleting it from the destination
3. There's a permissions issue preventing DataSync from reading the source location

The Enhanced mode logs provide structured JSON data that makes it easier to understand what's happening during the task execution. In this case, you can see:
- The specific file path being processed
- The source and destination location IDs
- The exact error code and description
- Timestamp of when this occurred

To resolve this issue, you would need to check the permissions for the DataSync task at the source location to ensure it has proper read access to verify file existence.  
**Sources**  
[Monitoring data transfers with Amazon CloudWatch Logs - AWS DataSync](https://docs.aws.amazon.com/datasync/latest/userguide/configure-logging.html)  
[Troubleshooting data verification issues - AWS DataSync](https://docs.aws.amazon.com/datasync/latest/userguide/troubleshooting-task-verification.html)  
[Monitoring your data transfers with task reports - AWS DataSync](https://docs.aws.amazon.com/datasync/latest/userguide/task-reports.html)

How do I interpret AWS DataSync enhanced log events?

Añade tu respuesta

Contenido relevante