Access Forbidden when accessing manifest.json from S3 Batch operation

I am trying to apply an ACL to a set of objects. For that I am trying to use the new S3 Batch operations service.

I have a manifest.json file generated by the S3 Inventory service. I have created a job that takes a role with Full S3 permissions. The bucket policy provides full S3 access for that role to the contents of the bucket and the bucket.

However, I am getting this error: Reading the manifest is forbidden: AccessDenied.

I have checked the bucket policy and the ACL for both the bucket and the object and my account is the owner of the bucket and has full access to the object. The object seems to be owned by a service account.