AWS Backup jobs for EFS file systems are empty

Question

I have a backup plan:

```json
{
    "BackupPlan": {
        "BackupPlanName": "prometheus_dev",
        "Rules": [
            {
                "RuleName": "prometheus_dev",
                "TargetBackupVaultName": "xxxxxxxxxxxx",
                "ScheduleExpression": "cron(15 * ? * * *)",
                "StartWindowMinutes": 60,
                "CompletionWindowMinutes": 180,
                "Lifecycle": {
                    "DeleteAfterDays": 7
                },
                "RuleId": "ffcd7e8c-9b14-4e2b-89f0-d8cbe5b5ae25",
                "CopyActions": [
                    {
                        "Lifecycle": {
                            "DeleteAfterDays": 7
                        },
                        "DestinationBackupVaultArn": "arn:aws:backup:eu-west-1:614797193252:backup-vault:xxxxxxxxxxxx"
                    }
                ],
                "EnableContinuousBackup": false
            }
        ]
    },
    "BackupPlanId": "830e8bb0-e3c6-4a96-8fe9-45a3ab97de40",
    "BackupPlanArn": "arn:aws:backup:eu-central-1:614797193252:backup-plan:830e8bb0-e3c6-4a96-8fe9-45a3ab97de40",
    "VersionId": "YzUxMjhkODgtZTc0MC00NDA1LWEwYzktNmE4NDFhMDE5MTA4",
    "CreationDate": "2023-10-13T13:42:22.048000+02:00",
    "LastExecutionDate": "2023-10-20T11:16:48.451000+02:00"
}
```

The file system is protected:

```json
{
    "ResourceArn": "arn:aws:elasticfilesystem:eu-central-1:614797193252:file-system/fs-0318e5506f10caf1e",
    "ResourceType": "EFS",
    "LastBackupTime": "2023-10-20T11:22:48.159000+02:00",
    "ResourceName": "prometheus_dev"
}
```

And backups are taken hourly:

```json
// ...
{
            "AccountId": "614797193252",
            "BackupJobId": "06178992-5DF7-17BA-1E38-C969357B644A",
            "BackupVaultName": "xxxxxxxxxxxx",
            "BackupVaultArn": "arn:aws:backup:eu-central-1:614797193252:backup-vault:xxxxxxxxxxxx",
            "RecoveryPointArn": "arn:aws:backup:eu-central-1:614797193252:recovery-point:58cb7957-5b0c-4038-a579-af78aadbc506",
            "ResourceArn": "arn:aws:elasticfilesystem:eu-central-1:614797193252:file-system/fs-0318e5506f10caf1e",
            "CreationDate": "2023-10-20T08:15:00+02:00",
            "CompletionDate": "2023-10-20T08:25:07.003000+02:00",
            "State": "COMPLETED",
            "PercentDone": "100.0",
            "BackupSizeInBytes": 0,
            "IamRoleArn": "arn:aws:iam::614797193252:role/xxxxxxxxxxxx",
            "CreatedBy": {
                "BackupPlanId": "830e8bb0-e3c6-4a96-8fe9-45a3ab97de40",
                "BackupPlanArn": "arn:aws:backup:eu-central-1:614797193252:backup-plan:830e8bb0-e3c6-4a96-8fe9-45a3ab97de40",
                "BackupPlanVersion": "YzUxMjhkODgtZTc0MC00NDA1LWEwYzktNmE4NDFhMDE5MTA4",
                "BackupRuleId": "ffcd7e8c-9b14-4e2b-89f0-d8cbe5b5ae25"
            },
            "StartBy": "2023-10-20T09:15:00+02:00",
            "ResourceType": "EFS",
            "IsParent": false,
            "ResourceName": "prometheus_dev"
        },
        {
            "AccountId": "614797193252",
            "BackupJobId": "DE456E1F-1A49-4A36-507A-32646DD0AE85",
            "BackupVaultName": "prometheus_eu_central_1_dev",
            "BackupVaultArn": "arn:aws:backup:eu-central-1:614797193252:backup-vault:xxxxxxxxxxxx",
            "RecoveryPointArn": "arn:aws:backup:eu-central-1:614797193252:recovery-point:998837d7-e0c8-4505-9d4f-ff19ca1f69c7",
            "ResourceArn": "arn:aws:elasticfilesystem:eu-central-1:614797193252:file-system/fs-0318e5506f10caf1e",
            "CreationDate": "2023-10-20T07:15:00+02:00",
            "CompletionDate": "2023-10-20T07:25:52.381000+02:00",
            "State": "COMPLETED",
            "PercentDone": "100.0",
            "BackupSizeInBytes": 0,
            "IamRoleArn": "arn:aws:iam::614797193252:role/xxxxxxxxxxxx",
            "CreatedBy": {
                "BackupPlanId": "830e8bb0-e3c6-4a96-8fe9-45a3ab97de40",
                "BackupPlanArn": "arn:aws:backup:eu-central-1:614797193252:backup-plan:830e8bb0-e3c6-4a96-8fe9-45a3ab97de40",
                "BackupPlanVersion": "YzUxMjhkODgtZTc0MC00NDA1LWEwYzktNmE4NDFhMDE5MTA4",
                "BackupRuleId": "ffcd7e8c-9b14-4e2b-89f0-d8cbe5b5ae25"
            },
            "StartBy": "2023-10-20T08:15:00+02:00",
            "ResourceType": "EFS",
            "IsParent": false,
            "ResourceName": "prometheus_dev"
        },
// ...
```

The problem is that ALL recovery points are empty, even if the file system is not:

```json
{
// ...
            "SizeInBytes": {
                "Value": 67465216,
                "Timestamp": "2023-10-20T10:41:44+02:00",
                "ValueInIA": 0,
                "ValueInStandard": 67465216
            }
// ...
}
```

I've also tried to start a backup job:

```bash
aws backup start-backup-job \
    --backup-vault-name prometheus_eu_central_1_dev \
    --resource-arn arn:aws:elasticfilesystem:eu-central-1:614797193252:file-system/fs-0318e5506f10caf1e \
    --iam-role-arn arn:aws:iam::614797193252:role/xxxxxxxxxxxx
```

Which produced the following response:

```json
{
    "BackupJobId": "7a010939-11cd-4d6f-bf2e-bbec0fc50452",
    "RecoveryPointArn": "arn:aws:backup:eu-central-1:614797193252:recovery-point:ac338f60-9a4e-4c24-817e-2e0e15a72d03",
    "CreationDate": "2023-10-20T11:22:48.159000+02:00",
    "IsParent": false
}
```

But again the backup is empty:

```json
{
    "AccountId": "614797193252",
    "BackupJobId": "7a010939-11cd-4d6f-bf2e-bbec0fc50452",
    "BackupVaultName": "prometheus_eu_central_1_dev",
    "BackupVaultArn": "arn:aws:backup:eu-central-1:614797193252:backup-vault:xxxxxxxxxxxx",
    "RecoveryPointArn": "arn:aws:backup:eu-central-1:614797193252:recovery-point:ac338f60-9a4e-4c24-817e-2e0e15a72d03",
    "ResourceArn": "arn:aws:elasticfilesystem:eu-central-1:614797193252:file-system/fs-0318e5506f10caf1e",
    "CreationDate": "2023-10-20T11:22:48.159000+02:00",
    "CompletionDate": "2023-10-20T11:22:55.105000+02:00",
    "State": "COMPLETED",
    "PercentDone": "100.0",
    "BackupSizeInBytes": 0,
    "IamRoleArn": "arn:aws:iam::614797193252:role/xxxxxxxxxxxx",
    "ResourceType": "EFS",
    "BytesTransferred": 0,
    "StartBy": "2023-10-20T19:22:48.159000+02:00",
    "IsParent": false,
    "ResourceName": "prometheus_dev"
}
```

Why are these backups empty?

[1]: https://i.stack.imgur.com/6Rz8p.png
  [2]: https://i.stack.imgur.com/73LbW.png
  [3]: https://i.stack.imgur.com/ycjiH.png

Answer

Turns out that, since backups are incremental, `BackupSizeInBytes` will be `0` if there were no changes.

The backup is being restored to a folder like `aws-backup-restore_2023-10-20T12-57-24-237975193Z` This happens because EFS does not perform destructive restores, even for new file systems: https://docs.aws.amazon.com/aws-backup/latest/devguide/restoring-efs.html

We can automate moving files from the backup folder to their original path using a Lambda, for example.

Finally, the console will show an empty restored filesystem, ie, 6kb. This updates eventually to the correct size.

AWS Backup jobs for EFS file systems are empty

Relevanter Inhalt