.NET 6 Lambda - Cannot use file stream for [/var/task/xxx.runtimeconfig.json]: Permission denied

Question

When re-deploying a cloudformation stack with .NET 6 lambdas referenced through a zip file stored on s3, a lot of the instances run into an error, lambda logs below, xxx being redacted project name:

INIT_START Runtime Version ARN: arn:aws:lambda:eu-west-1::runtime:c3ade6bc354f465cf7a9bb7193500f1ec1c23b10c575d100dd086311be8854c1

Cannot use file stream for [/var/task/xxx.runtimeconfig.json]: Permission denied

Invalid runtimeconfig.json [/var/task/xxx.runtimeconfig.json] [/var/task/xxx.runtimeconfig.dev.json]

RequestId: 13c5100b-c944-45cd-b5b8-923b2ad16e32 Error: Runtime exited with error: exit status 147 Runtime.ExitError

This issue seems to be transient and usually fixes itself after an hour or so and does not occur on every deploy, but still leads to an hour of reduced throughput with ~30% success rate on multiple lambdas when it does.  It looks like it only affects particular instances, which repeatedly log the same error on every invocation. Eventually they are replaced with healthy instances which no longer have the error but as mentioned this could take up to an hour.
Since its transient, it does not look like a permission issue. We've checked that all the contents are in the zip file we uploaded to s3 (including the runtimeconfig.json file) and the zip file and the runtimeconfig file is present and has full write permissions (-rw-rw-r--).
Best guess is to some kind of race happening in VM startup that eventually wins out as errored instances are replaced with healthy ones.
A similar issue happened with .NET 3.1 but believe the error file in question was *.deps.json with a similar error message, we were hoping upgrading the runtime would fix this issue.
It's quite hard to recreate to further diagnose this as this issue only seems to happen in our production environment, where we can peak at ~5k invocations per function so it could also be due to some fight over resources.

runtimeconfig.json file:

`{
  "runtimeOptions": {
    "tfm": "net6.0",
    "framework": {
      "name": "Microsoft.NETCore.App",
      "version": "6.0.0"
    },
    "configProperties": {
      "System.Reflection.Metadata.MetadataUpdater.IsSupported": false
    }
  }
}`

csproj file

`
    
        net6.0
        ...
        ...
        None
        true
        false

...

`

Regards,
Andrew

Answer

Please check s3 storage type.

.NET 6 Lambda - Cannot use file stream for [/var/task/xxx.runtimeconfig.json]: Permission denied

Relevant content