AWS SFTP Error "Too many open files in this session, maximum 100"

Question

Since this monday we are experiencing a problem when we try to upload a large amount of files. (49 files to be exact). After around 20 files the upload fails.

```
s-262d99d7572942eca.server.transfer.eu-central-1.amazonaws.com

/aws/transfer/s-262d99d7572942eca asdf.f7955cc50d0d1bc4

2022-05-09T23:02:59.165+02:00 asdf.f7955cc50d0d1bc4 CONNECTED SourceIP=77.0.176.252 User=asdf HomeDir=/beresa-test-komola/asdf Client=SSH-2.0-ssh2js0.4.10 Role=arn:aws:iam::747969442112:role/beresa-test UserPolicy="{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Sid": "AllowListingOfUserFolder",
 "Action": [
 "s3:ListBucket"
 ],
 "Effect": "Allow",
 "Resource": [
 "arn:aws:s3:::beresa-test-komola"
 ],
 "Condition": {
 "StringLike": {
 "s3:prefix": [
 "asdf/*",
 "asdf"
 ]
 }
 }
 },
 {
 "Sid": "HomeDirObjectAccess",
 "Effect": "Allow",
 "Action": [
 "s3:PutObject",
 "s3:GetObject",
 "s3:DeleteObject",
 "s3:GetObjectVersion"
 ],
 "Resource": "arn:aws:s3:::beresa-test-komola/asdf*"
 }
 ]
}" Kex=ecdh-sha2-nistp256 Ciphers=aes128-ctr,aes128-ctr

2022-05-09T23:02:59.583+02:00 asdf.f7955cc50d0d1bc4 OPEN Path=/beresa-test-komola/asdf/x/bla/10_x_a_10.jpg Mode=CREATE|TRUNCATE|WRITE

2022-05-09T23:03:03.394+02:00 asdf.f7955cc50d0d1bc4 CLOSE Path=/beresa-test-komola/asdf/x/bla/10_x_a_10.jpg BytesIn=4226625

2022-05-09T23:03:04.005+02:00 asdf.f7955cc50d0d1bc4 OPEN Path=/beresa-test-komola/asdf/x/bla/11_x_a_1.jpg Mode=CREATE|TRUNCATE|WRITE

2022-05-09T23:03:07.215+02:00 asdf.f7955cc50d0d1bc4 CLOSE Path=/beresa-test-komola/asdf/x/bla/11_x_a_1.jpg BytesIn=4226625

2022-05-09T23:03:07.757+02:00 asdf.f7955cc50d0d1bc4 OPEN Path=/beresa-test-komola/asdf/x/bla/12_x_a_37.jpg Mode=CREATE|TRUNCATE|WRITE

2022-05-09T23:03:10.902+02:00 asdf.f7955cc50d0d1bc4 CLOSE Path=/beresa-test-komola/asdf/x/bla/12_x_a_37.jpg BytesIn=4226625

2022-05-09T23:03:11.433+02:00 asdf.f7955cc50d0d1bc4 OPEN Path=/beresa-test-komola/asdf/x/bla/13_x_a_13.jpg Mode=CREATE|TRUNCATE|WRITE

2022-05-09T23:03:14.579+02:00 asdf.f7955cc50d0d1bc4 CLOSE Path=/beresa-test-komola/asdf/x/bla/13_x_a_13.jpg BytesIn=4226625

2022-05-09T23:03:14.942+02:00 asdf.f7955cc50d0d1bc4 OPEN Path=/beresa-test-komola/asdf/x/bla/14_x_a_43.jpg Mode=CREATE|TRUNCATE|WRITE

2022-05-09T23:03:18.016+02:00 asdf.f7955cc50d0d1bc4 CLOSE Path=/beresa-test-komola/asdf/x/bla/14_x_a_43.jpg BytesIn=4226625

2022-05-09T23:03:18.403+02:00 asdf.f7955cc50d0d1bc4 OPEN Path=/beresa-test-komola/asdf/x/bla/15_x_a_34.jpg Mode=CREATE|TRUNCATE|WRITE

2022-05-09T23:03:21.463+02:00 asdf.f7955cc50d0d1bc4 CLOSE Path=/beresa-test-komola/asdf/x/bla/15_x_a_34.jpg BytesIn=4226625

2022-05-09T23:03:21.906+02:00 asdf.f7955cc50d0d1bc4 OPEN Path=/beresa-test-komola/asdf/x/bla/16_x_a_44.jpg Mode=CREATE|TRUNCATE|WRITE

2022-05-09T23:03:25.025+02:00 asdf.f7955cc50d0d1bc4 CLOSE Path=/beresa-test-komola/asdf/x/bla/16_x_a_44.jpg BytesIn=4199266

2022-05-09T23:03:25.431+02:00 asdf.f7955cc50d0d1bc4 OPEN Path=/beresa-test-komola/asdf/x/bla/17_x_a_2.jpg Mode=CREATE|TRUNCATE|WRITE

2022-05-09T23:03:28.497+02:00 asdf.f7955cc50d0d1bc4 CLOSE Path=/beresa-test-komola/asdf/x/bla/17_x_a_2.jpg BytesIn=4199266

2022-05-09T23:03:28.857+02:00 asdf.f7955cc50d0d1bc4 OPEN Path=/beresa-test-komola/asdf/x/bla/18_x_a_5.jpg Mode=CREATE|TRUNCATE|WRITE

2022-05-09T23:03:31.947+02:00 asdf.f7955cc50d0d1bc4 CLOSE Path=/beresa-test-komola/asdf/x/bla/18_x_a_5.jpg BytesIn=4199266

2022-05-09T23:03:32.374+02:00 asdf.f7955cc50d0d1bc4 OPEN Path=/beresa-test-komola/asdf/x/bla/19_x_a_8.jpg Mode=CREATE|TRUNCATE|WRITE

2022-05-09T23:03:35.504+02:00 asdf.f7955cc50d0d1bc4 CLOSE Path=/beresa-test-komola/asdf/x/bla/19_x_a_8.jpg BytesIn=4199266

2022-05-09T23:03:35.986+02:00 asdf.f7955cc50d0d1bc4 OPEN Path=/beresa-test-komola/asdf/x/bla/1_x_a_16.jpg Mode=CREATE|TRUNCATE|WRITE

2022-05-09T23:03:39.104+02:00 asdf.f7955cc50d0d1bc4 CLOSE Path=/beresa-test-komola/asdf/x/bla/1_x_a_16.jpg BytesIn=4226625

2022-05-09T23:03:39.691+02:00 asdf.f7955cc50d0d1bc4 OPEN Path=/beresa-test-komola/asdf/x/bla/20_x_a_11.jpg Mode=CREATE|TRUNCATE|WRITE

2022-05-09T23:03:42.816+02:00 asdf.f7955cc50d0d1bc4 CLOSE Path=/beresa-test-komola/asdf/x/bla/20_x_a_11.jpg BytesIn=4199266

2022-05-09T23:03:43.224+02:00 asdf.f7955cc50d0d1bc4 OPEN Path=/beresa-test-komola/asdf/x/bla/21_x_a_14.jpg Mode=CREATE|TRUNCATE|WRITE

2022-05-09T23:03:46.274+02:00 asdf.f7955cc50d0d1bc4 CLOSE Path=/beresa-test-komola/asdf/x/bla/21_x_a_14.jpg BytesIn=4199266

2022-05-09T23:03:46.649+02:00 asdf.f7955cc50d0d1bc4 OPEN Path=/beresa-test-komola/asdf/x/bla/22_x_a_17.jpg Mode=CREATE|TRUNCATE|WRITE

2022-05-09T23:03:49.757+02:00 asdf.f7955cc50d0d1bc4 CLOSE Path=/beresa-test-komola/asdf/x/bla/22_x_a_17.jpg BytesIn=4199266

2022-05-09T23:03:50.141+02:00 asdf.f7955cc50d0d1bc4 OPEN Path=/beresa-test-komola/asdf/x/bla/23_x_a_20.jpg Mode=CREATE|TRUNCATE|WRITE

2022-05-09T23:03:53.307+02:00 asdf.f7955cc50d0d1bc4 CLOSE Path=/beresa-test-komola/asdf/x/bla/23_x_a_20.jpg BytesIn=4199266

2022-05-09T23:03:53.849+02:00 asdf.f7955cc50d0d1bc4 OPEN Path=/beresa-test-komola/asdf/x/bla/24_x_a_23.jpg Mode=CREATE|TRUNCATE|WRITE

2022-05-09T23:03:56.933+02:00 asdf.f7955cc50d0d1bc4 CLOSE Path=/beresa-test-komola/asdf/x/bla/24_x_a_23.jpg BytesIn=4199266

2022-05-09T23:03:57.358+02:00 asdf.f7955cc50d0d1bc4 OPEN Path=/beresa-test-komola/asdf/x/bla/25_x_a_26.jpg Mode=CREATE|TRUNCATE|WRITE

2022-05-09T23:04:00.585+02:00 asdf.f7955cc50d0d1bc4 CLOSE Path=/beresa-test-komola/asdf/x/bla/25_x_a_26.jpg BytesIn=4199266

2022-05-09T23:04:00.942+02:00 asdf.f7955cc50d0d1bc4 OPEN Path=/beresa-test-komola/asdf/x/bla/26_x_a_29.jpg Mode=CREATE|TRUNCATE|WRITE

2022-05-09T23:04:04.174+02:00 asdf.f7955cc50d0d1bc4 CLOSE Path=/beresa-test-komola/asdf/x/bla/26_x_a_29.jpg BytesIn=4199266

2022-05-09T23:04:04.603+02:00 asdf.f7955cc50d0d1bc4 OPEN Path=/beresa-test-komola/asdf/x/bla/27_x_a_32.jpg Mode=CREATE|TRUNCATE|WRITE

2022-05-09T23:04:07.771+02:00 asdf.f7955cc50d0d1bc4 CLOSE Path=/beresa-test-komola/asdf/x/bla/27_x_a_32.jpg BytesIn=4199266

2022-05-09T23:04:08.179+02:00 asdf.f7955cc50d0d1bc4 OPEN Path=/beresa-test-komola/asdf/x/bla/28_x_a_35.jpg Mode=CREATE|TRUNCATE|WRITE

2022-05-09T23:04:11.279+02:00 asdf.f7955cc50d0d1bc4 CLOSE Path=/beresa-test-komola/asdf/x/bla/28_x_a_35.jpg BytesIn=4199266

2022-05-09T23:04:11.716+02:00 asdf.f7955cc50d0d1bc4 OPEN Path=/beresa-test-komola/asdf/x/bla/29_x_a_38.jpg Mode=CREATE|TRUNCATE|WRITE

2022-05-09T23:04:14.853+02:00 asdf.f7955cc50d0d1bc4 CLOSE Path=/beresa-test-komola/asdf/x/bla/29_x_a_38.jpg BytesIn=4199266

2022-05-09T23:04:15.316+02:00 asdf.f7955cc50d0d1bc4 OPEN Path=/beresa-test-komola/asdf/x/bla/2_x_a_7.jpg Mode=CREATE|TRUNCATE|WRITE

2022-05-09T23:04:18.435+02:00 asdf.f7955cc50d0d1bc4 CLOSE Path=/beresa-test-komola/asdf/x/bla/2_x_a_7.jpg BytesIn=4226625

2022-05-09T23:04:18.906+02:00 asdf.f7955cc50d0d1bc4 OPEN Path=/beresa-test-komola/asdf/x/bla/30_x_a_41.jpg Mode=CREATE|TRUNCATE|WRITE

2022-05-09T23:04:22.140+02:00 asdf.f7955cc50d0d1bc4 CLOSE Path=/beresa-test-komola/asdf/x/bla/30_x_a_41.jpg BytesIn=4199266

2022-05-09T23:04:22.565+02:00 asdf.f7955cc50d0d1bc4 OPEN Path=/beresa-test-komola/asdf/x/bla/31_x_a_18.jpg Mode=CREATE|TRUNCATE|WRITE

2022-05-09T23:04:25.752+02:00 asdf.f7955cc50d0d1bc4 CLOSE Path=/beresa-test-komola/asdf/x/bla/31_x_a_18.jpg BytesIn=4159129

2022-05-09T23:04:26.141+02:00 asdf.f7955cc50d0d1bc4 ERROR Message="Too many open files in this session, maximum 100" Operation=OPEN Path=/beresa-test-komola/asdf/x/bla/32_x_a_3.jpg Mode=CREATE|TRUNCATE|WRITE
```

As you can see in the logs we are closing each path after opening it - we are uploading one file after the other. What could cause this as we are not even trying to write 100 files during the scp session?

Accepted Answer

The SFTP server is limiting the number of files that can remain open within the span of a single session. Although the message indicates files this count includes directory listings as well. Depending on your client software and how it is used it is possible to leave a listing operation in a directory improperly closed resulting in a "file handle" leak. This is fixable by identifying the problem code path on the client side and closing the open directory handle. The limitation of 100 open file handles is a new enforcement policy that was introduced this week.

AWS SFTP Error "Too many open files in this session, maximum 100"

Relevant questions

Enabling S3 Encryption-at-rest on a go-forward basis with s3fs

How can I optimize performance when uploading large files to S3 across multiple AWS Regions?

Is there an option in 'My Files' for a user to delete files?

S3 Intelligent - Tiering - 'Monitoring and Automation' costs

Can't upload files -- anyone else having problem

MongoError: Too many fields

S3 Bucket Public Access Settings

RDS Missing Cloudwatch Metrics + Too many open files

AWS SFTP Error "Too many open files in this session, maximum 100"

Clearing files on EFS from Lambda after DataSync Operation from S3