ECS Fargate writing and reading files is failing in a very odd way

I have a ECS service written in Go that is running on Fargate, and I am getting some very strange filesystem behavior that I cannot explain.

I have ephemeral storage mounted on /scratch, and my containers all run as root (because of course you need to apparently, storage other than EFS seems to be mounted as root owned, 755 mode...)

When I read and write with my stress test, as well as with my actual app, files I just wrote seem to vanish. I can write a file, and when I go to read it, it's gone...

The included test code writes 1,000 files, then reads them back, and then deletes them. This loop runs until it gets an error, or 1,000 times. This test works fine on my laptop, and in Kubernetes on both my home lab as well as EKS and GKE. No issues.

On my Fargate task:

2025/08/05 13:43:29 Iteration 99...
  loop sample path: /scratch/lakerunner/stress-test-3307732947
2025/08/05 13:43:29 Iteration 100...
  loop sample path: /scratch/lakerunner/stress-test-2039664141
Error: failed to read back file #668 (/scratch/lakerunner/stress-test-1072025920): open /scratch/lakerunner/stress-test-1072025920: no such file or directory

It fails randomly, sometimes as early as the first file.

I am stuck. I'm not sure where to go from here. I know I could use EFS, but I really just use the disk as scratch space, and that seems like a really rotten change. Local disk is very fast, EFS, not so much, and also of course costly.

Test code:

package cmd

import (
	"fmt"
	"log"
	"os"

	"github.com/spf13/cobra"
)

func init() {
	cmd := &cobra.Command{
		Use:   "diskstress",
		Short: "Run disk stress tests",
		RunE: func(_ *cobra.Command, _ []string) error {
			return stressTestTempFiles()
		},
	}

	rootCmd.AddCommand(cmd)
}

func stressTestTempFiles() error {
	fmt.Printf("Running disk stress test, tmpdir is %q\n", os.TempDir())

	for loop := range 1000 {
		log.Printf("Iteration %d...", loop+1)
		var filePaths []string

		for i := range 1000 {
			f, err := os.CreateTemp("", "stress-test-*")
			if err != nil {
				return fmt.Errorf("failed to create temp file #%d: %w", i, err)
			}

			n, err := f.WriteString("hello world")
			if err != nil {
				if err2 := f.Close(); err2 != nil {
					log.Printf("warning: failed to close file %s: %v", f.Name(), err2)
				}
				return fmt.Errorf("failed to write to file %s: %w", f.Name(), err)
			}
			if n != len("hello world") {
				if err2 := f.Close(); err2 != nil {
					log.Printf("warning: failed to close file %s: %v", f.Name(), err2)
				}
				return fmt.Errorf("short write to file %s: wrote %d bytes, expected %d", f.Name(), n, len("hello world"))
			}

			filePaths = append(filePaths, f.Name())
			if err2 := f.Close(); err2 != nil {
				log.Printf("warning: failed to close file %s: %v", f.Name(), err2)
			}
		}

		for i, path := range filePaths {
			if i == 0 {
				fmt.Printf("  loop sample path: %s\n", path)
			}
			data, err := os.ReadFile(path)
			if err != nil {
				return fmt.Errorf("failed to read back file #%d (%s): %w", i, path, err)
			}
			if string(data) != "hello world" {
				return fmt.Errorf("unexpected content in file %s: %q", path, data)
			}
		}

		for _, path := range filePaths {
			if err := os.Remove(path); err != nil {
				log.Printf("warning: failed to remove file %s: %v", path, err)
			}
		}
	}
	return nil
}