EFS Size not increasing

Question

I have mounted the EFS as Kubernetes volume in the OpenSearch instance running in the EKS Cluster but the size of EFS remains 30kb.
Initially, I thought my OpenSearch wasn't using EFS for storing the traces because its size was not increasing but when I removed the same and ran again I got the below error in Jaeger UI which is using OpenSearch as the backend storage, therefore, I restored the EFS, and the error wasn't coming again so can anyone tell me the reason that if OpenSearch is using EFS for storing the traces then why it's size is not increasing.

Although I can see the size of spans of my application as below in OpenSearch Dashboard which is also using OpenSearch as backend storage.

![trace size](/media/postImages/original/IM0B7Jq5JuQBqdroc5m8CwqQ)

```
{"level":"info","ts":1706710346.7175736,"caller":"spanstore/reader.go:583","msg":"es search services failed","traceQuery":{"ServiceName":"Prior-Auth","OperationName":"HTTP GET","Tags":{},"StartTimeMin":"2024-01-31T13:12:26.252Z","StartTimeMax":"2024-01-31T14:12:26.252Z","DurationMin":0,"DurationMax":0,"NumTraces":20},"error":"elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]"}
{"level":"error","ts":1706710346.717812,"caller":"app/http_handler.go:492","msg":"HTTP handler, Internal Server Error","error":"search services failed: elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]","stacktrace":"github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleError
	github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:492
github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search
	github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:241
net/http.HandlerFunc.ServeHTTP
	net/http/server.go:2136
github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleFunc.traceResponseHandler.func2
	github.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:536
net/http.HandlerFunc.ServeHTTP
	net/http/server.go:2136
github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleFunc.WithRouteTag.func3
	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.47.0/handler.go:281
net/http.HandlerFunc.ServeHTTP
	net/http/server.go:2136
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP
	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.47.0/handler.go:225
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1
	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.47.0/handler.go:83
net/http.HandlerFunc.ServeHTTP
	net/http/server.go:2136
net/http.HandlerFunc.ServeHTTP
	net/http/server.go:2136
github.com/gorilla/mux.(*Router).ServeHTTP
	github.com/gorilla/mux@v1.8.1/mux.go:212
github.com/jaegertracing/jaeger/cmd/query/app.createHTTPServer.additionalHeadersHandler.func4
	github.com/jaegertracing/jaeger/cmd/query/app/additional_headers_handler.go:28
net/http.HandlerFunc.ServeHTTP
	net/http/server.go:2136
github.com/jaegertracing/jaeger/cmd/query/app.createHTTPServer.CompressHandler.CompressHandlerLevel.func6
	github.com/gorilla/handlers@v1.5.1/compress.go:141
net/http.HandlerFunc.ServeHTTP
	net/http/server.go:2136
github.com/gorilla/handlers.recoveryHandler.ServeHTTP
	github.com/gorilla/handlers@v1.5.1/recovery.go:78
net/http.serverHandler.ServeHTTP
	net/http/server.go:2938
net/http.(*conn).serve
	net/http/server.go:2009"}
```

Answer

When you mount an Amazon EFS (Elastic File System) as a Kubernetes volume and use it to store data, the size of the EFS file system might not appear to increase immediately. EFS automatically scales as you add more data to it, and the reported size might not always reflect the actual data usage accurately. You can use commands like df -h on your EKS nodes to check the mounted EFS file systems. Additionally, AWS CloudWatch metrics, including those for EFS, might experience some delay in updates. Wait for a period, and then recheck the metrics to see if they accurately reflect the current usage.

Hope it clarifies and if does I would appreciate answer to be accepted so that community can benefit for clarity, thanks ;)

Hope it clarifies and if does I would  appreciate answer to be accepted so that community can benefit for clarity, thanks ;)

EFS Size not increasing

Relevant content