InfluxDB3 fragmentation parquet files

Question

I am running InfluxDB 3 on AWS Timestream for InfluxDB (db.influx.medium instance). We have had persistent issues with query performance due to high fragmentation of Parquet files, and we are repeatedly hitting the configured query limit of 432 Parquet files.

I only have 1 tag and 6 fields. Is it possible to eliminate this fragmentation? How AWS manage this fragmentation?

Taz · Answer

Hey,

Hope you're keeping well.

In Amazon Timestream for InfluxDB, Parquet file fragmentation is primarily driven by how data is ingested, partitioned, and compacted in the storage engine. High fragmentation often occurs when data arrives in small batches or with many distinct time intervals, which limits the engine’s ability to merge files. To reduce fragmentation, try batching writes into larger time windows, ensure ingestion is ordered by time, and avoid excessive small writes. AWS automatically runs background compaction, but it can only merge files when data distribution and ingestion patterns allow.

Thanks and regards,  
Taz

InfluxDB3 fragmentation parquet files

Add your answer

Relevant content