Timestream InfluxDB Service Recovery

Question

I created a parameter group and adjusted the queryConcurrency and queryQueueSize to 0. I applied the group to the running Timestream InfluxDB database. After a while, it went into a **FAILED** status.

I can still connect to the database and run influx cli command, but the database is performing poorly. At this point, I have no idea how to recover the database. Any recommendations are welcome.

Here are the results from the cli commands:

```
aws timestream-influxdb get-db-instance --identifier [REDACTED]                                                                                                                                                                                                                                                                    
{
    "id": "[REDACTED]",
    "name": "[REDACTED]",
    "arn": "[REDACTED]",
    "status": "FAILED",
    "endpoint": "[REDACTED]timestream-influxdb.amazonaws.com",
    "dbInstanceType": "db.influx.large",
    "dbStorageType": "InfluxIOIncludedT1",
    "allocatedStorage": 250,
    "deploymentType": "WITH_MULTIAZ_STANDBY",
    "vpcSubnetIds": [REDACTED],
    "publiclyAccessible": false,
    "vpcSecurityGroupIds": [REDACTED],
    "availabilityZone": "us-east-1c",
    "secondaryAvailabilityZone": "us-east-1b",
    "logDeliveryConfiguration": {[REDACTED]},
    "influxAuthParametersSecretArn": "[REDACTED]"
}
```

```
aws timestream-influxdb update-db-instance --identifier [REDACTED] --db-parameter-group-identifier [REDACTED]

An error occurred (ValidationException) when calling the UpdateDbInstance operation: DbInstance with identifier [REDACTED] and customer id [REDACTED] is not in an updatable state
```

I can tell the settings didn't take by using the influx cli to look at the settings:

```
influx server-config
{
        "assets-path": "",
        "bolt-path": "/influx/metadata/influxdb.bolt",
        "e2e-testing": false,
        "engine-path": "/influx/data",
        "feature-flags": null,
        "flux-log-enabled": false,
        "hardening-enabled": false,
        "http-bind-address": ":8086",
        "http-idle-timeout": 180000000000,
        "http-read-header-timeout": 10000000000,
        "http-read-timeout": 0,
        "http-write-timeout": 0,
        "influxql-max-select-buckets": 0,
        "influxql-max-select-point": 0,
        "influxql-max-select-series": 0,
        "instance-id": "s9nr3pjwv0",
        "log-level": "info",
        "metrics-disabled": false,
        "nats-max-payload-bytes": 0,
        "nats-port": 0,
        "no-tasks": false,
        "pprof-disabled": true,
        "query-concurrency": 1024,
        "query-initial-memory-bytes": 0,
        "query-max-memory-bytes": 0,
        "query-memory-bytes": 0,
        "query-queue-size": 1024,
        "reporting-disabled": true,
        "secret-store": "bolt",
        "session-length": 60,
        "session-renew-disabled": false,
        "sqlite-path": "/influx/metadata/influxd.sqlite",
        "storage-cache-max-memory-size": 1073741824,
        "storage-cache-snapshot-memory-size": 26214400,
        "storage-cache-snapshot-write-cold-duration": "10m0s",
        "storage-compact-full-write-cold-duration": "4h0m0s",
        "storage-compact-throughput-burst": 50331648,
        "storage-max-concurrent-compactions": 0,
        "storage-max-index-log-file-size": 1048576,
        "storage-no-validate-field-size": false,
        "storage-retention-check-interval": "30m0s",
        "storage-series-file-max-concurrent-snapshot-compactions": 0,
        "storage-series-id-set-cache-size": 0,
        "storage-shard-precreator-advance-period": "30m0s",
        "storage-shard-precreator-check-interval": "10m0s",
        "storage-tsm-use-madv-willneed": false,
        "storage-validate-keys": false,
        "storage-wal-fsync-delay": "0s",
        "storage-wal-max-concurrent-writes": 0,
        "storage-wal-max-write-delay": 600000000000,
        "storage-write-timeout": 10000000000,
        "store": "disk",
        "testing-always-allow-setup": false,
        "tls-cert": "/opt/amazon/certs/influx/cert.pem",
        "tls-key": "/opt/amazon/certs/influx/key.pem",
        "tls-min-version": "1.2",
        "tls-strict-ciphers": true,
        "tracing-type": "",
        "ui-disabled": false,
        "vault-addr": "",
        "vault-cacert": "",
        "vault-capath": "",
        "vault-client-cert": "",
        "vault-client-key": "",
        "vault-client-timeout": 0,
        "vault-max-retries": 0,
        "vault-skip-verify": false,
        "vault-tls-server-name": "",
        "vault-token": ""
}
```

There are now `lvl=warn msg="internal error not returned to client" log_id=0rBRhDdl000 handler=error_logger error="context canceled"` errors in the log following compaction and snapshot log entries.

Balwanth · Answer

I can see from the message that your update-db-instance command wasn't successful in the first place (An error occurred (ValidationException) when calling the UpdateDbInstance operation: DbInstance with identifier [REDACTED] and customer id [REDACTED] is not in an updatable state). Can you open a support ticket with Instance ARN to check further on what's going on? Thank you!

Balwanth

answered 2 years ago

re:Post Agent · Answer

I understand you're facing a critical issue with your Timestream for InfluxDB instance. Based on the information provided, it appears that changing the query concurrency and query queue size to 0 has caused the database to enter a FAILED state and perform poorly. Here are some recommendations to help recover your database:

1. Revert parameter group changes: 
Try to revert the changes in your parameter group. Set the query-concurrency and query-queue-size back to their default values (1024 for both). However, as your database is in a FAILED state, you may not be able to apply these changes directly.

2. Create a new parameter group:
If you can't modify the existing parameter group, create a new one with default settings. Then attempt to associate this new parameter group with your database instance.

3. Restart the database:
If possible, try to restart your database instance. This might help apply the default or new parameter group settings.

4. Contact AWS Support:
Given the severity of the issue and the FAILED status, it's highly recommended to contact AWS Support. They have access to more detailed information about your instance and can provide specific guidance or perform actions to recover your database.

5. Consider point-in-time recovery:
If you have point-in-time recovery enabled, you might be able to restore your database to a state before the problematic changes were applied.

6. Monitor logs:
Continue monitoring your database logs for any additional error messages or indications of what might be causing the FAILED state.

7. Avoid setting concurrency to 0:
In the future, avoid setting query-concurrency or query-queue-size to 0. These settings can severely impact database performance and stability. The query-concurrency of 0 allows unlimited concurrent queries, which can overwhelm system resources.

Remember, when making changes to database parameters, it's crucial to test in a non-production environment first and make incremental changes rather than dramatic ones.

If none of these steps resolve the issue, you may need to consider creating a new database instance and migrating your data. However, this should be a last resort and only done with guidance from AWS Support to ensure no data loss occurs.  
**Sources**  
[Connecting to an Amazon Timestream for InfluxDB DB instance - Amazon Timestream](https://docs.aws.amazon.com/timestream/latest/developerguide/timestream-for-influx-db-connecting.html)  
[Configuring a DB instance - Amazon Timestream](https://docs.aws.amazon.com/timestream/latest/developerguide/timestream-for-influx-configuring.html)

Timestream InfluxDB Service Recovery

Add your answer

Relevant content