Query CloudWatch Logs with result over 7GB

When dealing with large amounts of log data in CloudWatch Logs, especially for queries that result in outputs as large as 7GB, you'll need to use a combination of strategies to retrieve all the data efficiently. Here's how you can approach this using the GetQueryResultsCommand:

1. First, use the StartQuery operation to initiate your query for the 7-day period you're interested in. This will return a queryId.

2. Then, use the GetQueryResultsCommand with the queryId you received. However, keep in mind that CloudWatch Logs Insights has some limitations:

• Queries time out after 60 minutes of runtime.
• You can have up to 30 concurrent CloudWatch Logs Insights queries.
• Query results are available for 7 days.

3. The GetQueryResultsCommand doesn't provide a nextToken like GetLogEventsCommand does. Instead, it returns partial results if the query is still running. You'll need to repeatedly call GetQueryResultsCommand with the same queryId until the query status is "Complete" or "Failed".

4. In each response, check the 'status' field. If it's "Running", wait for a short interval (e.g., a few seconds) and then call GetQueryResultsCommand again.

5. Each response will include a 'results' array. You'll need to accumulate these results across multiple API calls.

6. Be aware that there's a limit to the number of results returned in a single call. You may need to adjust your query to return fewer results per call, possibly by breaking your 7-day period into smaller time chunks.

7. If your query is timing out, consider breaking it into multiple smaller queries over shorter time periods.

Remember, queries added to dashboards run every time the dashboard is loaded or refreshed and count toward your limit of 30 concurrent CloudWatch Logs Insights queries. So, for large data retrieval tasks like this, it's better to run ad-hoc queries rather than adding them to dashboards.

Also, note that CloudWatch Logs Insights charges based on the amount of data scanned by your queries. Retrieving 7GB of data will incur significant costs, so optimize your queries as much as possible to scan only the necessary data.

Given the large amount of data you're dealing with, you might want to consider alternative approaches, such as exporting log data to S3 for analysis with other tools that are better suited for handling large datasets.
Sources
StartQuery - Amazon CloudWatch Logs
Add query to dashboard or export query results - Amazon CloudWatch Logs
Analyzing log data with CloudWatch Logs Insights - Amazon CloudWatch Logs
GetQueryResults - Amazon CloudWatch Logs