How to pull data from AWS Security Hub using Scheduler?

Hello,

Yes, It is a JSON based but it's AWS own format named AWS Security Finding Format (ASFF) It is true for all resources that SecurityHub supports and is able to see. It should be noted that Each Security Hub Findings - Imported event contains a single finding. In order to see those events you'll need to create an EventBridge rule based on the format for each type of event.

Once you have that set up, the event could trigger an automatic action like:

• Invoking an AWS Lambda function
• Invoking the Amazon EC2 run command
• Relaying the event to Amazon Kinesis Data Streams
• Activating an AWS Step Functions state machine
• Notifying an Amazon SNS topic or an Amazon SQS queue
• Sending a finding to a third-party ticketing, chat, SIEM, or incident response and management tool.

In general, EventBridge is the way forward, but rather than using a scheduled based approach you'll need to resort to an event-based one. In order to intercept all findings, instead of rule being triggered by just specific one, you'll need to adjust the filter and essentially create a catch-all rule for SecurityHub which will then trigger your ETL job. The filter in the rule would look like this:

{ "source": [ "aws.securityhub" ] }

with regard to the ETL, it really depends on your use case, having Kinesis Data Firehose dumping it to S3 and then using Athena as you suggest on your own would work. Another common approach is to send the data to ElasticSearch (or now OpenSearch). You can take the reference of this blog post described them both, you can adjust it based on your needs.

Blog post URL - https://aws.amazon.com/blogs/architecture/visualize-aws-security-hub-findings-using-analytics-and-business-intelligence-tools/