By using AWS re:Post, you agree to the Terms of Use

Unanswered Questions tagged with Monitoring

Sort by most recent
  • 1
  • 12 / page

Browse through the questions and answers listed below or filter and sort to narrow down your results.

Not able to get the data in query result in the Athena for the AWS config from S3 bucket

Hi, I have been trying to implement a monitoring solution to monitor the resources for AWS accounts in the organizations with AWS config, AWS Athena and Quicksight. I have set up all the services however Athena is not able to query all the data from the S3 bucket where the config data for all the accounts are stored. It is able to only query the data for the current account from where I am running the query. I can see the config data for all accounts in the S3 bucket as well. **Athena table creation query** ``` CREATE EXTERNAL TABLE aws_config_configuration_snapshot ( fileversion STRING, configSnapshotId STRING, configurationitems ARRAY < STRUCT < configurationItemVersion : STRING, configurationItemCaptureTime : STRING, configurationStateId : BIGINT, awsAccountId : STRING, configurationItemStatus : STRING, resourceType : STRING, resourceId : STRING, resourceName : STRING, ARN : STRING, awsRegion : STRING, availabilityZone : STRING, configurationStateMd5Hash : STRING, configuration : STRING, supplementaryConfiguration : MAP < STRING, STRING >, tags: MAP < STRING, STRING >, resourceCreationTime : STRING > > ) PARTITIONED BY (accountid STRING, dt STRING, region STRING) ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe' LOCATION 's3://<S3_BUCKET_NAME>/AWSLogs/'; ``` **The lambda function used for data partitioning as per accounts** ``` import datetime import re import boto3 import os TABLE_NAME = 'aws_config_configuration_snapshot' DATABASE_NAME = 'sampledb' ACCOUNT_ID = None # Determined at runtime LATEST_PARTITION_VALUE = 'latest' athena = boto3.client('athena') def lambda_handler(event, context): global ACCOUNT_ID object_key = event['Records'][0]['s3']['object']['key'] match = get_configuration_snapshot_object_key_match(object_key) if match is None: print('Ignoring event for non-configuration snapshot object key', object_key) return print('Adding partitions for configuration snapshot object key', object_key) ACCOUNT_ID = context.invoked_function_arn.split(':')[4] object_key_parent = 's3://{bucket_name}/{object_key_parent}/'.format( bucket_name=event['Records'][0]['s3']['bucket']['name'], object_key_parent=os.path.dirname(object_key)) configuration_snapshot_accountid = get_configuration_snapshot_accountid(match) configuration_snapshot_region = get_configuration_snapshot_region(match) configuration_snapshot_date = get_configuration_snapshot_date(match) drop_partition(configuration_snapshot_accountid, configuration_snapshot_region, LATEST_PARTITION_VALUE) add_partition(configuration_snapshot_accountid, configuration_snapshot_region, LATEST_PARTITION_VALUE, object_key_parent) add_partition(configuration_snapshot_accountid, configuration_snapshot_region, get_configuration_snapshot_date(match).strftime('%Y-%m-%d'), object_key_parent) def get_configuration_snapshot_object_key_match(object_key): # Matches object keys like AWSLogs/123456789012/Config/us-east-1/2018/4/11/ConfigSnapshot/123456789012_Config_us-east-1_ConfigSnapshot_20180411T054711Z_a970aeff-cb3d-4c4e-806b-88fa14702hdb.json.gz return re.match('^AWSLogs/(\d+)/Config/([\w-]+)/(\d+)/(\d+)/(\d+)/ConfigSnapshot/[^\\\]+$', object_key) def get_configuration_snapshot_accountid(match): print('AccountId:', match.group(1)) return match.group(1) def get_configuration_snapshot_region(match): return match.group(2) def get_configuration_snapshot_date(match): return datetime.date(int(match.group(3)), int(match.group(4)), int(match.group(5))) def add_partition(accountid_partition_value, region_partition_value, dt_partition_value, partition_location): execute_query('ALTER TABLE {table_name} ADD PARTITION {partition} location \'{partition_location}\''.format( table_name=TABLE_NAME, partition=build_partition_string(accountid_partition_value, region_partition_value, dt_partition_value), partition_location=partition_location)) def drop_partition(accountid_partition_value, region_partition_value, dt_partition_value): execute_query('ALTER TABLE {table_name} DROP PARTITION {partition}'.format( table_name=TABLE_NAME, partition=build_partition_string(accountid_partition_value, region_partition_value, dt_partition_value))) def build_partition_string(accountid_partition_value, region_partition_value, dt_partition_value): return "(accountid='{accountid_partition_value}', dt='{dt_partition_value}', region='{region_partition_value}')".format( accountid_partition_value=accountid_partition_value, dt_partition_value=dt_partition_value, region_partition_value=region_partition_value) def execute_query(query): print('Executing query:', query) query_output_location = 's3://aws-athena-query-results-{account_id}-{region}'.format( account_id=ACCOUNT_ID, region=os.environ['AWS_REGION']) start_query_response = athena.start_query_execution( QueryString=query, QueryExecutionContext={ 'Database': DATABASE_NAME }, ResultConfiguration={ 'OutputLocation': query_output_location, } ) print('Query started') is_query_running = True while is_query_running: get_query_execution_response = athena.get_query_execution( QueryExecutionId=start_query_response['QueryExecutionId'] ) query_state = get_query_execution_response['QueryExecution']['Status']['State'] is_query_running = query_state in ('RUNNING','QUEUED') if not is_query_running and query_state != 'SUCCEEDED': raise Exception('Query failed') print('Query completed') ``` **sample query tried:** ``` CREATE OR REPLACE VIEW v_config_ec2_vpcs AS SELECT DISTINCT "accountId" "633328536665" , "region" "us-east-1" , "configurationItem"."resourceid" "ResourceId" , "configurationItem"."tags"['name'] "TagName" , "json_extract_scalar"("configurationItem"."configuration", '$.isdefault') "IsDefault" , "json_extract_scalar"("configurationItem"."configuration", '$.cidrblockassociationset[0].cidrblock') "CidrBlock0" , "json_extract_scalar"("configurationItem"."configuration", '$.cidrblockassociationset[1].cidrblock') "CidrBlock1" , "json_extract_scalar"("configurationItem"."configuration", '$.cidrblockassociationset[2].cidrblock') "CidrBlock2" , "json_extract_scalar"("configurationItem"."configuration", '$.cidrblockassociationset[3].cidrblock') "CidrBlock3" , "json_extract_scalar"("configurationItem"."configuration", '$.cidrblockassociationset[4].cidrblock') "CidrBlock4" FROM default.aws_config_configuration_snapshot CROSS JOIN UNNEST("configurationitems") t (configurationItem) WHERE (("dt" = 'latest') AND ("configurationItem"."resourcetype" = 'AWS::EC2::VPC')) ``` It is not able to get the data from the S3 bucket for all the AWS account for some reason(only the data for the current account the data is queried.). I have checked the s3 bucket policy and it is set up as per the given below solution. Solution referred: * https://aws.amazon.com/blogs/mt/visualizing-aws-config-data-using-amazon-athena-and-amazon-quicksight/ * https://aws.amazon.com/blogs/mt/how-to-query-your-aws-resource-configuration-states-using-aws-config-and-amazon-athena/ Thanks and Regards, Mahesh B.
0
answers
0
votes
95
views
asked 3 months ago

Unable to create new OpsItems from EventBridge when using Input Transformer for deduplication and adding category and severity values

I’m able to generate a new OpsItem for any EC2, SecurityGroup, or VPC configuration change using an EventBridge rule with the following event pattern. { "source": ["aws.config"], "detail-type": ["Config Configuration Item Change"], "detail": { "messageType": ["ConfigurationItemChangeNotification"], "configurationItem": { "resourceType": ["AWS::EC2::Instance", "AWS::EC2::SecurityGroup", "AWS::EC2::VPC"] } } } The rule and target work great when using Matched event for the Input but I noticed that launching one EC2 using the AWS wizard creates at least three OpsItems, one for each resourceType. Therefore I’d like to implement a deduplication string to cut down on the number of OpsItems generated to one if possible and I’d also like to attach a category and severity to the new OpsItem. I’m trying to use an Input Transformer as recommended by the AWS documentation but even the most simplest of Input Transformers when applied prevent any new OpsItems from being generated. When I've tested, I've also ensured that all previous OpsItems were resolved. Can anyone tell me what might be blocking the creation of any new OpsItems when using this Input Transformer configuration? Here’s what I have configured now. Input path { "awsAccountId": "$.detail.configurationItem.awsAccountId", "awsRegion": "$.detail.configurationItem.awsRegion", "configurationItemCaptureTime": "$.detail.configurationItem.configurationItemCaptureTime", "detail-type": "$.detail-type", "messageType": "$.detail.messageType", "notificationCreationTime": "$.detail.notificationCreationTime", "region": "$.region", "resourceId": "$.detail.configurationItem.resourceId", "resourceType": "$.detail.configurationItem.resourceType", "resources": "$.resources", "source": "$.source", "time": "$.time" } Input template { "awsAccountId": "<awsAccountId>", "awsRegion": "<awsRegion>", "configurationItemCaptureTime": "<configurationItemCaptureTime>", "resourceId": "<resourceId>", "resourceType": "<resourceType>", "title": "Template under ConfigDrift-EC2-Dedup4", "description": "Configuration Drift Detected.", "category": "Security", "severity": "3", "origination": "EventBridge Rule - ConfigDrift-EC2-Dedup", "detail-type": "<detail-type>", "source": "<source>", "time": "<time>", "region": "<region>", "resources": "<resources>", "messageType": "<messageType>", "notificationCreationTime": "<notificationCreationTime>", "operationalData": { "/aws/dedup": { "type": "SearchableString", "value": "{\"dedupString\":\"ConfigurationItemChangeNotification\"}" } } } Output when using the AWS supplied Sample event called “Config Configuration Item Change” { "awsAccountId": "123456789012", "awsRegion": "us-east-1", "configurationItemCaptureTime": "2022-03-16T01:10:50.837Z", "resourceId": "fs-01f0d526165b57f95", "resourceType": "AWS::EFS::FileSystem", "title": "Template under ConfigDrift-EC2-Dedup4", "description": "Configuration Drift Detected.", "category": "Security", "severity": "3", "origination": "EventBridge Rule - ConfigDrift-EC2-Dedup", "detail-type": "Config Configuration Item Change", "source": "aws.config", "time": "2022-03-16T01:10:51Z", "region": "us-east-1", "resources": "arn:aws:elasticfilesystem:us-east-1:123456789012:file-system/fs-01f0d526165b57f95", "messageType": "ConfigurationItemChangeNotification", "notificationCreationTime": "2022-03-16T01:10:51.976Z", "operationalData": { "/aws/dedup": { "type": "SearchableString", "value": "{"dedupString":"ConfigurationItemChangeNotification"}" } } }
0
answers
0
votes
21
views
asked 5 months ago

Proper conversion of AWS Log Insights to Metrics for visualization and monitoring

TL;DR; ---- What is the proper way to create a metric so that it generates reliable information about the log insights? What is desired ------ The current Log insights can be seen similar to the following [![AWS Log insights][1]][1] However, it becomes easier to analyse these logs using the metrics (mostly because you can have multiple sources of data in the same plot and even perform math operations between them). Solution according to docs ----- Allegedly, a log can be converted to a metric filter following a guide like [this][2]. However, this approach does not seem to work entirely right (I guess because of the time frames that have to be imposed in the metric plots), providing incorrect information, for example: [![Dashboard][3]][3] Issue with solution ----- In the previous image I've created a dashboard containing the metric count (the number 7), corresponding to the sum of events each 5 minutes. Also I've added a preview of the log insight corresponding to the information used to create the event. However, as it can be seen, the number of logs is 4, but the event count displays 7. Changing the time frame in the metric generates other types of issues (e.g., selecting a very small time frame like 1 sec won't retrieve any data, or a slightly smaller time frame will now provide another wrong number: 3, when there are 4 logs, for example). P.S. ----- I've also tried converting the log insights to metrics using [this lambda function][4] as suggested by [Danil Smirnov][5] to no avail, as it seems to generate the same issues. [1]: https://i.stack.imgur.com/0pPdp.png [2]: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CountingLogEventsExample.html [3]: https://i.stack.imgur.com/Dy5td.png [4]: https://serverlessrepo.aws.amazon.com/#!/applications/arn:aws:serverlessrepo:us-east-1:085576722239:applications~logs-insights-to-metric [5]: https://blog.smirnov.la/cloudwatch-logs-insights-to-metrics-a2d197aac379
0
answers
0
votes
35
views
asked 7 months ago

SageMaker - All metrics in statistics.json by Model Quality Monitor are "0.0 +/- 0.0", but confusion matrix is built correctly for multi-class classification!!

I have scheduled an hourly model-quality-monitoring job in AWS SageMaker. both the jobs, ground-truth-merge and model-quality-monitoring completes successfully without any errors. but, all the metrics calculated by the job are "0.0 +/- 0.0" while the confustion matrix gets calculated as expected. I have done everything as mentioned in [this notebook for model-quality-monitoring from sagemaker-examples](https://github.com/aws/amazon-sagemaker-examples/blob/main/sagemaker_model_monitor/model_quality/model_quality_churn_sdk.ipynb) with very few changes and they are: 1. I have changed the model from xgboost churn to model trained on my data. 2. my input to the endpoint was csv like in the example-notebook, but output was json. 3. i have changed the problem-type from BinaryClassfication to MulticlassClassification wherever necessary. confustion matrix was built successfully, but all metrics are 0 for some reason. So, I would like the monitoring job to calculate the multi-classification metrics on data properly. **All Logs** Here's the `statistics.json` file that model-quality-monitor saved to S3 with confustion matrix built, but with 0s in all the metrics: ``` { "version" : 0.0, "dataset" : { "item_count" : 4432, "start_time" : "2022-02-23T03:00:00Z", "end_time" : "2022-02-23T04:00:00Z", "evaluation_time" : "2022-02-23T04:13:20.193Z" }, "multiclass_classification_metrics" : { "confusion_matrix" : { "0" : { "0" : 709, "2" : 530, "1" : 247 }, "2" : { "0" : 718, "2" : 497, "1" : 265 }, "1" : { "0" : 700, "2" : 509, "1" : 257 } }, "accuracy" : { "value" : 0.0, "standard_deviation" : 0.0 }, "weighted_recall" : { "value" : 0.0, "standard_deviation" : 0.0 }, "weighted_precision" : { "value" : 0.0, "standard_deviation" : 0.0 }, "weighted_f0_5" : { "value" : 0.0, "standard_deviation" : 0.0 }, "weighted_f1" : { "value" : 0.0, "standard_deviation" : 0.0 }, "weighted_f2" : { "value" : 0.0, "standard_deviation" : 0.0 }, "accuracy_best_constant_classifier" : { "value" : 0.3352888086642599, "standard_deviation" : 0.003252410977346705 }, "weighted_recall_best_constant_classifier" : { "value" : 0.3352888086642599, "standard_deviation" : 0.003252410977346705 }, "weighted_precision_best_constant_classifier" : { "value" : 0.1124185852154987, "standard_deviation" : 0.0021869336610830254 }, "weighted_f0_5_best_constant_classifier" : { "value" : 0.12965524348784485, "standard_deviation" : 0.0024239410000317335 }, "weighted_f1_best_constant_classifier" : { "value" : 0.16838092925822584, "standard_deviation" : 0.0028615098045768348 }, "weighted_f2_best_constant_classifier" : { "value" : 0.24009212108475822, "standard_deviation" : 0.003326031863819311 } } } ``` Here's how couple of lines of captured data looks like(*prettified for readability, but each line has no tab spaces as shown below*) : ``` { "captureData": { "endpointInput": { "observedContentType": "text/csv", "mode": "INPUT", "data": "0,1,628,210,30", "encoding": "CSV" }, "endpointOutput": { "observedContentType": "application/json", "mode": "OUTPUT", "data": "{\"label\":\"Transfer\",\"prediction\":2,\"probabilities\":[0.228256680901919,0.0,0.7717433190980809]}\n", "encoding": "JSON" } }, "eventMetadata": { "eventId": "a7cfba60-39ee-4796-bd85-343dcadef024", "inferenceId": "5875", "inferenceTime": "2022-02-23T04:12:51Z" }, "eventVersion": "0" } { "captureData": { "endpointInput": { "observedContentType": "text/csv", "mode": "INPUT", "data": "0,3,628,286,240", "encoding": "CSV" }, "endpointOutput": { "observedContentType": "application/json", "mode": "OUTPUT", "data": "{\"label\":\"Adoption\",\"prediction\":0,\"probabilities\":[0.99,0.005,0.005]}\n", "encoding": "JSON" } }, "eventMetadata": { "eventId": "7391ac1e-6d27-4f84-a9ad-9fbd6130498a", "inferenceId": "5876", "inferenceTime": "2022-02-23T04:12:51Z" }, "eventVersion": "0" } ``` Here's couple of lines from my ground-truths that I have uploaded to S3 look like(*prettified for readability, but each line has no tab spaces as shown below*): ``` { "groundTruthData": { "data": "0", "encoding": "CSV" }, "eventMetadata": { "eventId": "1" }, "eventVersion": "0" } { "groundTruthData": { "data": "1", "encoding": "CSV" }, "eventMetadata": { "eventId": "2" }, "eventVersion": "0" }, ``` Here's couple of lines from the ground-truth-merged file look like(*prettified for readability, but each line has no tab spaces as shown below*). this file is created by the ground-truth-merge job, which is one of the two jobs that model-quality-monitoring schedule runs: ``` { "eventVersion": "0", "groundTruthData": { "data": "2", "encoding": "CSV" }, "captureData": { "endpointInput": { "data": "1,2,1050,37,1095", "encoding": "CSV", "mode": "INPUT", "observedContentType": "text/csv" }, "endpointOutput": { "data": "{\"label\":\"Return_to_owner\",\"prediction\":1,\"probabilities\":[0.14512373737373732,0.6597074314574313,0.1951688311688311]}\n", "encoding": "JSON", "mode": "OUTPUT", "observedContentType": "application/json" } }, "eventMetadata": { "eventId": "c9e21f63-05f0-4dec-8f95-b8a1fa3483c1", "inferenceId": "4432", "inferenceTime": "2022-02-23T04:00:00Z" } } { "eventVersion": "0", "groundTruthData": { "data": "1", "encoding": "CSV" }, "captureData": { "endpointInput": { "data": "0,2,628,5,90", "encoding": "CSV", "mode": "INPUT", "observedContentType": "text/csv" }, "endpointOutput": { "data": "{\"label\":\"Adoption\",\"prediction\":0,\"probabilities\":[0.7029623691085284,0.0,0.29703763089147156]}\n", "encoding": "JSON", "mode": "OUTPUT", "observedContentType": "application/json" } }, "eventMetadata": { "eventId": "5f1afc30-2ffd-42cf-8f4b-df97f1c86cb1", "inferenceId": "4433", "inferenceTime": "2022-02-23T04:00:01Z" } } ``` Since, the confusion matrix was constructed properly, I presume that I fed the data to sagemaker-model-monitor the right-way. But, why are all the metrics 0.0, while confustion-matrix looks as expected? EDIT 1: Logs for the job are available [here](https://controlc.com/1e1781d2).
0
answers
1
votes
17
views
asked 7 months ago
  • 1
  • 12 / page