By using AWS re:Post, you agree to the Terms of Use
/Monitoring/

Questions tagged with Monitoring

Sort by most recent
  • 1
  • 90 / page

Browse through the questions and answers listed below or filter and sort to narrow down your results.

Unable to create new OpsItems from EventBridge when using Input Transformer for deduplication and adding category and severity values

Apologize to all for the duplicate post. I created my login under the wrong account when I initially posted this question. I’m able to generate a new OpsItem for any EC2, SecurityGroup, or VPC configuration change using an EventBridge rule with the following event pattern. { "source": "aws.config", "detail-type": "Config Configuration Item Change", "detail": { "messageType": "ConfigurationItemChangeNotification", "configurationItem": { "resourceType": "AWS::EC2::Instance", "AWS::EC2::SecurityGroup", "AWS::EC2::VPC" } } } The rule and target work great when using Matched event for the Input but I noticed that launching one EC2 using the AWS wizard creates at least three OpsItems, one for each resourceType. Therefore I’d like to implement a deduplication string to cut down on the number of OpsItems generated to one if possible and I’d also like to attach a category and severity to the new OpsItem. I’m trying to use an Input Transformer as recommended by the AWS documentation but even the most simplest of Input Transformers when applied prevent any new OpsItems from being generated. When I've tested, I've also ensured that all previous OpsItems were resolved. Can anyone tell me what might be blocking the creation of any new OpsItems when using this Input Transformer configuration? Here’s what I have configured now. Input path { "awsAccountId": "$.detail.configurationItem.awsAccountId", "awsRegion": "$.detail.configurationItem.awsRegion", "configurationItemCaptureTime": "$.detail.configurationItem.configurationItemCaptureTime", "detail-type": "$.detail-type", "messageType": "$.detail.messageType", "notificationCreationTime": "$.detail.notificationCreationTime", "region": "$.region", "resourceId": "$.detail.configurationItem.resourceId", "resourceType": "$.detail.configurationItem.resourceType", "resources": "$.resources", "source": "$.source", "time": "$.time" } Input template { "awsAccountId": "<awsAccountId>", "awsRegion": "<awsRegion>", "configurationItemCaptureTime": "<configurationItemCaptureTime>", "resourceId": "<resourceId>", "resourceType": "<resourceType>", "title": "Template under ConfigDrift-EC2-Dedup4", "description": "Configuration Drift Detected.", "category": "Security", "severity": "3", "origination": "EventBridge Rule - ConfigDrift-EC2-Dedup", "detail-type": "<detail-type>", "source": "<source>", "time": "<time>", "region": "<region>", "resources": "<resources>", "messageType": "<messageType>", "notificationCreationTime": "<notificationCreationTime>", "operationalData": { "/aws/dedup": { "type": "SearchableString", "value": "{\"dedupString\":\"ConfigurationItemChangeNotification\"}" } } } Output when using the AWS supplied Sample event called “Config Configuration Item Change” { "awsAccountId": "123456789012", "awsRegion": "us-east-1", "configurationItemCaptureTime": "2022-03-16T01:10:50.837Z", "resourceId": "fs-01f0d526165b57f95", "resourceType": "AWS::EFS::FileSystem", "title": "Template under ConfigDrift-EC2-Dedup4", "description": "Configuration Drift Detected.", "category": "Security", "severity": "3", "origination": "EventBridge Rule - ConfigDrift-EC2-Dedup", "detail-type": "Config Configuration Item Change", "source": "aws.config", "time": "2022-03-16T01:10:51Z", "region": "us-east-1", "resources": "arn:aws:elasticfilesystem:us-east-1:123456789012:file-system/fs-01f0d526165b57f95", "messageType": "ConfigurationItemChangeNotification", "notificationCreationTime": "2022-03-16T01:10:51.976Z", "operationalData": { "/aws/dedup": { "type": "SearchableString", "value": "{"dedupString":"ConfigurationItemChangeNotification"}" } } }
1
answers
0
votes
3
views
asked 12 days ago

Unable to create new OpsItems from EventBridge when using Input Transformer for deduplication and adding category and severity values

I’m able to generate a new OpsItem for any EC2, SecurityGroup, or VPC configuration change using an EventBridge rule with the following event pattern. { "source": ["aws.config"], "detail-type": ["Config Configuration Item Change"], "detail": { "messageType": ["ConfigurationItemChangeNotification"], "configurationItem": { "resourceType": ["AWS::EC2::Instance", "AWS::EC2::SecurityGroup", "AWS::EC2::VPC"] } } } The rule and target work great when using Matched event for the Input but I noticed that launching one EC2 using the AWS wizard creates at least three OpsItems, one for each resourceType. Therefore I’d like to implement a deduplication string to cut down on the number of OpsItems generated to one if possible and I’d also like to attach a category and severity to the new OpsItem. I’m trying to use an Input Transformer as recommended by the AWS documentation but even the most simplest of Input Transformers when applied prevent any new OpsItems from being generated. When I've tested, I've also ensured that all previous OpsItems were resolved. Can anyone tell me what might be blocking the creation of any new OpsItems when using this Input Transformer configuration? Here’s what I have configured now. Input path { "awsAccountId": "$.detail.configurationItem.awsAccountId", "awsRegion": "$.detail.configurationItem.awsRegion", "configurationItemCaptureTime": "$.detail.configurationItem.configurationItemCaptureTime", "detail-type": "$.detail-type", "messageType": "$.detail.messageType", "notificationCreationTime": "$.detail.notificationCreationTime", "region": "$.region", "resourceId": "$.detail.configurationItem.resourceId", "resourceType": "$.detail.configurationItem.resourceType", "resources": "$.resources", "source": "$.source", "time": "$.time" } Input template { "awsAccountId": "<awsAccountId>", "awsRegion": "<awsRegion>", "configurationItemCaptureTime": "<configurationItemCaptureTime>", "resourceId": "<resourceId>", "resourceType": "<resourceType>", "title": "Template under ConfigDrift-EC2-Dedup4", "description": "Configuration Drift Detected.", "category": "Security", "severity": "3", "origination": "EventBridge Rule - ConfigDrift-EC2-Dedup", "detail-type": "<detail-type>", "source": "<source>", "time": "<time>", "region": "<region>", "resources": "<resources>", "messageType": "<messageType>", "notificationCreationTime": "<notificationCreationTime>", "operationalData": { "/aws/dedup": { "type": "SearchableString", "value": "{\"dedupString\":\"ConfigurationItemChangeNotification\"}" } } } Output when using the AWS supplied Sample event called “Config Configuration Item Change” { "awsAccountId": "123456789012", "awsRegion": "us-east-1", "configurationItemCaptureTime": "2022-03-16T01:10:50.837Z", "resourceId": "fs-01f0d526165b57f95", "resourceType": "AWS::EFS::FileSystem", "title": "Template under ConfigDrift-EC2-Dedup4", "description": "Configuration Drift Detected.", "category": "Security", "severity": "3", "origination": "EventBridge Rule - ConfigDrift-EC2-Dedup", "detail-type": "Config Configuration Item Change", "source": "aws.config", "time": "2022-03-16T01:10:51Z", "region": "us-east-1", "resources": "arn:aws:elasticfilesystem:us-east-1:123456789012:file-system/fs-01f0d526165b57f95", "messageType": "ConfigurationItemChangeNotification", "notificationCreationTime": "2022-03-16T01:10:51.976Z", "operationalData": { "/aws/dedup": { "type": "SearchableString", "value": "{"dedupString":"ConfigurationItemChangeNotification"}" } } }
0
answers
0
votes
1
views
asked 12 days ago

AWS SDK SQS get number of messages in a dead letter queue

Hello community, I somehow can't find the right information. I have following simple task to solve: create a lambda that checks if a dead letter queue has messages and if it has, read how many. Before I did that I had an alarm set on an SQS metric. I chose the 'ApproximateNumberOfMessagesVisible' metric since 'NumberOfMessagesSent' (which was my first choice) does not work for DLQueues. I have read this article: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html. >The NumberOfMessagesSent and NumberOfMessagesReceived for a dead-letter queue don't match > If you send a message to a dead-letter queue manually, it is captured by the NumberOfMessagesSent metric. However, if a message is sent to a dead-letter queue as a result of a failed processing attempt, it isn't captured by this metric. Thus, it is possible for the values of **NumberOfMessagesSent** and NumberOfMessagesReceived to be different. That is nice to know, but I was missing the information: which metric shall I use if **NumberOfMessagesSent** won't work? I was being pragmatic here so I created an error, a message was sent to the DLQ as a result of a failed processing attempt. Now I looked at the queue in the AWS console under the monitoring-tab and I checked which metric spiked. It was **ApproximateNumberOfMessagesVisible**, which sounded suitable, so I used it. Now I wanted to get alerted more often so I chose to build a lambda function that checks how many messages are in the DLQueue. I use Javascript / Typescript so I found this: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_GetQueueAttributes.html. Code looked something like this: ``` const params = { QueueUrl: url, AttributeNames: ['ApproximateNumberOfMessagesVisible'] } const resp = SQS.getQueueAttributes(params).promise() ``` It was kind of a bummer that the attribute I wanted was not in there, or better: it was not valid. > Valid Values: All | Policy | VisibilityTimeout | MaximumMessageSize | MessageRetentionPeriod | ApproximateNumberOfMessages | ApproximateNumberOfMessagesNotVisible | CreatedTimestamp | LastModifiedTimestamp | QueueArn | ApproximateNumberOfMessagesDelayed | DelaySeconds | ReceiveMessageWaitTimeSeconds | RedrivePolicy | FifoQueue | ContentBasedDeduplication | ... My first attempt was to use CloudWatch metrics. So I tried this: https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/cloudwatch-examples-getting-metrics.html ``` var params = { Dimensions: [ { Name: 'LogGroupName', /* required */ }, ], MetricName: 'IncomingLogEvents', Namespace: 'AWS/Logs' }; cw.listMetrics(params, function(err, data) { if (err) { console.log("Error", err); } else { console.log("Metrics", JSON.stringify(data.Metrics)); } }); ``` but I could not get this working since I did not know what to add to Dimensions / Name to make this working. Please note that I am not working very long with AWS (only 6 months). Maybe I am on a total wrong track. Summarized: I want to achieve that my lambda gets the number of messages in a DLQ. I hope someone can help me Cheers Aleks
1
answers
0
votes
5
views
asked a month ago

Elastic beanstalk Enhanced health not generating healthd/application.log files

I have Enhanced health reporting turned on for my Elastic beanstalk environment. The environment is: 1. Multicontainer docker setup running in “Amazon Linux 2” 2. It has an nginx proxy (Configuration > Software shows: Log streaming: disabled / Proxy server: nginx / Rotate logs: disabled / X-Ray daemon: disabled) 3. Enhanced monitoring is on (Configuration > Monitoring shows: CloudWatch Custom Metrics-Environment: CloudWatch Custom Metrics-Instance: / Health event log streaming: disabled / Ignore HTTP 4xx: enabled / Ignore load balancer 4xx: disabled System: Enhanced) However, on the Health page, none of the requests, response, or latency fields are populating, while load & CPU utilization are populating. It is my understanding that this data is populated from a log file that is written to `/var/log/nginx/healthd/`, but that directory is empty. It seems like this is a bug or some sort of misconfiguration. Does anyone know why this might be happening? I included some relevant info from the machine below. --- The healthd config file (I commented out the `group_id`, which is a uuid in the actual file): ``` $ cat /etc/healthd/config.yaml group_id: XXXX log_to_file: true endpoint: https://elasticbeanstalk-health.us-east-2.amazonaws.com appstat_log_path: /var/log/nginx/healthd/application.log appstat_unit: sec appstat_timestamp_on: completion ``` The output of the healthd daemon log—showing warnings for not finding previous application.log.YYYY-MM-DD-HH files: ``` $ head /var/log/healthd/daemon.log # Logfile created on 2022-04-02 21:02:22 +0000 by logger.rb/66358 A, [2022-04-02T21:02:24.123304 #4122] ANY -- : healthd daemon 1.0.6 initialized W, [2022-04-02T21:02:24.266469 #4122] WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist W, [2022-04-02T21:02:29.266806 #4122] WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist W, [2022-04-02T21:02:34.404332 #4122] WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist W, [2022-04-02T21:02:39.406846 #4122] WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist W, [2022-04-02T21:02:44.410108 #4122] WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist W, [2022-04-02T21:02:49.410342 #4122] WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist W, [2022-04-02T21:02:54.410611 #4122] WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist W, [2022-04-02T21:02:59.410860 #4122] WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist ``` The /var/logs/nginx/ directory with perms and ownership. Is `nginx` supposed to own healthd? ``` $ ls -l /var/log/nginx/ total 12 -rw-r--r-- 1 root root 11493 Apr 4 21:15 access.log drwxr-xr-x 2 nginx nginx 6 Apr 2 21:01 healthd drwxr-xr-x 2 root root 6 Apr 2 21:02 rotated ``` The empty /var/logs/nginx/healthd/ directory: ``` $ ls /var/log/nginx/healthd/ # this directory is empty ```
1
answers
3
votes
4
views
asked a month ago

Proper conversion of AWS Log Insights to Metrics for visualization and monitoring

TL;DR; ---- What is the proper way to create a metric so that it generates reliable information about the log insights? What is desired ------ The current Log insights can be seen similar to the following [![AWS Log insights][1]][1] However, it becomes easier to analyse these logs using the metrics (mostly because you can have multiple sources of data in the same plot and even perform math operations between them). Solution according to docs ----- Allegedly, a log can be converted to a metric filter following a guide like [this][2]. However, this approach does not seem to work entirely right (I guess because of the time frames that have to be imposed in the metric plots), providing incorrect information, for example: [![Dashboard][3]][3] Issue with solution ----- In the previous image I've created a dashboard containing the metric count (the number 7), corresponding to the sum of events each 5 minutes. Also I've added a preview of the log insight corresponding to the information used to create the event. However, as it can be seen, the number of logs is 4, but the event count displays 7. Changing the time frame in the metric generates other types of issues (e.g., selecting a very small time frame like 1 sec won't retrieve any data, or a slightly smaller time frame will now provide another wrong number: 3, when there are 4 logs, for example). P.S. ----- I've also tried converting the log insights to metrics using [this lambda function][4] as suggested by [Danil Smirnov][5] to no avail, as it seems to generate the same issues. [1]: https://i.stack.imgur.com/0pPdp.png [2]: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CountingLogEventsExample.html [3]: https://i.stack.imgur.com/Dy5td.png [4]: https://serverlessrepo.aws.amazon.com/#!/applications/arn:aws:serverlessrepo:us-east-1:085576722239:applications~logs-insights-to-metric [5]: https://blog.smirnov.la/cloudwatch-logs-insights-to-metrics-a2d197aac379
0
answers
0
votes
2
views
asked 2 months ago

SageMaker - All metrics in statistics.json by Model Quality Monitor are "0.0 +/- 0.0", but confusion matrix is built correctly for multi-class classification!!

I have scheduled an hourly model-quality-monitoring job in AWS SageMaker. both the jobs, ground-truth-merge and model-quality-monitoring completes successfully without any errors. but, all the metrics calculated by the job are "0.0 +/- 0.0" while the confustion matrix gets calculated as expected. I have done everything as mentioned in [this notebook for model-quality-monitoring from sagemaker-examples](https://github.com/aws/amazon-sagemaker-examples/blob/main/sagemaker_model_monitor/model_quality/model_quality_churn_sdk.ipynb) with very few changes and they are: 1. I have changed the model from xgboost churn to model trained on my data. 2. my input to the endpoint was csv like in the example-notebook, but output was json. 3. i have changed the problem-type from BinaryClassfication to MulticlassClassification wherever necessary. confustion matrix was built successfully, but all metrics are 0 for some reason. So, I would like the monitoring job to calculate the multi-classification metrics on data properly. **All Logs** Here's the `statistics.json` file that model-quality-monitor saved to S3 with confustion matrix built, but with 0s in all the metrics: ``` { "version" : 0.0, "dataset" : { "item_count" : 4432, "start_time" : "2022-02-23T03:00:00Z", "end_time" : "2022-02-23T04:00:00Z", "evaluation_time" : "2022-02-23T04:13:20.193Z" }, "multiclass_classification_metrics" : { "confusion_matrix" : { "0" : { "0" : 709, "2" : 530, "1" : 247 }, "2" : { "0" : 718, "2" : 497, "1" : 265 }, "1" : { "0" : 700, "2" : 509, "1" : 257 } }, "accuracy" : { "value" : 0.0, "standard_deviation" : 0.0 }, "weighted_recall" : { "value" : 0.0, "standard_deviation" : 0.0 }, "weighted_precision" : { "value" : 0.0, "standard_deviation" : 0.0 }, "weighted_f0_5" : { "value" : 0.0, "standard_deviation" : 0.0 }, "weighted_f1" : { "value" : 0.0, "standard_deviation" : 0.0 }, "weighted_f2" : { "value" : 0.0, "standard_deviation" : 0.0 }, "accuracy_best_constant_classifier" : { "value" : 0.3352888086642599, "standard_deviation" : 0.003252410977346705 }, "weighted_recall_best_constant_classifier" : { "value" : 0.3352888086642599, "standard_deviation" : 0.003252410977346705 }, "weighted_precision_best_constant_classifier" : { "value" : 0.1124185852154987, "standard_deviation" : 0.0021869336610830254 }, "weighted_f0_5_best_constant_classifier" : { "value" : 0.12965524348784485, "standard_deviation" : 0.0024239410000317335 }, "weighted_f1_best_constant_classifier" : { "value" : 0.16838092925822584, "standard_deviation" : 0.0028615098045768348 }, "weighted_f2_best_constant_classifier" : { "value" : 0.24009212108475822, "standard_deviation" : 0.003326031863819311 } } } ``` Here's how couple of lines of captured data looks like(*prettified for readability, but each line has no tab spaces as shown below*) : ``` { "captureData": { "endpointInput": { "observedContentType": "text/csv", "mode": "INPUT", "data": "0,1,628,210,30", "encoding": "CSV" }, "endpointOutput": { "observedContentType": "application/json", "mode": "OUTPUT", "data": "{\"label\":\"Transfer\",\"prediction\":2,\"probabilities\":[0.228256680901919,0.0,0.7717433190980809]}\n", "encoding": "JSON" } }, "eventMetadata": { "eventId": "a7cfba60-39ee-4796-bd85-343dcadef024", "inferenceId": "5875", "inferenceTime": "2022-02-23T04:12:51Z" }, "eventVersion": "0" } { "captureData": { "endpointInput": { "observedContentType": "text/csv", "mode": "INPUT", "data": "0,3,628,286,240", "encoding": "CSV" }, "endpointOutput": { "observedContentType": "application/json", "mode": "OUTPUT", "data": "{\"label\":\"Adoption\",\"prediction\":0,\"probabilities\":[0.99,0.005,0.005]}\n", "encoding": "JSON" } }, "eventMetadata": { "eventId": "7391ac1e-6d27-4f84-a9ad-9fbd6130498a", "inferenceId": "5876", "inferenceTime": "2022-02-23T04:12:51Z" }, "eventVersion": "0" } ``` Here's couple of lines from my ground-truths that I have uploaded to S3 look like(*prettified for readability, but each line has no tab spaces as shown below*): ``` { "groundTruthData": { "data": "0", "encoding": "CSV" }, "eventMetadata": { "eventId": "1" }, "eventVersion": "0" } { "groundTruthData": { "data": "1", "encoding": "CSV" }, "eventMetadata": { "eventId": "2" }, "eventVersion": "0" }, ``` Here's couple of lines from the ground-truth-merged file look like(*prettified for readability, but each line has no tab spaces as shown below*). this file is created by the ground-truth-merge job, which is one of the two jobs that model-quality-monitoring schedule runs: ``` { "eventVersion": "0", "groundTruthData": { "data": "2", "encoding": "CSV" }, "captureData": { "endpointInput": { "data": "1,2,1050,37,1095", "encoding": "CSV", "mode": "INPUT", "observedContentType": "text/csv" }, "endpointOutput": { "data": "{\"label\":\"Return_to_owner\",\"prediction\":1,\"probabilities\":[0.14512373737373732,0.6597074314574313,0.1951688311688311]}\n", "encoding": "JSON", "mode": "OUTPUT", "observedContentType": "application/json" } }, "eventMetadata": { "eventId": "c9e21f63-05f0-4dec-8f95-b8a1fa3483c1", "inferenceId": "4432", "inferenceTime": "2022-02-23T04:00:00Z" } } { "eventVersion": "0", "groundTruthData": { "data": "1", "encoding": "CSV" }, "captureData": { "endpointInput": { "data": "0,2,628,5,90", "encoding": "CSV", "mode": "INPUT", "observedContentType": "text/csv" }, "endpointOutput": { "data": "{\"label\":\"Adoption\",\"prediction\":0,\"probabilities\":[0.7029623691085284,0.0,0.29703763089147156]}\n", "encoding": "JSON", "mode": "OUTPUT", "observedContentType": "application/json" } }, "eventMetadata": { "eventId": "5f1afc30-2ffd-42cf-8f4b-df97f1c86cb1", "inferenceId": "4433", "inferenceTime": "2022-02-23T04:00:01Z" } } ``` Since, the confusion matrix was constructed properly, I presume that I fed the data to sagemaker-model-monitor the right-way. But, why are all the metrics 0.0, while confustion-matrix looks as expected? EDIT 1: Logs for the job are available [here](https://controlc.com/1e1781d2).
0
answers
1
votes
5
views
asked 3 months ago
  • 1
  • 90 / page