logging of success/failed status of cron job at EKS

The issue you're experiencing with the pod_status_failed and pod_status_succeeded metrics not being reported for your cron jobs in Amazon EKS (Elastic Kubernetes Service) could be related to how these metrics are collected and the nature of cron jobs in Kubernetes.

Cron jobs in Kubernetes create pods that run to completion and then terminate. The pod_status_failed and pod_status_succeeded metrics are typically used for long-running pods rather than short-lived ones like those created by cron jobs.

To effectively monitor the success or failure of your cron jobs, you might want to consider the following alternatives:

1. Use CloudWatch Container Insights: This provides more comprehensive metrics for Amazon EKS at the cluster, node, pod, task, and service level. It might capture more detailed information about your cron job executions.

2. Implement custom logging: You can modify your cron job to output specific log messages indicating success or failure. These logs can be captured and sent to CloudWatch Logs using a logging solution like Fluent Bit or the CloudWatch agent.

3. Use Kubernetes Events: Kubernetes generates events for various activities, including job completions and failures. You can capture these events and send them to CloudWatch for monitoring.

4. Prometheus metrics: If you're using Prometheus for metrics, you can set up custom metrics in your cron job to explicitly report its status upon completion.

5. Check job status directly: You can use kubectl or the Kubernetes API to query the status of completed jobs and send this information to CloudWatch.

If you need more detailed monitoring, you might want to consider implementing a custom solution that explicitly reports the job status to CloudWatch at the end of each job execution. This could involve adding a step in your job that uses the AWS SDK to send a custom metric or log entry to CloudWatch indicating the job's final status.

Remember to ensure that your EKS nodes have the necessary IAM permissions (like the CloudWatchAgentServerPolicy) to send metrics and logs to CloudWatch.

If you continue to face issues, you may want to verify your CloudWatch agent configuration and ensure it's properly set up to collect the metrics you're interested in from your EKS cluster.
Sources
Logging and monitoring on Amazon EKS - AWS Prescriptive Guidance
Monitor your cluster performance and view logs - Amazon EKS
Logging for Amazon EKS - AWS Prescriptive Guidance

Hi, It really depends on the way the metrics being scrapped from your EKS cluster into CloudWatch. Please have a look on this guide : https://docs.aws.amazon.com/prescriptive-guidance/latest/implementing-logging-monitoring-cloudwatch/kubernetes-eks-metrics.html to get more clarity on how to setup it. Thanks