Ok I managed to resolved the issue using an Admin Account and then re-applying the RBAC

Here's a comprehensive guide to recover from accidental RBAC deletion in EKS and prevent similar issues in the future:

Recovery Steps:

Using AWS IAM Authentication:

Ensure you have AWS admin access

aws eks update-kubeconfig --name cluster-name --region region-name

Verify your AWS IAM role has appropriate permissions in aws-auth ConfigMap

kubectl get configmap aws-auth -n kube-system -o yaml

Restore RBAC Rules:

Create cluster-admin role binding

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: eks-admin
subjects:
- kind: User
  name: admin
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io
Update aws-auth ConfigMap:
apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: arn:aws:iam::ACCOUNT_ID:role/YOUR_NODE_INSTANCE_ROLE
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes
    - rolearn: arn:aws:iam::ACCOUNT_ID:role/YOUR_ADMIN_ROLE
      username: admin
      groups:
        - system:masters

Best Practices:

Regular Backups:

Backup RBAC configurations

kubectl get clusterroles,clusterrolebindings,roles,rolebindings --all-namespaces -o yaml > rbac-backup.yaml

Use tools like Velero for cluster backups

velero backup create my-backup --include-namespaces kube-system

Use GitOps:

Example using ArgoCD

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cluster-rbac
spec:
  source:
    repoURL: https://your-git-repo
    path: rbac-configs
    targetRevision: HEAD
  destination:
    server: https://kubernetes.default.svc
    namespace: default

Implement Least Privilege Access Public Resources:

Official Documentation: EKS RBAC Documentation Kubernetes RBAC Documentation Tools and Utilities: Velero Backup Tool AWS IAM Authenticator Best Practices Guides: EKS Best Practices Guide Kubernetes Security Best Practices Preventive Measures:

Version Control:

Store RBAC configurations in Git

git add rbac-configs/
git commit -m "Update RBAC configurations"
git push

Implement Change Controls:

Use OPA/Gatekeeper for policy enforcement

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: require-change-approval
spec:
  match:
    kinds:
      - apiGroups: ["rbac.authorization.k8s.io"]
        kinds: ["ClusterRole", "ClusterRoleBinding"]

Monitoring and Alerting:

Example Prometheus Alert

groups:
- name: RBACChanges
  rules:
  - alert: RBACModification
    expr: kube_clusterrole_changes > 0
    for: 1m
    labels:
      severity: critical
Regular Auditing:

Enable audit logging

kubectl logs -n kube-system kube-apiserver-* | grep "rbac"

Use AWS CloudTrail for API activity monitoring

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=UpdateClusterConfig

Emergency Contact List:

• Keep AWS Support contact information
• Maintain internal escalation procedures
• Document recovery procedures

Remember to:

• Always test RBAC changes in a non-production environment first
• Maintain documentation of your RBAC configuration
• Regularly review and update access permissions
• Implement proper change management procedures
• Use Infrastructure as Code (IaC) for RBAC management

These practices should help prevent and recover from RBAC-related issues in your EKS cluster.