mircok8s set up metallb loadbalancer on ec2 machines

I created a k8s cluster for learning purposes and got stuck with the metallb loadbalancer. Here are my steps so far: I am using microk8s distribution on three ec2 medium instances with ubuntu server 24.04 LTS. I successfully installed mircok8s on all machines and added them to a cluster. I have a 4th machine that I use as a jumphost, where I execute kubectl commands etc.

ubuntu@jumphost:~$ kubectl get nodes -o wide
NAME             STATUS   ROLES    AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION   CONTAINER-RUNTIME
ip-172-31-1-38   Ready    <none>   171m   v1.30.1   172.31.1.38   <none>        Ubuntu 24.04 LTS   6.8.0-1009-aws   containerd://1.6.28
ip-172-31-3-37   Ready    <none>   168m   v1.30.1   172.31.3.37   <none>        Ubuntu 24.04 LTS   6.8.0-1009-aws   containerd://1.6.28
ip-172-31-9-25   Ready    <none>   168m   v1.30.1   172.31.9.25   <none>        Ubuntu 24.04 LTS   6.8.0-1009-aws   containerd://1.6.28

I installed helm on the jumphost and installed metallb via helm:

helm repo add metallb https://metallb.github.io/metallb
helm install metallb metallb/metallb --namespace=metallb-system --create-namespace

Following one of several online tutorials, I created an IPAddressPool and an L2Advertisement like this:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: pool-aws
  namespace: metallb-system
spec:
  addresses:
  # this is the public ip of node ip-172-31-9-25
  - 15.237.174.206/32
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: l2-aws
  namespace: metallb-system
spec:
  ipAddressPools:
  - pool-aws

I used the public ip address of my first node ("ip-172-31-9-25"). Everything looks fine to me at this point, however I cannot reach the service from outside the cluster.

ubuntu@jumphost:~$ kubectl get svc
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)        AGE
kubernetes   ClusterIP      10.152.183.1    <none>           443/TCP        3h5m
web-app      LoadBalancer   10.152.183.75   15.237.174.206   80:32756/TCP   3s

ubuntu@jumphost:~$ kubectl describe svc web-app
Name:                     web-app
Namespace:                default
Labels:                   app.kubernetes.io/name=web-app
Annotations:              metallb.universe.tf/ip-allocated-from-pool: pool-aws
Selector:                 app.kubernetes.io/name=web-app
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.152.183.138
IPs:                      10.152.183.138
LoadBalancer Ingress:     15.237.174.206
Port:                     http  80/TCP
TargetPort:               80/TCP
NodePort:                 http  32382/TCP
Endpoints:                10.1.152.6:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason        Age               From                Message
  ----    ------        ----              ----                -------
  Normal  IPAllocated   10s               metallb-controller  Assigned IP ["15.237.174.206"]
  Normal  nodeAssigned  7s                metallb-speaker     announcing from node "ip-172-31-9-25" with protocol "layer2"
  Normal  nodeAssigned  7s                metallb-speaker     announcing from node "ip-172-31-3-37" with protocol "layer2"
  Normal  nodeAssigned  7s                metallb-speaker     announcing from node "ip-172-31-1-38" with protocol "layer2"

I tried curl 15.237.174.206 from the jumphost and my windows machine. There is no apparent firewall (security group) problem - I shut down microk8s and run a docker image directly on the node machine (docker container run --name my_nginx -d -p 80:80 nginx) and was able to reach it from both the jumphost and my windows machine.

What I noticed: In AWS, the ec2 machines do not have a second network interface. I applied exactly the some configuration to another a cluster that I created on another cloud provider (the ubuntu virtual machines have two network interfaces there, one private and one public) and everything just worked. I used exactly the same yaml files.

I hope that someone can tell my what I missed or what I should do for debugging this further.

Best regards, mike