AWS EKS Deployment facing 502 Bad Gateway Error
0
I'm struggling with a persistent 502 Bad Gateway error. I've addressed all the issues I could find, including checking php-fpm, nginx services, and verifying that all pod logs and service logs are normal. Additionally, the events list appears fine, and the S3 bucket logs from the load balancer don't offer any clues about the problem. I have provided all files below and some logs i can get. Any help would be appreciated.
I am deploying a Laravel 10 app, PHP 8.1. Image built on the basis of this dockerfile.
Dockerfile:
# Use the official PHP 8.1 image as the base
FROM php:8.1-fpm
# Set working directory
WORKDIR /var/www/html
# Install system dependencies
RUN apt-get update && apt-get install -y \
git \
curl \
libpng-dev \
libonig-dev \
libxml2-dev \
zip \
unzip \
nginx \
supervisor
# Clear cache
RUN apt-get clean && rm -rf /var/lib/apt/lists/*
# Install PHP extensions
RUN docker-php-ext-install pdo_mysql mbstring exif pcntl bcmath gd
# Install Composer
COPY --from=composer:latest /usr/bin/composer /usr/bin/composer
# Copy existing application directory contents
COPY . /var/www/html
# Copy existing application directory permissions
COPY --chown=www-data:www-data . /var/www/html
# Nginx config
COPY ./nginx/nginx.conf /etc/nginx/nginx.conf
# Supervisor config
COPY ./supervisord.conf /etc/supervisor/supervisord.conf
RUN chmod +x /usr/local/sbin/php-fpm
RUN chmod +x /usr/local/etc/php-fpm.conf
# Expose port 80 for Nginx
EXPOSE 80
# Start Supervisor to manage Nginx and PHP-FPM
CMD ["/usr/bin/supervisord"]
Nginx Config File
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
include /etc/nginx/conf.d/*.conf; # Include other configurations if needed
server {
listen 80;
server_name _;
root /var/www/html/public;
index index.php index.html;
location / {
try_files $uri $uri/ /index.php?$query_string;
}
location ~ \.php$ {
include snippets/fastcgi-php.conf;
fastcgi_pass ebc-service:80;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
location ~ /\.ht {
deny all;
}
}
}
Supervisord Config File
[supervisord]
nodaemon=true
[program:nginx]
command=/usr/sbin/nginx -g 'daemon off;'
autostart=true
autorestart=true
[program:php-fpm]
command=/usr/local/sbin/php-fpm --fpm-config /usr/local/etc/php-fpm.conf
autostart=true
autorestart=true
k8s_full.yaml
apiVersion: v1
kind: Namespace
metadata:
name: ebc
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: ebc
name: ebc-deployment
spec:
replicas: 2
selector:
matchLabels:
app.kubernetes.io/name: ebc-app
template:
metadata:
labels:
app.kubernetes.io/name: ebc-app
spec:
tolerations:
- key: "eks.amazonaws.com/compute-type"
operator: "Equal"
value: "fargate"
effect: "NoSchedule"
containers:
- name: ebc-app
image: [aws-account-id].dkr.ecr.eu-west-2.amazonaws.com/[repo-name]:v7
imagePullPolicy: Always
ports:
- containerPort: 80
resources:
requests:
memory: "16Gi"
cpu: "4"
ephemeral-storage: "20Gi"
limits:
memory: "64Gi"
cpu: "8"
ephemeral-storage: "50Gi"
imagePullSecrets:
- name: ebc-docker-ecr
---
apiVersion: v1
kind: Service
metadata:
namespace: ebc
name: ebc-service
spec:
ports:
- port: 80
targetPort: 80
protocol: TCP
type: NodePort
selector:
app.kubernetes.io/name: ebc-app
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
namespace: ebc
name: ebc-ingress
annotations:
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
spec:
ingressClassName: alb
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: ebc-service
port:
number: 80
Checks for problems:
kubectl get pods -n ebc
NAME READY STATUS RESTARTS AGE
ebc-deployment-5df7d6c984-hsccw 1/1 Running 0 177m
ebc-deployment-5df7d6c984-qjwcf 1/1 Running 0 3h
kubectl logs ebc-deployment-5df7d6c984-hsccw -n ebc
/usr/lib/python3/dist-packages/supervisor/options.py:474: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
self.warnings.warn(
2024-07-23 09:46:38,417 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message.
2024-07-23 09:46:38,419 INFO supervisord started with pid 1
2024-07-23 09:46:39,420 INFO spawned: 'nginx' with pid 7
2024-07-23 09:46:39,422 INFO spawned: 'php-fpm' with pid 8
2024-07-23 09:46:40,459 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-07-23 09:46:40,459 INFO success: php-fpm entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
kubectl describe pod ebc-deployment-5df7d6c984-hsccw -n ebc
Name: ebc-deployment-5df7d6c984-hsccw
Namespace: ebc
Priority: 2000001000
Priority Class Name: system-node-critical
Service Account: default
Node: fargate-ip-192-168-100-132.eu-west-2.compute.internal/192.168.100.132
Start Time: Tue, 23 Jul 2024 10:44:17 +0100
Labels: app.kubernetes.io/name=ebc-app
eks.amazonaws.com/fargate-profile=ebc-profile
pod-template-hash=5df7d6c984
Annotations: CapacityProvisioned: 4vCPU 17GB
Logging: LoggingDisabled: LOGGING_CONFIGMAP_NOT_FOUND
kubectl.kubernetes.io/restartedAt: 2024-07-23T10:41:12+01:00
Status: Running
IP: 192.168.100.132
IPs:
IP: 192.168.100.132
Controlled By: ReplicaSet/ebc-deployment-5df7d6c984
Containers:
ebc-app:
Container ID: containerd://2019194b04471fe0d0961a16a8216d281563d81e28901e11deb92479389d2361
Image: [aws-account-id].dkr.ecr.eu-west-2.amazonaws.com/[repo-name]:v7
Image ID: [aws-account-id].dkr.ecr.eu-west-2.amazonaws.com/[repo-name]@sha256:[sha256]
Port: 80/TCP
Host Port: 0/TCP
State: Running
Started: Tue, 23 Jul 2024 10:46:38 +0100
Ready: True
Restart Count: 0
Limits:
cpu: 8
ephemeral-storage: 50Gi
memory: 64Gi
Requests:
cpu: 4
ephemeral-storage: 20Gi
memory: 16Gi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rlrc5 (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-rlrc5:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: eks.amazonaws.com/compute-type=fargate:NoSchedule
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events: <none>
kubectl get endpoints -n ebc
NAME ENDPOINTS AGE
ebc-service 192.168.100.132:80,192.168.162.68:80 4d20h
kubectl get services -n ebc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ebc-service NodePort 10.100.60.51 <none> 80:32177/TCP 4d20h
kubectl describe service ebc-service -n ebc
Name: ebc-service
Namespace: ebc
Labels: <none>
Annotations: <none>
Selector: app.kubernetes.io/name=ebc-app
Type: NodePort
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.100.60.51
IPs: 10.100.60.51
Port: <unset> 80/TCP
TargetPort: 80/TCP
NodePort: <unset> 32177/TCP
Endpoints: 192.168.100.132:80,192.168.162.68:80
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
kubectl get deployments -n ebc
NAME READY UP-TO-DATE AVAILABLE AGE
ebc-deployment 2/2 2 2 4d20h
kubectl get nodes -n ebc
NAME STATUS ROLES AGE VERSION
fargate-ip-192-168-100-132.eu-west-2.compute.internal Ready <none> 3h v1.30.0-eks-404b9c6
fargate-ip-192-168-140-145.eu-west-2.compute.internal Ready <none> 4d v1.30.0-eks-404b9c6
fargate-ip-192-168-150-94.eu-west-2.compute.internal Ready,SchedulingDisabled <none> 5d1h v1.30.0-eks-404b9c6
fargate-ip-192-168-162-12.eu-west-2.compute.internal Ready <none> 4d v1.30.0-eks-404b9c6
fargate-ip-192-168-162-68.eu-west-2.compute.internal Ready <none> 3h2m v1.30.0-eks-404b9c6
fargate-ip-192-168-190-76.eu-west-2.compute.internal Ready <none> 5d v1.30.0-eks-404b9c6
kubectl describe ingress ebc-ingress -n ebc
Name: ebc-ingress
Namespace: ebc
Address: [ingress-url].eu-west-2.elb.amazonaws.com
Ingress Class: alb
Default backend: <default>
Rules:
Host Path Backends
---- ---- --------
*
/ ebc-service:80 (192.168.100.132:80,192.168.162.68:80)
Annotations: alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
Events: <none>
- Topics
- DevOpsComputeContainers
- Language
- English
asked 2 years ago1.8K views
1 Answer
- Newest
- Most votes
- Most comments
Are these answers helpful? Upvote the correct answer to help the community benefit from your knowledge.
0
Hello,
I took a quick look at your nginx configuration and I think the fastcgi_pass configuration in "location ~ .php$" in server is wrong.
According to How to Configure PHP-FPM with NGINX, I think you can do the following to fix this problem:
- Check your /usr/local/etc/php-fpm.conf, find the line like listen = xxxxxx and copy xxxxxx that cloud be 127.0.0.1:9000 or /path/to/unix/socket(example: /var/run/php7.2-fpm-wordpress-site.sock)
- Change the fastcgi_pass in location ~ .php$ in your nginx configuration to 127.0.0.1:9000 or unix:</path/to/unix/socket>(example: unix:/var/run/php7.2-fpm-wordpress-site.sock)
- Rebuild and restart your service.
Hope this can help you :)
answered 2 years ago
Relevant content
asked 4 years ago
- Accepted Answer
asked 2 years ago
asked a year ago
AWS OFFICIALUpdated 3 months ago
