HTTP 1xx codes causes environment to be incorrectly classified as unhealthy

0

Hi,

We believe that HTTP 1xx return codes (in our case 101 for websockets) are being counted by the enhanced health monitoring.

When a large enough count of these 101 codes are returned by the server, it decreases the overall 2xx percentage. When the 2xx percentage drops below a certain level, Elastic Beanstalk marks the environment as degraded.

We are constantly finding our environment marked as degraded due to this reason.

Is it possible not to have the 1xx codes counted in the health check?

Atavism
asked 3 years ago196 views
1 Answer
0

Update:

We ended up modifying our nginx configuration to not log the 101's in the healthd logs (we currently use Amazon Linux 1 and nginx).

Essentially what we did was change /etc/nginx/conf.d/healthd_http.conf from:

log_format healthd '$msec"$uri"'
'$status"$request_time"$upstream_response_time"'
'$http_x_forwarded_for';

To:

map $status $logflag {
101 0;
default 1;
}
log_format healthd '$msec"$uri"'
'$status"$request_time"$upstream_response_time"'
'$http_x_forwarded_for';

And change /etc/nginx/conf.d/elasticbeanstalk/healthd.conf from:

if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})T(\d{2})") {
set $year $1;
set $month $2;
set $day $3;
set $hour $4;
}

access_log /var/log/nginx/healthd/application.log.$year-$month-$day-$hour healthd;

To:

if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})T(\d{2})") {
set $year $1;
set $month $2;
set $day $3;
set $hour $4;
}

access_log /var/log/nginx/healthd/application.log.$year-$month-$day-$hour healthd if=$logflag;

To achieve this, we used the .ebextensions/ method. i.e:
We created the folders in the root of our package .ebextensions/nginx/conf.d/ and .ebextensions/nginx/conf.d/elasticbeanstalk/

In the .ebextensions/nginx/conf.d/ folder we created the healthd_http.conf file (.ebextensions/nginx/conf.d/healthd_http.conf) and added the contents:

map $status $logflag {
101 0;
default 1;
}
log_format healthd '$msec"$uri"'
'$status"$request_time"$upstream_response_time"'
'$http_x_forwarded_for';

And in the .ebextensions/nginx/conf.d/elasticbeanstalk/ folder we created the healthd.conf file (.ebextensions/nginx/conf.d/elasticbeanstalk/healthd.conf) and added the contents:

if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})T(\d{2})") {
set $year $1;
set $month $2;
set $day $3;
set $hour $4;
}

access_log /var/log/nginx/healthd/application.log.$year-$month-$day-$hour healthd if=$logflag;

And once deployed, the alerts stopped for us.

We adapted the solution from a "Programmer All" article - we couldn't find the author's name, but thanks to them! Their solution is for Amazon Linux 2, so if you want that one google with the quotes "Health status of Elastic Beanstalk changed to 'Serve' due to the http code 101" and it should be the first result.

Atavism
answered 3 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions