npm install hangs in ec2 instance provisioned through elastic beanstalk

Question

**npm install hangs with no error**

having some issue with npm on AWS/EC2 instance invoked through Elastic Beanstalk

also the same installation runs fine on non-aws  ubuntu and macos

if I try to install some smaller packages like lodash, mongoose,  etc.. it works fine, however when trying to install express or other larger packages it hangs

**machine type:** t2.large and t2.medium (running amazon linux)

**versions tried:**
#1 node 18.20.4/npm 10.7.1
#2 node 20.18.0/npm 10.8.2

**uname -r: ** 6.1.115-126.197.amzn2023.x86_64

**disk: 8GB - lots of free disk space**

```
	[ec2-user@ip-172-31-46-96 ~]$ df -h
	Filesystem      Size  Used Avail Use% Mounted on
	devtmpfs        4.0M     0  4.0M   0% /dev
	tmpfs           3.9G     0  3.9G   0% /dev/shm
	tmpfs           1.6G   17M  1.6G   2% /run
	/dev/xvda1      8.0G  5.5G  2.5G  70% /
	tmpfs           3.9G   20M  3.9G   1% /tmp
```

**here is what top output looks like when npm install hangs - so, not short on memory**

**top output**
`22:36:46 up 26 min,  3 users,  load average: 3.87, 2.80, 1.68
Tasks: 126 total,   1 running, 125 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.2 us,  0.0 sy,  0.0 ni, 94.3 id,  0.0 wa,  0.0 hi,  0.0 si,  5.5 st
MiB Mem :   7930.2 total,   3466.0 free,   1015.9 used,   3448.3 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   6585.0 avail Mem `

**dmesg shows a lot of these messages**

```
[ 1349.092480] INFO: task iou-sqp-3249:3260 blocked for more than 1105 seconds.
[ 1349.093925]       Not tainted 6.1.115-126.197.amzn2023.x86_64 #1
[ 1349.095186] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1349.097048] task:iou-sqp-3249    state:D stack:0     pid:3260  ppid:3247   flags:0x00004000
[ 1349.098968] Call Trace:
[ 1349.099781]  
[ 1349.100549]  __schedule+0x1ad/0x530
[ 1349.101497]  ? _raw_spin_unlock+0xa/0x30
[ 1349.102529]  schedule+0x5a/0xd0
[ 1349.103434]  schedule_preempt_disabled+0x11/0x20
[ 1349.104520]  __mutex_lock.constprop.0+0x372/0x6c0
[ 1349.105634]  ? io_submit_sqes+0x303/0x390
[ 1349.106644]  io_sq_thread+0x275/0x4e0
[ 1349.107616]  ? membarrier_register_private_expedited+0x90/0x90
[ 1349.108860]  ? io_sqd_handle_event+0xd0/0xd0
[ 1349.109904]  ret_from_fork+0x22/0x30
[ 1349.110863]  
```

**Here is the last bit of entries from the npm log**

```
`
`600 http fetch GET 200 https://registry.npmjs.org/binary-extensions 3ms (cache hit)
5601 silly packumentCache full:https://registry.npmjs.org/binary-extensions set size:49673 disposed:false
5602 silly placeDep node_modules/ts-node-dev binary-extensions@2.3.0 OK for: is-binary-path@2.1.0 want: ^2.0.0
5603 silly placeDep node_modules/ts-node-dev source-map@0.6.1 OK for: source-map-support@0.5.21 want: ^0.6.0
5604 silly placeDep node_modules/unset-value has-values@0.1.4 OK for: has-value@0.3.1 want: ^0.1.4
5605 silly placeDep node_modules/unset-value/node_modules/has-value isobject@2.1.0 OK for: has-value@0.3.1 want: ^2.0.0
5606 silly placeDep node_modules/wrap-ansi ansi-regex@5.0.1 OK for: strip-ansi@6.0.1 want: ^5.0.1
5607 silly placeDep node_modules/readdirp/node_modules/expand-brackets is-descriptor@0.1.7 OK for: define-property@0.2.5 want: ^0.1.0
5608 silly reify moves {}``
```

running top and free show there is lots of free disk space and memory available when this happens

don't see any errors in npm log files

tried to clean npm cache before running install, didn't help

moved from  t2.medium to t2.large to get extra horsepower in terms of mem and space

tried running npm with --no-progress flag...didn't work

satyen · Accepted Answer

turns out it is caused by a bug in amazon linux kernel ....can't believe amazon is throwing out crap to waste developer time  w/o testing their kernels thoroughly (1 full week wasted in my case)

just switch to a diff version of the kernel than the one I was using (6.1.115-126.197.amzn2023.x86_64)  and everything works...turns out amazon does know of this bug but still has it out for deployment by suckers like us...I hope I don't sound too negative, but that is what happens when you waste a weeks precious time because of slack attitudes of others  who just don't care

Elad · Answer

In my case, the `npm install` hang issue on AWS Elastic Beanstalk was not related to the AWS Linux kernel version, but instead caused by a lack of available RAM in the `t3.micro` instance type.

The `npm install` command can be quite RAM-intensive. Either switch to a larger instance such as `t3.small` or set up swap memory (at least 1GB). In addition, make sure that `vm.swappiness` is not set to `0` otherwise the system won't use the swap and the `npm install` will hang.

JMV8 · Answer

This is a known bug in the kernel which is affecting Amazon Linux 2023. A fix has been released to AL2023 but is not available in the latest beanstalk images. Downgrading your environment to(or creating a new environment on) the prior version will work, though isn't a great solution because there are some unfixed CVEs in those images such as CVE-2024-45490. I believe this also affects some other environments and not just NodeJS.

https://github.com/amazonlinux/amazon-linux-2023/issues/840#issuecomment-2485782075

https://github.com/amazonlinux/amazon-linux-2023/issues/856

https://github.com/nodejs/node/issues/55587

I assume there will be new Elastic Beanstalk images released soon but I am not sure.

re:Post Agent · Answer

The issue you're experiencing with npm install hanging on your EC2 instance provisioned through Elastic Beanstalk is likely related to network connectivity or DNS resolution problems rather than resource constraints.

Given that smaller packages install fine, but larger packages like Express cause the installation to hang, it suggests that the problem occurs when npm needs to download and install many dependencies.

Here are a few potential solutions and workarounds you can try:

1. Use npm shrinkwrap: Create an npm-shrinkwrap.json file to lock down your application's dependencies. This can help ensure that dependencies are only installed once and prevent npm from trying to fetch new versions during deployment.

2. Include node_modules in your source bundle: If you're experiencing deployment issues when Elastic Beanstalk runs npm install, consider including the node_modules directory with the dependency modules in your application source bundle. This can circumvent issues with installing dependencies from the public npm registry.

3. Check network and DNS settings: Ensure that your EC2 instance has proper network connectivity and can resolve DNS queries to the npm registry. You may need to check your VPC settings, security groups, and network ACLs.

4. Use a custom npm registry: If the public npm registry is causing issues, consider setting up and using a private npm registry within your VPC.

5. Increase the npm timeout: Try running npm install with an increased timeout, e.g., npm install --timeout=600000 (10 minutes).

6. Use a different Node.js version: Although you've tried Node.js 18 and 20, you could experiment with other versions supported by Elastic Beanstalk to see if the issue persists.

7. Investigate IO issues: The dmesg output you provided suggests there might be IO-related issues. You could try using a different instance type with better IO performance, such as a t3 or m5 instance.

If these solutions don't resolve the issue, you may need to further investigate by enabling more verbose logging for npm and checking Elastic Beanstalk logs for any additional clues about the root cause of the problem.  
**Sources**  
[Configuring your application's dependencies on Elastic Beanstalk - AWS Elastic Beanstalk](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/nodejs-platform-dependencies.html)  
[Locking dependencies with npm shrinkwrap on Elastic Beanstalk - AWS Elastic Beanstalk](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/nodejs-platform-shrinkwrap.html)  
[Elastic Beanstalk supported platforms - AWS Elastic Beanstalk](https://docs.aws.amazon.com/elasticbeanstalk/latest/platforms/platforms-supported.html)

npm install hangs in ec2 instance provisioned through elastic beanstalk

Add your answer

Relevant content