I have a process called server (developed in rust, but I don't think that is critical), it should run with about constant load 24/7. The CPU load for the process is in the sustainable zone. The system is running Ubuntu 20.04
Yesterday it was killed with an out of memory message. Looking at kern.log, I can see that there are a number of messages before the process was killed, the first was "unattended-upgr invoked oom-killer: gfp_mask=0x1100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0"
Feb 10 07:01:38 ip- kernel: [1133764.101038] unattended-upgr invoked oom-killer: gfp_mask=0x1100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0
Feb 10 07:01:38 ip- kernel: [1133764.101047] CPU: 0 PID: 640430 Comm: unattended-upgr Not tainted 5.15.0-1028-aws #32~20.04.1-Ubuntu
Feb 10 07:01:38 ip- kernel: [1133764.101050] Hardware name: Xen HVM domU, BIOS 4.11.amazon 08/24/2006
Looking at the memory usage you can see that my server app is using 155MB RAM. Surprisingly snapd is using 184MB, amazon-ssm-agen 307MB, and ssm-agent-worke 310MB
Feb 10 07:01:38 ip- kernel: [1133764.101272] Tasks state (memory values in pages):
Feb 10 07:01:38 ip- kernel: [1133764.101272] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Feb 10 07:01:38 ip- kernel: [1133764.101275] [ 266] 0 266 70052 4501 98304 0 -1000 multipathd
Feb 10 07:01:38 ip- kernel: [1133764.101279] [ 441] 0 441 59353 451 90112 0 0 accounts-daemon
Feb 10 07:01:38 ip- kernel: [1133764.101281] [ 442] 0 442 637 166 40960 0 0 acpid
Feb 10 07:01:38 ip- kernel: [1133764.101283] [ 446] 0 446 2137 582 53248 0 0 cron
Feb 10 07:01:38 ip- kernel: [1133764.101286] [ 449] 103 449 1982 753 61440 0 -900 dbus-daemon
Feb 10 07:01:38 ip- kernel: [1133764.101288] [ 458] 0 458 7469 2827 94208 0 0 networkd-dispat
Feb 10 07:01:38 ip- kernel: [1133764.101290] [ 459] 0 459 58181 224 81920 0 0 polkitd
Feb 10 07:01:38 ip- kernel: [1133764.101292] [ 464] 0 464 307844 1343 180224 0 0 amazon-ssm-agen
Feb 10 07:01:38 ip- kernel: [1133764.101294] [ 470] 0 470 4418 867 69632 0 0 systemd-logind
Feb 10 07:01:38 ip- kernel: [1133764.101297] [ 473] 0 473 98274 773 122880 0 0 udisksd
Feb 10 07:01:38 ip- kernel: [1133764.101298] [ 474] 0 474 951 455 49152 0 0 atd
Feb 10 07:01:38 ip- kernel: [1133764.101300] [ 498] 0 498 60334 581 106496 0 0 ModemManager
Feb 10 07:01:38 ip- kernel: [1133764.101302] [ 501] 0 501 1840 435 49152 0 0 agetty
Feb 10 07:01:38 ip- kernel: [1133764.101304] [ 515] 0 515 1459 382 45056 0 0 agetty
Feb 10 07:01:38 ip- kernel: [1133764.101306] [ 556] 0 556 27032 2749 114688 0 0 unattended-upgr
Feb 10 07:01:38 ip- kernel: [1133764.101308] [ 639] 0 639 3046 704 61440 0 -1000 sshd
Feb 10 07:01:38 ip- kernel: [1133764.101310] [ 788] 0 788 310204 1846 192512 0 0 ssm-agent-worke
Feb 10 07:01:38 ip- kernel: [1133764.101312] [ 8869] 0 8869 2397 949 57344 0 -1000 systemd-udevd
Feb 10 07:01:38 ip- kernel: [1133764.101314] [ 11347] 100 11347 6817 875 73728 0 0 systemd-network
Feb 10 07:01:38 ip- kernel: [1133764.101316] [ 11355] 101 11355 6137 1403 81920 0 0 systemd-resolve
Feb 10 07:01:38 ip- kernel: [1133764.101318] [ 11359] 0 11359 17149 1077 159744 0 -250 systemd-journal
Feb 10 07:01:38 ip- kernel: [1133764.101320] [ 11463] 102 11463 22721 511 86016 0 0 systemd-timesyn
Feb 10 07:01:38 ip- kernel: [1133764.101323] [ 11642] 104 11642 56125 343 81920 0 0 rsyslogd
Feb 10 07:01:38 ip- kernel: [1133764.101325] [ 190175] 1000 190175 4732 413 73728 0 0 systemd
Feb 10 07:01:38 ip- kernel: [1133764.101327] [ 190176] 1000 190176 42626 1132 94208 0 0 (sd-pam)
Feb 10 07:01:38 ip- kernel: [1133764.101329] [ 191381] 0 191381 184017 4350 249856 0 -900 snapd
Feb 10 07:01:38 ip- kernel: [1133764.101332] [ 355693] 1000 355693 2164 983 61440 0 0 tmux: server
Feb 10 07:01:38 ip- kernel: [1133764.101334] [ 355694] 1000 355694 2506 872 53248 0 0 bash
Feb 10 07:01:38 ip- kernel: [1133764.101336] [ 611154] 0 611154 3453 978 65536 0 0 sshd
Feb 10 07:01:38 ip- kernel: [1133764.101338] [ 611258] 1000 611258 3574 982 65536 0 0 sshd
Feb 10 07:01:38 ip- kernel: [1133764.101340] [ 611259] 1000 611259 2510 901 53248 0 0 bash
Feb 10 07:01:38 ip- kernel: [1133764.101342] [ 622945] 0 622945 3452 977 65536 0 0 sshd
Feb 10 07:01:38 ip- kernel: [1133764.101343] [ 623052] 1000 623052 3485 819 65536 0 0 sshd
Feb 10 07:01:38 ip- kernel: [1133764.101345] [ 623054] 1000 623054 3038 1483 57344 0 0 bash
Feb 10 07:01:38 ip- kernel: [1133764.101347] [ 629386] 1000 629386 155370 35002 393216 0 0 server
Feb 10 07:01:38 ip- kernel: [1133764.101349] [ 638483] 0 638483 654 434 45056 0 0 apt.systemd.dai
Feb 10 07:01:38 ip- kernel: [1133764.101351] [ 638487] 0 638487 654 434 45056 0 0 apt.systemd.dai
Feb 10 07:01:38 ip- kernel: [1133764.101353] [ 638521] 0 638521 81187 14207 425984 0 0 unattended-upgr
Feb 10 07:01:38 ip- kernel: [1133764.101355] [ 640399] 0 640399 70904 836 151552 0 0 packagekitd
Feb 10 07:01:38 ip- kernel: [1133764.101357] [ 640430] 0 640430 92707 24658 425984 0 0 unattended-upgr
Immediately after this, my process was killed:
Feb 10 07:01:38 ip- kernel: [1133764.101359] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-296.scope,task=server,pid=629386,uid=1000
Feb 10 07:01:38 ip- kernel: [1133764.101385] Out of memory: Killed process 629386 (server) total-vm:621480kB, anon-rss:138304kB, file-rss:1704kB, shmem-rss:0kB, UID:1000 pgtables:384kB oom_score_adj:0
Why was my process killed? It was not the reason that the system ran out of memory. How much of the available memory can I use on a Lightsail server?
AWS OFFICIALUpdated 2 years ago
