Lightsail killed my process due to out of memory, unattended-upgr invoked oom-killer

0

I have a process called server (developed in rust, but I don't think that is critical), it should run with about constant load 24/7. The CPU load for the process is in the sustainable zone. The system is running Ubuntu 20.04

Yesterday it was killed with an out of memory message. Looking at kern.log, I can see that there are a number of messages before the process was killed, the first was "unattended-upgr invoked oom-killer: gfp_mask=0x1100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0"

Feb 10 07:01:38 ip- kernel: [1133764.101038] unattended-upgr invoked oom-killer: gfp_mask=0x1100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0
Feb 10 07:01:38 ip- kernel: [1133764.101047] CPU: 0 PID: 640430 Comm: unattended-upgr Not tainted 5.15.0-1028-aws #32~20.04.1-Ubuntu
Feb 10 07:01:38 ip- kernel: [1133764.101050] Hardware name: Xen HVM domU, BIOS 4.11.amazon 08/24/2006

Looking at the memory usage you can see that my server app is using 155MB RAM. Surprisingly snapd is using 184MB, amazon-ssm-agen 307MB, and ssm-agent-worke 310MB

Feb 10 07:01:38 ip- kernel: [1133764.101272] Tasks state (memory values in pages):
Feb 10 07:01:38 ip- kernel: [1133764.101272] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Feb 10 07:01:38 ip- kernel: [1133764.101275] [    266]     0   266    70052     4501    98304        0         -1000 multipathd
Feb 10 07:01:38 ip- kernel: [1133764.101279] [    441]     0   441    59353      451    90112        0             0 accounts-daemon
Feb 10 07:01:38 ip- kernel: [1133764.101281] [    442]     0   442      637      166    40960        0             0 acpid
Feb 10 07:01:38 ip- kernel: [1133764.101283] [    446]     0   446     2137      582    53248        0             0 cron
Feb 10 07:01:38 ip- kernel: [1133764.101286] [    449]   103   449     1982      753    61440        0          -900 dbus-daemon
Feb 10 07:01:38 ip- kernel: [1133764.101288] [    458]     0   458     7469     2827    94208        0             0 networkd-dispat
Feb 10 07:01:38 ip- kernel: [1133764.101290] [    459]     0   459    58181      224    81920        0             0 polkitd
Feb 10 07:01:38 ip- kernel: [1133764.101292] [    464]     0   464   307844     1343   180224        0             0 amazon-ssm-agen
Feb 10 07:01:38 ip- kernel: [1133764.101294] [    470]     0   470     4418      867    69632        0             0 systemd-logind
Feb 10 07:01:38 ip- kernel: [1133764.101297] [    473]     0   473    98274      773   122880        0             0 udisksd
Feb 10 07:01:38 ip- kernel: [1133764.101298] [    474]     0   474      951      455    49152        0             0 atd
Feb 10 07:01:38 ip- kernel: [1133764.101300] [    498]     0   498    60334      581   106496        0             0 ModemManager
Feb 10 07:01:38 ip- kernel: [1133764.101302] [    501]     0   501     1840      435    49152        0             0 agetty
Feb 10 07:01:38 ip- kernel: [1133764.101304] [    515]     0   515     1459      382    45056        0             0 agetty
Feb 10 07:01:38 ip- kernel: [1133764.101306] [    556]     0   556    27032     2749   114688        0             0 unattended-upgr
Feb 10 07:01:38 ip- kernel: [1133764.101308] [    639]     0   639     3046      704    61440        0         -1000 sshd
Feb 10 07:01:38 ip- kernel: [1133764.101310] [    788]     0   788   310204     1846   192512        0             0 ssm-agent-worke
Feb 10 07:01:38 ip- kernel: [1133764.101312] [   8869]     0  8869     2397      949    57344        0         -1000 systemd-udevd
Feb 10 07:01:38 ip- kernel: [1133764.101314] [  11347]   100 11347     6817      875    73728        0             0 systemd-network
Feb 10 07:01:38 ip- kernel: [1133764.101316] [  11355]   101 11355     6137     1403    81920        0             0 systemd-resolve
Feb 10 07:01:38 ip- kernel: [1133764.101318] [  11359]     0 11359    17149     1077   159744        0          -250 systemd-journal
Feb 10 07:01:38 ip- kernel: [1133764.101320] [  11463]   102 11463    22721      511    86016        0             0 systemd-timesyn
Feb 10 07:01:38 ip- kernel: [1133764.101323] [  11642]   104 11642    56125      343    81920        0             0 rsyslogd
Feb 10 07:01:38 ip- kernel: [1133764.101325] [ 190175]  1000 190175     4732      413    73728        0             0 systemd
Feb 10 07:01:38 ip- kernel: [1133764.101327] [ 190176]  1000 190176    42626     1132    94208        0             0 (sd-pam)
Feb 10 07:01:38 ip- kernel: [1133764.101329] [ 191381]     0 191381   184017     4350   249856        0          -900 snapd
Feb 10 07:01:38 ip- kernel: [1133764.101332] [ 355693]  1000 355693     2164      983    61440        0             0 tmux: server
Feb 10 07:01:38 ip- kernel: [1133764.101334] [ 355694]  1000 355694     2506      872    53248        0             0 bash
Feb 10 07:01:38 ip- kernel: [1133764.101336] [ 611154]     0 611154     3453      978    65536        0             0 sshd
Feb 10 07:01:38 ip- kernel: [1133764.101338] [ 611258]  1000 611258     3574      982    65536        0             0 sshd
Feb 10 07:01:38 ip- kernel: [1133764.101340] [ 611259]  1000 611259     2510      901    53248        0             0 bash
Feb 10 07:01:38 ip- kernel: [1133764.101342] [ 622945]     0 622945     3452      977    65536        0             0 sshd
Feb 10 07:01:38 ip- kernel: [1133764.101343] [ 623052]  1000 623052     3485      819    65536        0             0 sshd
Feb 10 07:01:38 ip- kernel: [1133764.101345] [ 623054]  1000 623054     3038     1483    57344        0             0 bash
Feb 10 07:01:38 ip- kernel: [1133764.101347] [ 629386]  1000 629386   155370    35002   393216        0             0 server
Feb 10 07:01:38 ip- kernel: [1133764.101349] [ 638483]     0 638483      654      434    45056        0             0 apt.systemd.dai
Feb 10 07:01:38 ip- kernel: [1133764.101351] [ 638487]     0 638487      654      434    45056        0             0 apt.systemd.dai
Feb 10 07:01:38 ip- kernel: [1133764.101353] [ 638521]     0 638521    81187    14207   425984        0             0 unattended-upgr
Feb 10 07:01:38 ip- kernel: [1133764.101355] [ 640399]     0 640399    70904      836   151552        0             0 packagekitd
Feb 10 07:01:38 ip- kernel: [1133764.101357] [ 640430]     0 640430    92707    24658   425984        0             0 unattended-upgr

Immediately after this, my process was killed:

Feb 10 07:01:38 ip- kernel: [1133764.101359] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-296.scope,task=server,pid=629386,uid=1000
Feb 10 07:01:38 ip- kernel: [1133764.101385] Out of memory: Killed process 629386 (server) total-vm:621480kB, anon-rss:138304kB, file-rss:1704kB, shmem-rss:0kB, UID:1000 pgtables:384kB oom_score_adj:0

Why was my process killed? It was not the reason that the system ran out of memory. How much of the available memory can I use on a Lightsail server?

gefragt vor einem Jahr50 Aufrufe
Keine Antworten

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen