지난 2~3일 동안 저는 PHP8.1-FPM 프로세스가 밤 12시~오전 12시 30분쯤에 충돌하는 것을 발견했습니다.
내 서버의 트래픽은 아침에 가장 많고 밤에 가장 적습니다. 그러나 FPM 프로세스는 하루 종일 잘 실행되고 밤에는 충돌이 발생합니다. 이것은 매우 이상합니다.
로그에서 찾은 내용은 다음과 같습니다.
1일차
[05-Dec-2022 00:19:47] NOTICE: [pool www] child 10868 started
[05-Dec-2022 00:25:02] NOTICE: [pool www] child 9967 exited with code 0 after 7446.963850 seconds from start
[05-Dec-2022 00:25:02] NOTICE: [pool www] child 10871 started
[05-Dec-2022 00:27:37] NOTICE: [pool www] child 9969 exited with code 0 after 7433.300005 seconds from start
[05-Dec-2022 00:27:37] NOTICE: [pool www] child 10872 started
[05-Dec-2022 00:32:41] WARNING: [pool www] child 10194 exited on signal 9 (SIGKILL) after 5713.957769 seconds from start
[05-Dec-2022 00:32:41] NOTICE: [pool www] child 10876 started
[05-Dec-2022 00:32:41] NOTICE: Terminating ...
[05-Dec-2022 00:32:41] NOTICE: exiting, bye-bye!
2일차
[05-Dec-2022 22:42:26] NOTICE: [pool www] child 23687 started
[05-Dec-2022 22:42:26] NOTICE: [pool www] child 21662 exited with code 0 after 11118.073861 seconds from start
[05-Dec-2022 22:42:26] NOTICE: [pool www] child 23688 started
[05-Dec-2022 22:42:31] NOTICE: [pool www] child 21663 exited with code 0 after 11071.941161 seconds from start
[05-Dec-2022 22:42:31] NOTICE: [pool www] child 23689 started
[05-Dec-2022 22:42:41] NOTICE: [pool www] child 21664 exited with code 0 after 11046.438863 seconds from start
[05-Dec-2022 22:42:41] NOTICE: [pool www] child 23690 started
[05-Dec-2022 23:31:34] NOTICE: [pool www] child 22058 exited with code 0 after 11163.770384 seconds from start
[05-Dec-2022 23:31:34] NOTICE: [pool www] child 24124 started
[06-Dec-2022 00:09:32] WARNING: [pool www] child 23076 exited on signal 9 (SIGKILL) after 8637.063701 seconds from start
[06-Dec-2022 00:09:32] NOTICE: [pool www] child 24580 started
[06-Dec-2022 00:09:32] WARNING: [pool www] child 23080 exited on signal 9 (SIGKILL) after 8607.986722 seconds from start
[06-Dec-2022 00:09:33] NOTICE: [pool www] child 24581 started
[06-Dec-2022 00:09:33] WARNING: [pool www] child 23089 exited on signal 9 (SIGKILL) after 8507.937877 seconds from start
[06-Dec-2022 00:09:33] NOTICE: [pool www] child 24582 started
[06-Dec-2022 00:09:33] WARNING: [pool www] child 23380 exited on signal 9 (SIGKILL) after 6752.142496 seconds from start
[06-Dec-2022 00:09:33] NOTICE: [pool www] child 24583 started
[06-Dec-2022 00:09:33] WARNING: [pool www] child 23403 exited on signal 9 (SIGKILL) after 6643.858307 seconds from start
[06-Dec-2022 00:09:33] NOTICE: [pool www] child 24584 started
[06-Dec-2022 00:09:33] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 147 total children
[06-Dec-2022 00:09:33] NOTICE: Terminating ...
[06-Dec-2022 00:09:33] NOTICE: exiting, bye-bye!
[06-Dec-2022 00:23:49] NOTICE: fpm is running, pid 24611
[06-Dec-2022 00:23:49] NOTICE: ready to handle connections
[06-Dec-2022 00:23:49] NOTICE: systemd monitor interval set to 10000ms
[06-Dec-2022 00:23:53] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 59 idle, and 63 total children
서버 구성 Digitalocean 4 코어 AMD CPU 8GB RAM $54 USD 기본 드롭릿. PHP8.1 FPM 설정
pm = dynamic
pm.max_children = 241
pm.start_servers = 60
pm.min_spare_servers = 60
pm.max_spare_servers = 180
pm.max_requests = 500
-- 업데이트 이 애플리케이션은 이미지 서버입니다. 객체 스토리지의 동적으로 크기가 조정된 이미지 또는 다른 서버의 이미지를 제공합니다. 이미지 크기를 조정하고 제공하는 매우 작은 PHP 스크립트가 있습니다. Nginx는 처리된 이미지를 캐싱하는 데 사용됩니다.
-- dmesg 출력 업데이트
[147949.035475] 0 pages in swap cache
[147949.035475] Swap cache stats: add 0, delete 0, find 0/0
[147949.035476] Free swap = 0kB
[147949.035477] Total swap = 0kB
[147949.035478] 2097014 pages RAM
[147949.035478] 0 pages HighMem/MovableOnly
[147949.035479] 61947 pages reserved
[147949.035480] 0 pages hwpoisoned
[147949.035480] Tasks state (memory values in pages):
[147949.035481] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[147949.035485] [ 380] 0 380 9935 864 106496 0 -250 systemd-journal
[147949.035489] [ 417] 0 417 72328 6775 114688 0 -1000 multipathd
[147949.035496] [ 539] 103 539 22338 415 73728 0 0 systemd-timesyn
[147949.035498] [ 606] 100 606 4029 626 73728 0 0 systemd-network
[147949.035500] [ 625] 101 625 6378 1681 94208 0 0 systemd-resolve
[147949.035502] [ 650] 0 650 2880 1060 69632 0 -1000 systemd-udevd
[147949.035505] [ 747] 0 747 1821 618 57344 0 0 cron
[147949.035506] [ 749] 102 749 2191 891 57344 0 -900 dbus-daemon
[147949.035508] [ 752] 998 752 345261 1828 208896 0 -900 do-agent
[147949.035511] [ 757] 0 757 20699 669 61440 0 0 irqbalance
[147949.035513] [ 758] 0 758 8249 2870 102400 0 0 networkd-dispat
[147949.035515] [ 767] 0 767 58621 410 81920 0 0 polkitd
[147949.035517] [ 768] 104 768 55600 1093 77824 0 0 rsyslogd
[147949.035522] [ 771] 0 771 292597 4332 307200 0 -900 snapd
[147949.035524] [ 776] 0 776 3873 574 69632 0 0 systemd-logind
[147949.035526] [ 778] 0 778 98136 1305 122880 0 0 udisksd
[147949.035529] [ 786] 0 786 79236 954 118784 0 0 ModemManager
[147949.035531] [ 787] 0 787 1554 192 49152 0 0 agetty
[147949.035534] [ 798] 0 798 1543 214 53248 0 0 agetty
[147949.035535] [ 826] 0 826 3855 1169 73728 0 -1000 sshd
[147949.035538] [ 863] 0 863 27523 2822 114688 0 0 unattended-upgr
[147949.035540] [ 904] 0 904 40076 1619 110592 0 0 nginx
[147949.035541] [ 1373] 0 1373 73985 1009 163840 0 0 packagekitd
[147949.035544] [ 1712] 0 1712 343392 174 180224 0 -900 droplet-agent
[147949.035547] [ 12216] 0 12216 53067 2645 151552 0 0 php-fpm8.1
[147949.035549] [ 12743] 33 12743 50015 15472 225280 0 0 nginx
[147949.035552] [ 12744] 33 12744 51002 16135 233472 0 0 nginx
[147949.035554] [ 12745] 33 12745 50614 16019 233472 0 0 nginx
[147949.035556] [ 12746] 33 12746 50765 16062 233472 0 0 nginx
[147949.035558] [ 12747] 33 12747 40214 5403 147456 0 0 nginx
[147949.035560] [ 23025] 33 23025 81250 26391 356352 0 0 php-fpm8.1
[147949.035562] [ 23027] 33 23027 63171 13406 245760 0 0 php-fpm8.1
[147949.035564] [ 23028] 33 23028 66013 15875 266240 0 0 php-fpm8.1
[147949.035566] [ 23029] 33 23029 81755 23040 335872 0 0 php-fpm8.1
[147949.035568] [ 23030] 33 23030 81289 25643 352256 0 0 php-fpm8.1
[147949.035570] [ 23031] 33 23031 65772 15814 266240 0 0 php-fpm8.1
[147949.035572] [ 23032] 33 23032 60680 10956 225280 0 0 php-fpm8.1
[147949.035574] [ 23033] 33 23033 81290 22731 327680 0 0 php-fpm8.1
[147949.035576] [ 23034] 33 23034 79989 22259 327680 0 0 php-fpm8.1
[147949.035578] [ 23035] 33 23035 62802 12404 241664 0 0 php-fpm8.1
[147949.035579] [ 23036] 33 23036 81250 22808 327680 0 0 php-fpm8.1
[147949.035581] [ 23037] 33 23037 81760 23549 331776 0 0 php-fpm8.1
[147949.035584] [ 23038] 33 23038 63141 12398 237568 0 0 php-fpm8.1
[147949.035585] [ 23040] 33 23040 65876 16118 266240 0 0 php-fpm8.1
[147949.035588] [ 23041] 33 23041 54628 5099 176128 0 0 php-fpm8.1
[147949.035593] [ 23042] 33 23042 80737 22410 323584 0 0 php-fpm8.1
[147949.035595] [ 23043] 33 23043 63388 12852 245760 0 0 php-fpm8.1
[147949.035596] [ 23044] 33 23044 79976 26488 364544 0 0 php-fpm8.1
[147949.035599] [ 23045] 33 23045 81353 25677 352256 0 0 php-fpm8.1
[147949.035601] [ 23046] 33 23046 64263 14482 253952 0 0 php-fpm8.1
[147949.035603] [ 23048] 33 23048 81247 28102 368640 0 0 php-fpm8.1
[147949.035604] [ 23049] 33 23049 81763 17517 286720 0 0 php-fpm8.1
[147949.035606] [ 23050] 33 23050 54633 5260 176128 0 0 php-fpm8.1
[147949.035608] [ 23051] 33 23051 81745 27413 368640 0 0 php-fpm8.1
[147949.035610] [ 23052] 33 23052 81250 20292 307200 0 0 php-fpm8.1
[147949.035612] [ 23053] 33 23053 62492 12051 237568 0 0 php-fpm8.1
[147949.035614] [ 23054] 33 23054 79479 19786 311296 0 0 php-fpm8.1
[147949.035616] [ 23055] 33 23055 82274 24380 339968 0 0 php-fpm8.1
[147949.035618] [ 23056] 33 23056 81762 23765 335872 0 0 php-fpm8.1
[147949.035619] [ 23057] 33 23057 60234 10616 221184 0 0 php-fpm8.1
[147949.035624] [ 23058] 33 23058 62266 12114 237568 0 0 php-fpm8.1
[147949.035626] [ 23059] 33 23059 64558 15277 253952 0 0 php-fpm8.1
[147949.035628] [ 23060] 33 23060 81670 17767 286720 0 0 php-fpm8.1
[147949.035630] [ 23061] 33 23061 81740 23569 335872 0 0 php-fpm8.1
[147949.035632] [ 23062] 33 23062 81765 27157 364544 0 0 php-fpm8.1
[147949.035634] [ 23063] 33 23063 55149 5538 180224 0 0 php-fpm8.1
[147949.035636] [ 23064] 33 23064 53607 4475 167936 0 0 php-fpm8.1
[147949.035638] [ 23065] 33 23065 81251 26365 356352 0 0 php-fpm8.1
[147949.035640] [ 23066] 33 23066 79991 19315 303104 0 0 php-fpm8.1
[147949.035642] [ 23067] 33 23067 79803 20677 315392 0 0 php-fpm8.1
[147949.035643] [ 23068] 33 23068 65398 16096 262144 0 0 php-fpm8.1
[147949.035645] [ 23069] 33 23069 62633 12246 233472 0 0 php-fpm8.1
[147949.035647] [ 23070] 33 23070 79262 19484 299008 0 0 php-fpm8.1
[147949.035649] [ 23071] 33 23071 79788 30080 376832 0 0 php-fpm8.1
[147949.035651] [ 23072] 33 23072 62276 12389 237568 0 0 php-fpm8.1
[147949.035653] [ 23073] 33 23073 79974 21148 323584 0 0 php-fpm8.1
[147949.035655] [ 23074] 33 23074 81250 20299 307200 0 0 php-fpm8.1
[147949.035656] [ 23075] 33 23075 63702 13680 249856 0 0 php-fpm8.1
[147949.035658] [ 23077] 33 23077 80732 17341 282624 0 0 php-fpm8.1
[147949.035660] [ 23078] 33 23078 79772 20834 303104 0 0 php-fpm8.1
[147949.035662] [ 23079] 33 23079 65346 15686 262144 0 0 php-fpm8.1
[147949.035664] [ 23081] 33 23081 63206 13763 245760 0 0 php-fpm8.1
[147949.035666] [ 23082] 33 23082 81761 24195 339968 0 0 php-fpm8.1
[147949.035667] [ 23083] 33 23083 64690 15490 258048 0 0 php-fpm8.1
[147949.035669] [ 23084] 33 23084 54811 5039 176128 0 0 php-fpm8.1
[147949.035674] [ 23085] 33 23085 63144 12510 237568 0 0 php-fpm8.1
[147949.035676] [ 23086] 33 23086 62481 12562 237568 0 0 php-fpm8.1
[147949.035680] [ 23087] 33 23087 65662 15683 262144 0 0 php-fpm8.1
[147949.035682] [ 23088] 33 23088 81757 27232 364544 0 0 php-fpm8.1
[147949.035684] [ 23090] 33 23090 81762 18209 290816 0 0 php-fpm8.1
[147949.035686] [ 23091] 33 23091 65152 15280 258048 0 0 php-fpm8.1
[147949.035688] [ 23092] 33 23092 79988 21290 327680 0 0 php-fpm8.1
[147949.035690] [ 23171] 33 23171 81760 18679 294912 0 0 php-fpm8.1
[147949.035692] [ 23173] 33 23173 79410 17029 286720 0 0 php-fpm8.1
[147949.035694] [ 23175] 33 23175 65824 16263 266240 0 0 php-fpm8.1
[147949.035696] [ 23185] 33 23185 65365 16180 262144 0 0 php-fpm8.1
[147949.035698] [ 23186] 33 23186 65231 15066 262144 0 0 php-fpm8.1
[147949.035699] [ 23187] 33 23187 79988 20002 315392 0 0 php-fpm8.1
[147949.035701] [ 23375] 33 23375 68209 18499 282624 0 0 php-fpm8.1
[147949.035703] [ 23378] 33 23378 60998 11722 225280 0 0 php-fpm8.1
[147949.035705] [ 23379] 33 23379 79435 18467 286720 0 0 php-fpm8.1
[147949.035707] [ 23388] 33 23388 79258 21188 319488 0 0 php-fpm8.1
[147949.035709] [ 23389] 33 23389 79453 20039 311296 0 0 php-fpm8.1
[147949.035711] [ 23390] 33 23390 61517 11708 229376 0 0 php-fpm8.1
[147949.035713] [ 23391] 33 23391 55516 5670 184320 0 0 php-fpm8.1
[147949.035715] [ 23392] 33 23392 63335 12141 237568 0 0 php-fpm8.1
[147949.035717] [ 23393] 33 23393 81245 19769 303104 0 0 php-fpm8.1
[147949.035721] [ 23394] 33 23394 80731 18069 290816 0 0 php-fpm8.1
[147949.035723] [ 23395] 33 23395 81229 17634 290816 0 0 php-fpm8.1
[147949.035725] [ 23396] 33 23396 62102 11782 233472 0 0 php-fpm8.1
[147949.035727] [ 23397] 33 23397 61251 11551 229376 0 0 php-fpm8.1
[147949.035729] [ 23398] 33 23398 65025 15476 258048 0 0 php-fpm8.1
[147949.035731] [ 23399] 33 23399 56609 6542 192512 0 0 php-fpm8.1
[147949.035733] [ 23400] 33 23400 61918 11956 233472 0 0 php-fpm8.1
[147949.035735] [ 23401] 33 23401 78978 25135 352256 0 0 php-fpm8.1
[147949.035736] [ 23402] 33 23402 61182 11420 229376 0 0 php-fpm8.1
[147949.035738] [ 23404] 33 23404 65174 15748 262144 0 0 php-fpm8.1
[147949.035740] [ 23405] 33 23405 81247 22004 319488 0 0 php-fpm8.1
[147949.035742] [ 23406] 33 23406 53598 4420 167936 0 0 php-fpm8.1
[147949.035744] [ 23407] 33 23407 61984 11988 233472 0 0 php-fpm8.1
[147949.035746] [ 23408] 33 23408 63977 14485 249856 0 0 php-fpm8.1
[147949.035748] [ 23409] 33 23409 81247 25246 348160 0 0 php-fpm8.1
[147949.035749] [ 23410] 33 23410 64153 14631 253952 0 0 php-fpm8.1
[147949.035751] [ 23411] 33 23411 81279 23444 331776 0 0 php-fpm8.1
[147949.035753] [ 23412] 33 23412 65221 15610 262144 0 0 php-fpm8.1
[147949.035755] [ 23413] 33 23413 81753 27004 360448 0 0 php-fpm8.1
[147949.035757] [ 23414] 33 23414 63471 14123 245760 0 0 php-fpm8.1
[147949.035759] [ 23415] 33 23415 63396 14222 245760 0 0 php-fpm8.1
[147949.035761] [ 23467] 33 23467 80729 15421 266240 0 0 php-fpm8.1
[147949.035763] [ 23469] 33 23469 65609 15655 262144 0 0 php-fpm8.1
[147949.035767] [ 23470] 33 23470 63897 14733 249856 0 0 php-fpm8.1
[147949.035769] [ 23472] 33 23472 81239 22790 327680 0 0 php-fpm8.1
[147949.035771] [ 23473] 33 23473 62872 13170 241664 0 0 php-fpm8.1
[147949.035773] [ 23474] 33 23474 64835 15560 258048 0 0 php-fpm8.1
[147949.035775] [ 23475] 33 23475 65215 15739 262144 0 0 php-fpm8.1
[147949.035777] [ 23476] 33 23476 63137 12595 237568 0 0 php-fpm8.1
[147949.035778] [ 23477] 33 23477 78749 17139 282624 0 0 php-fpm8.1
[147949.035780] [ 23478] 33 23478 81236 22468 323584 0 0 php-fpm8.1
[147949.035783] [ 23479] 33 23479 81238 22144 319488 0 0 php-fpm8.1
[147949.035784] [ 23480] 33 23480 61533 12097 229376 0 0 php-fpm8.1
[147949.035786] [ 23481] 33 23481 61742 12461 233472 0 0 php-fpm8.1
[147949.035788] [ 23668] 33 23668 81120 27387 364544 0 0 php-fpm8.1
[147949.035790] [ 23669] 33 23669 81281 25762 352256 0 0 php-fpm8.1
[147949.035792] [ 23670] 33 23670 80711 22075 323584 0 0 php-fpm8.1
[147949.035794] [ 23671] 33 23671 81629 21215 311296 0 0 php-fpm8.1
[147949.035796] [ 23672] 33 23672 54106 4783 172032 0 0 php-fpm8.1
[147949.035798] [ 23673] 33 23673 80732 20166 303104 0 0 php-fpm8.1
[147949.035800] [ 23674] 33 23674 62566 13115 237568 0 0 php-fpm8.1
[147949.035802] [ 23675] 33 23675 62266 11933 237568 0 0 php-fpm8.1
[147949.035804] [ 23676] 33 23676 62778 12260 237568 0 0 php-fpm8.1
[147949.035805] [ 23677] 33 23677 93296 27104 364544 0 0 php-fpm8.1
[147949.035807] [ 23678] 33 23678 79960 26685 364544 0 0 php-fpm8.1
[147949.035809] [ 23679] 33 23679 79235 17993 286720 0 0 php-fpm8.1
[147949.035814] [ 23680] 33 23680 63334 13892 245760 0 0 php-fpm8.1
[147949.035816] [ 23681] 33 23681 82264 22326 323584 0 0 php-fpm8.1
[147949.035817] [ 23682] 33 23682 62045 12131 233472 0 0 php-fpm8.1
[147949.035819] [ 23683] 33 23683 81649 24578 344064 0 0 php-fpm8.1
[147949.035821] [ 23684] 33 23684 81755 25202 348160 0 0 php-fpm8.1
[147949.035823] [ 23685] 33 23685 56617 6875 192512 0 0 php-fpm8.1
[147949.035825] [ 23686] 33 23686 55403 5763 180224 0 0 php-fpm8.1
[147949.035827] [ 23687] 33 23687 78956 21007 319488 0 0 php-fpm8.1
[147949.035829] [ 23688] 33 23688 58896 9008 208896 0 0 php-fpm8.1
[147949.035830] [ 23689] 33 23689 61751 11976 233472 0 0 php-fpm8.1
[147949.035832] [ 23690] 33 23690 79253 29982 372736 0 0 php-fpm8.1
[147949.035834] [ 23803] 0 23803 4285 846 73728 0 0 sshd
[147949.035836] [ 23806] 0 23806 4260 971 73728 0 0 systemd
[147949.035838] [ 23807] 0 23807 42651 1255 90112 0 0 (sd-pam)
[147949.035840] [ 23863] 0 23863 2287 943 53248 0 0 bash
[147949.035842] [ 23878] 0 23878 2152 868 53248 0 0 htop
[147949.035843] [ 24124] 33 24124 63362 13980 245760 0 0 php-fpm8.1
[147949.035846] [ 24573] 33 24573 54409 4318 176128 0 0 php-fpm8.1
[147949.035848] [ 24574] 0 24574 53067 1478 126976 0 0 php-fpm8.1
[147949.035850] [ 24575] 0 24575 53150 1663 126976 0 0 php-fpm8.1
[147949.035854] [ 24576] 0 24576 53067 1235 122880 0 0 php-fpm8.1
[147949.035856] [ 24577] 0 24577 53067 1589 122880 0 0 php-fpm8.1
[147949.035857] [ 24578] 0 24578 53067 1591 122880 0 0 php-fpm8.1
[147949.035859] [ 24579] 0 24579 53067 1572 122880 0 0 php-fpm8.1
[147949.035861] [ 24580] 0 24580 53067 1615 122880 0 0 php-fpm8.1
[147949.035863] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=systemd-resolved.service,mems_allowed=0,global_oom,task_memcg=/system.slice/php8.1-fpm.service,task=php-fpm8.1,pid=23071,uid=33
[147949.035882] Out of memory: Killed process 23071 (php-fpm8.1) total-vm:319152kB, anon-rss:112388kB, file-rss:1824kB, shmem-rss:6108kB, UID:33 pgtables:368kB oom_score_adj:0
답변1
이 시점에서 서버의 메모리가 부족해졌습니다.
오래 지속되고 메모리 집약적인 여러 작업을 시작하는 예약된 작업이 있을 수 있습니다. 그러면 Linux 커널 메모리 부족 킬러가 실행되어 대부분의 메모리를 소비하는 프로세스가 종료됩니다.
메모리나 스왑 공간을 늘려야 합니다. 또는 애플리케이션의 메모리 사용량을 조사하여 줄일 수 있는지 확인하세요.