Unix SORT 不尊重其記憶體限制

Unix SORT 不尊重其記憶體限制

我面臨著由於過度使用內存而被殺死的問題sort,並且無法理解為什麼它大大超出了給定的限制。

Context: 
Available physical RAM: 8GB 
Memory in use by system: <500MB (FreeBSD 13) 
Free memory: ~7.5GB 
Swap: 2G
No system hard limit on mem usage ($ ulimit -> unlimited)
File being sorted: 300+ GB 
HDD space: 1Tb+ free both on target or temporary dir.

即使當我運行排序時指定可用資源限制;或沒有任何明確參數(應自動設定為可用的 90%),它最終會因過度使用而被殺死。

[mario@casa-server ~/pjy]$ sort -S 7G -T /filesWD-WMAY03648941/tmp/ -o /filesWD-WMAY03648941/pjA-sorted.txt pjyA.txt

Killed
[mario@casa-server ~/pjy]$
[mario@casa-server ~/pjy]$ sort -T /filesWD-WMAY03648941/tmp/ -o /filesWD-WMAY03648941/pjA-sorted.txt pjyA.txt

Killed
[mario@casa-server ~/pjy]$

顯然它確實超出了資源限制。當使用 -S 3G 運行時,我有時可以觀察到大約 5G 的使用量。

# ~5G USAGE EVEN WITH EXPLICIT -S 3G LIMIT


  [mario@casa-server ~]$ sudo procstat -r 3183
  PID COMM             RESOURCE                          VALUE        
 3183 sort             user time                    00:27:51.849171   
 3183 sort             system time                  00:03:28.220262   
 3183 sort             maximum RSS                          5423072 KB
 3183 sort             integral shared memory              10505728 KB
 3183 sort             integral unshared data               1910064 KB
 3183 sort             integral unshared stack             30560128 KB
 3183 sort             page reclaims                        8172561   
 3183 sort             page faults                             1724   
 3183 sort             swaps                                      0   
 3183 sort             block reads                            24296   
 3183 sort             block writes                           17719   
 3183 sort             messages sent                              0   
 3183 sort             messages received                          0   
 3183 sort             signals received                           0   
 3183 sort             voluntary context switches              5868   
 3183 sort             involuntary context switches           30700   
[mario@casa-server ~]$ 


# TOP excerpt
 PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
 3183 mario         1 103    0  4657M  4603M CPU3     3  38:23  99.88% sort -S 3G -T /filesWD-WMAY03648941/tmp/ -o /files



# NO SO LIMIT ON USAGE
[mario@casa-server ~]$ sudo procstat -l 3183
  PID COMM             RLIMIT                  SOFT             HARD     
 3183 sort             cputime             infinity         infinity     
 3183 sort             filesize            infinity         infinity     
 3183 sort             datasize               32768 MB         32768 MB  
 3183 sort             stacksize             524288 KB        524288 KB  
 3183 sort             coredumpsize        infinity         infinity     
 3183 sort             memoryuse           infinity         infinity     
 3183 sort             memorylocked           65536 B          65536 B   
 3183 sort             maxprocesses           12186            12186     
 3183 sort             openfiles             234576           234576     
 3183 sort             sbsize              infinity         infinity     
 3183 sort             vmemoryuse          infinity         infinity     
 3183 sort             pseudo-terminals    infinity         infinity     
 3183 sort             swapuse             infinity         infinity     
 3183 sort             kqueues             infinity         infinity     
 3183 sort             umtxp               infinity         infinity     
[mario@casa-server ~]$
[mario@casa-server ~]$ ulimit 
unlimited

這就解釋了為什麼它在以 7G 限制(7.5 可用)運行時被殺死,或者當限制被自動設定時:它確實超過了使用所有 mem+swap(並且按預期被操作系統殺死)。

但我在這裡缺少什麼?man sort

 -S size, --buffer-size=size
         Use size for the maximum size of the memory buffer.  Size
         modifiers %,b,K,M,G,T,P,E,Z,Y can be used.  If a memory limit is
         not explicitly specified, sort takes up to about 90% of available
         memory.  If the file size is too big to fit into the memory
         buffer, the temporary disk files are used to perform the sorting.

如果-S 3G設定了,為什麼會超過使用RAM而不是HDD臨時資料夾?

PS:我試著問這個與其他「因記憶體不足而被殺」相關的問題,但有些版主管理不善。我確實認為這是一個相關的問題(將其標記為不相關的問題只會使解決方案支離破碎)。另外,我閱讀了許多其他可能的解決方案,但它們不適合這種情況。

謝謝大家

相關內容