我在 Python 的 nvidia GPU 上遇到記憶體分配速度極慢的問題。
當在新的 Python 會話中運行 GPU 計算時，tensorflow/pytorch 會以微小的增量分配內存，持續大約四分鐘，直到它突然分配一大塊內存並執行實際計算。所有後續計算均立即執行。
有人知道可能出了什麼問題？或如何取得記憶體分配期間實際發生的情況的日誌？
我嘗試重新安裝 CUDA 庫和 nvidia 驅動程式。重新安裝驅動程式可以暫時解決該問題，然後記憶體分配再次掛起。
Python輸出：
Python 3.11.3 (main, Apr 5 2023, 14:15:06) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import timeit
>>> timeit.timeit('import tensorflow as tf;tf.random.uniform([10])', number=1)
2023-04-17 09:08:24.062130: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-04-17 09:08:24.641429: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-04-17 09:12:12.879503: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 21368 MB memory: -> device: 0, name: GRID RTX6000-24Q, pci bus id: 0000:02:02.0, compute capability: 7.5
229.68861908599501
英偉達-SMI：
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.182.03 Driver Version: 470.182.03 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GRID RTX6000-24Q On | 00000000:02:02.0 Off | N/A |
| N/A N/A P8 N/A / N/A | 23527MiB / 24576MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 122079 C ...Model-js4zUkog/bin/python 21743MiB |
+-----------------------------------------------------------------------------+
NVCC：
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0
答案1
我發現記憶體分配緩慢是由於 nvidia 限制我的 GPU 造成的，因為無法驗證許可證。
我檢查了：
sudo cat /var/log/syslog | grep nvidia
並發現：
Apr 18 11:35:43 srv-apu102 nvidia-gridd: Valid GRID license not found. GPU features and performance are restricted. To enable full functionality please configure licensing details. Apr 18 11:42:32 srv-apu102 nvidia-gridd: Acquiring license. (Info: http://10.1.2.56:7070/request; NVIDIA RTX Virtual Workstation) Apr 18 11:42:32 srv-apu102 nvidia-gridd: Calling load_byte_array(tra) Apr 18 11:42:35 srv-apu102 nvidia-gridd: Error: Failed server communication. Server URL : http://10.1.2.56:7070/request - #012[1,7e2,2,0[74000008,7,110001f3]] Generic communications error.#012[1,7e2,2,0[75000001,7,30010255]] General data transfer failure. Couldn't connect to server
我希望這對其他人有幫助，儘管那些似乎認為我的問題是關於 Ubuntu Lunar 的人投了反對票…