Archives par mot-clé : cuda

Llama.cpp : Tuxedo 17 (en Ubuntu 24) + GIGABYTE AORUS RTX 5060 Ti AI Box .

Publié le par
Répondre

Sur le Tuxedo 17 ( https://www.tuxedocomputers.com/ ) j’ai ajouté via le Thunderbold un « GIGABYTE AORUS RTX 5060 Ti AI Box ».
Mon installation :
– Ubuntu : 24.04
– CUDA : 13.3
– NVIDIA : 610.43.03

# lsb_release -a
No LSB modules are available.
Distributor ID: Tuxedo
Description:    TUXEDO OS
Release:        24.04
Codename:       noble
# nvidia-smi 
Thu Jul 16 17:29:22 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 610.43.03              KMD Version: 610.43.03     CUDA UMD Version: 13.3     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3060 ...    Off |   00000000:01:00.0 Off |                  N/A |
| N/A   46C    P8             11W /  115W |       1MiB /   6144MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 5060 Ti     Off |   00000000:05:00.0 Off |                  N/A |
|  0%   34C    P0             15W /  180W |       2MiB /  16311MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

Ma version de Thunderbold est 4 :

# lspci | grep -i thunderbolt
00:07.0 PCI bridge: Intel Corporation Tiger Lake-H Thunderbolt 4 PCI Express Root Port #0 (rev 05)
00:0d.0 USB controller: Intel Corporation Tiger Lake-H Thunderbolt 4 USB Controller (rev 05)
00:0d.2 USB controller: Intel Corporation Tiger Lake-H Thunderbolt 4 NHI #0 (rev 05)

A noter que pour le build de llama.cpp j’ai ajouté une directive de compilation : CMAKE_CUDA_ARCHITECTURES qui a pour valeur par défaut 75 donc 7.5.

# nvidia-smi --query-gpu=compute_cap --format=csv
compute_cap
8.6
12.0
...
# cmake -B build -DGGML_CUDA=ON  -DCMAKE_CUDA_COMPILER=`which nvcc` -DLLAMA_CURL=ON -DCMAKE_CUDA_ARCHITECTURES=86
...

J’ai d’abord essayé le soft de Gigabyte : https://www.gigabyte.com/Support/Utility/Graphics-Card . Mais il n’est pas fonctionnel.

Puis j’ai essayé une utilisation normale, et j’avais des crash sans arrêts.Avec des erreurs du type :

... nvidia-modeset: ERROR: GPU:1: Error while waiting for GPU progress: ...

Ensuite je suis tombé sur ce site : https://github.com/mmhorda/ai-box-rtx-5060ti-egpu-guide , et j’ai compris qu’il y avait un problème de débit.

En fait il y a un bug ouvert sur le sujet : https://github.com/NVIDIA/open-gpu-kernel-modules/issues/979#issuecomment-4172636691 : « RTX 5080 via Thunderbolt 5 eGPU: Hard lock on CUDA operations (nvidia-smi works at idle) « .

J’ai finalement trouvé un projet avec un fix pour Ubuntu : https://github.com/lokmantsui/aorus-5090-egpu/tree/nvidia-610.43.02-ubuntu-5060ti

# git clone https://github.com/lokmantsui/aorus-5090-egpu.git -b nvidia-610.43.02-ubuntu-5060ti

J’ai du faire des modifications car il détectait pas la bonne carte NVIDIA. Et finalement c’est tombé en marche.

Pour mon installation la commande grup ressemble à ceci :

GRUB_CMDLINE_LINUX="iommu.passthrough=1 thunderbolt.host_reset=false pcie_aspm.policy=performance thunderbolt.clx=0 pcie_port_pm=off pci=resource_alignment=35@0000:00:05.0"

J’ai donc fait mes premiers tests avec gemma-3-1b-it-q4_k_m.gguf :

# llama-bench --list-devices 
ggml_cuda_init: found 2 CUDA devices (Total VRAM: 21692 MiB):
  Device 0: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes, VRAM: 15888 MiB
  Device 1: NVIDIA GeForce RTX 3060 Laptop GPU, compute capability 8.6, VMM: yes, VRAM: 5803 MiB
Available devices:
  CUDA0: NVIDIA GeForce RTX 5060 Ti (15888 MiB, 15752 MiB free)
  CUDA1: NVIDIA GeForce RTX 3060 Laptop GPU (5803 MiB, 5685 MiB free)

# llama-bench -m /models/gemma-3-1b-it-q4_k_m.gguf -dev CUDA0,CUDA1
ggml_cuda_init: found 2 CUDA devices (Total VRAM: 21692 MiB):
  Device 0: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes, VRAM: 15888 MiB
  Device 1: NVIDIA GeForce RTX 3060 Laptop GPU, compute capability 8.6, VMM: yes, VRAM: 5803 MiB
| model                          |       size |     params | backend    | ngl | dev          |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------ | --------------: | -------------------: |
| gemma3 1B Q4_K - Medium        | 762.49 MiB |   999.89 M | CUDA       |  -1 | CUDA0        |           pp512 |   19994.19 ± 1751.94 |
| gemma3 1B Q4_K - Medium        | 762.49 MiB |   999.89 M | CUDA       |  -1 | CUDA0        |           tg128 |        268.49 ± 0.38 |
| gemma3 1B Q4_K - Medium        | 762.49 MiB |   999.89 M | CUDA       |  -1 | CUDA1        |           pp512 |    11158.41 ± 527.54 |
| gemma3 1B Q4_K - Medium        | 762.49 MiB |   999.89 M | CUDA       |  -1 | CUDA1        |           tg128 |        229.10 ± 0.37 |

build: c3d47e696 (10030)
# llama-bench -m /models/gemma-3-1b-it-q4_k_m.gguf 
ggml_cuda_init: found 2 CUDA devices (Total VRAM: 21692 MiB):
  Device 0: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes, VRAM: 15888 MiB
  Device 1: NVIDIA GeForce RTX 3060 Laptop GPU, compute capability 8.6, VMM: yes, VRAM: 5803 MiB
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| gemma3 1B Q4_K - Medium        | 762.49 MiB |   999.89 M | CUDA       |  -1 |           pp512 |   16582.32 ± 1005.41 |
| gemma3 1B Q4_K - Medium        | 762.49 MiB |   999.89 M | CUDA       |  -1 |           tg128 |        261.64 ± 1.81 |

build: c3d47e696 (10030)

Et donc il est préférable d’utiliser la carte la plus puissante et non pas les deux cartes en même temps.

J’ai refais un test avec un modèle plus gros : Qwen3.6-27B-Q3_K_M.gguf

# llama-bench -m /models/Qwen3.6-27B-Q3_K_M.gguf 
ggml_cuda_init: found 2 CUDA devices (Total VRAM: 21692 MiB):
  Device 0: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes, VRAM: 15888 MiB
  Device 1: NVIDIA GeForce RTX 3060 Laptop GPU, compute capability 8.6, VMM: yes, VRAM: 5803 MiB
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen35 27B Q3_K - Medium       |  12.64 GiB |    26.90 B | CUDA       |  -1 |           pp512 |        707.38 ± 5.11 |
| qwen35 27B Q3_K - Medium       |  12.64 GiB |    26.90 B | CUDA       |  -1 |           tg128 |         18.31 ± 0.01 |

build: c3d47e696 (10030)

# llama-bench -m /models/Qwen3.6-27B-Q3_K_M.gguf -dev CUDA0
ggml_cuda_init: found 2 CUDA devices (Total VRAM: 21692 MiB):
  Device 0: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes, VRAM: 15888 MiB
  Device 1: NVIDIA GeForce RTX 3060 Laptop GPU, compute capability 8.6, VMM: yes, VRAM: 5803 MiB
| model                          |       size |     params | backend    | ngl | dev          |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------ | --------------: | -------------------: |
| qwen35 27B Q3_K - Medium       |  12.64 GiB |    26.90 B | CUDA       |  -1 | CUDA0        |           pp512 |        841.06 ± 9.58 |
| qwen35 27B Q3_K - Medium       |  12.64 GiB |    26.90 B | CUDA       |  -1 | CUDA0        |           tg128 |         20.50 ± 0.02 |

build: c3d47e696 (10030)

Prochaine étape, l’ajout d’Hermes Agent IA.

Ollama n’utilise pas le GPU de la carte … Misère

En passant

Répondre

Ma version d’OS est « Ubuntu 22.04.5 LTS« .

Ma version de carte/drivers NVIDIA :

# uname -a
Linux 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

# nvidia-smi -L
GPU 0: Quadro 4000 (UUID: GPU-13797e5d-a72f-4c72-609f-686fa4a8c956)

# nvidia-smi 
Mon Jan 20 16:41:52 2025       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.157                Driver Version: 390.157                   |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro 4000         Off  | 00000000:00:10.0 Off |                  N/A |
| 36%   62C   P12    N/A /  N/A |      1MiB /  1985MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

# cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module  390.157  Wed Oct 12 09:19:07 UTC 2022
GCC version:  gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)

# nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Tue_Oct_29_23:50:19_PDT_2024
Cuda compilation tools, release 12.6, V12.6.85
Build cuda_12.6.r12.6/compiler.35059454_0

# ubuntu-drivers devices
== /sys/devices/pci0000:00/0000:00:10.0 ==
modalias : pci:v000010DEd000006DDsv000010DEsd00000780bc03sc00i00
vendor   : NVIDIA Corporation
model    : GF100GL [Quadro 4000]
driver   : nvidia-driver-390 - distro non-free recommended
driver   : xserver-xorg-video-nouveau - distro free builtin

Journal de ollama :

# journalctl -u ollama -f --no-pager
ollama[2974]: llama_kv_cache_init: kv_size = 8192, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 32, can_shift = 1
ollama[2974]: llama_kv_cache_init:        CPU KV buffer size =  1024.00 MiB
ollama[2974]: llama_new_context_with_model: KV self size  = 1024.00 MiB, K (f16):  512.00 MiB, V (f16):  512.00 MiB
ollama[2974]: llama_new_context_with_model:        CPU  output buffer size =     0.56 MiB
ollama[2974]: llama_new_context_with_model:        CPU compute buffer size =   560.01 MiB
ollama[2974]: llama_new_context_with_model: graph nodes  = 1030
ollama[2974]: llama_new_context_with_model: graph splits = 1
ollama[2974]: time=2025-01-20T16:30:12.070Z level=INFO source=server.go:594 msg="llama runner started in 4.28 seconds"

Quand je fais l’installation j’ai bien « NVIDIA GPU installed »

# curl -fsSL https://ollama.com/install.sh | shsh
>>> Cleaning up old version at /usr/local/lib/ollama
>>> Installing ollama to /usr/local
>>> Downloading Linux amd64 bundle
######################################################################## 100,0%
>>> Adding ollama user to render group...
>>> Adding ollama user to video group...
>>> Adding current user to ollama group...
>>> Creating ollama systemd service...
>>> Enabling and starting ollama service...
>>> NVIDIA GPU installed.

J’ai un problème de chargement

ollama[3917]: time=2025-01-20T16:52:30.680Z level=INFO source=routes.go:1238 msg="Listening on 127.0.0.1:11434 (version 0.5.7)"
ollama[3917]: time=2025-01-20T16:52:30.681Z level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners="[cuda_v12_avx rocm_avx cpu cpu_avx cpu_avx2 cuda_v11_avx]"
ollama[3917]: time=2025-01-20T16:52:30.681Z level=INFO source=gpu.go:226 msg="looking for compatible GPUs"
ollama[3917]: time=2025-01-20T16:52:30.702Z level=INFO source=gpu.go:630 msg="Unable to load cudart library /usr/lib/x86_64-linux-gnu/libcuda.so.390.157: symbol lookup for cuDeviceGetUuid failed: /usr/lib/x86_64-linux-gnu/libcuda.so.390.157: undefined symbol: cuDeviceGetUuid"
ollama[3917]: time=2025-01-20T16:52:30.741Z level=INFO source=gpu.go:392 msg="no compatible GPUs were discovered"
ollama[3917]: time=2025-01-20T16:52:30.742Z level=INFO source=types.go:131 msg="inference compute" id=0 library=cpu variant=avx compute="" driver=0.0 name="" total="61.4 GiB" available="59.4 GiB"

A suivre …