.png)
Procurei por um tempo e não consigo encontrar resposta ou mesmo direções para seguir em frente.
Então. Cluster XCP-NG de três servidores HP DL360p G8, MSA 2060 iSCSI NAS com 12 drives SAS 10K, QNAP TS-1273U-RP, switch Mikrotik CRS317. A rede de armazenamento é em bridge dedicada no mikrotik. Todos os dispositivos conectados com cabos de cobre de 3 metros. Todos os dispositivos mostram que o link é 10G. Até configurei MTU para 9000 para todos os dispositivos. Cada servidor possui placa Ethernet com duas interfaces. Um é usado apenas para rede de armazenamento (eth1 em todos os três servidores). Sub-rede diferente para rede de armazenamento e rede de gerenciamento. O back-end da rede Xen é openvswitch.
Os quadros Jumbo estão funcionando:
ping -M do -s 8972 -c 2 10.100.200.10 -- QNAP
PING 10.100.200.10 (10.100.200.10) 8972(9000) bytes of data.
8980 bytes from 10.100.200.10: icmp_seq=1 ttl=64 time=1.01 ms
8980 bytes from 10.100.200.10: icmp_seq=2 ttl=64 time=0.349 ms
--- 10.100.200.10 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.349/0.682/1.015/0.333 ms
ping -M do -s 8972 -c 2 10.100.200.8 -- MSA 2060
PING 10.100.200.8 (10.100.200.8) 8972(9000) bytes of data.
8980 bytes from 10.100.200.8: icmp_seq=1 ttl=64 time=9.83 ms
8980 bytes from 10.100.200.8: icmp_seq=2 ttl=64 time=0.215 ms
--- 10.100.200.8 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.215/5.023/9.832/4.809 ms
Problema: quando copio a máquina virtual de um armazenamento (QNAP) para outro (MSA), a velocidade de gravação é de cerca de 45 MB/s. Quando copio algum arquivo grande do QNAP (ex: instalação iso) para o armazenamento local dos servidores, a velocidade é de cerca de 100 MB/s e nesse servidor htopmostra um núcleo com 100% de carga
É claramente visível que a rede está funcionando como uma rede 1G.
Algumas informações sobre hardware.
ethtool -i eth1
driver: ixgbe
version: 5.5.2
firmware-version: 0x18b30001
expansion-rom-version:
bus-info: 0000:07:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
ethtool eth1
Settings for eth1:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: 10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
lspci | grep net
07:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
07:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
Então executei o servidor iper3 neste host: iperf3 -s -4
Resultados no host do servidor:
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.04 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-10.04 sec 5.48 GBytes 4.69 Gbits/sec receiver
[ 7] 0.00-10.04 sec 0.00 Bytes 0.00 bits/sec sender
[ 7] 0.00-10.04 sec 5.44 GBytes 4.66 Gbits/sec receiver
[SUM] 0.00-10.04 sec 0.00 Bytes 0.00 bits/sec sender
[SUM] 0.00-10.04 sec 10.9 GBytes 9.35 Gbits/sec receiver
E cliente em outro host: iperf3 -c 10.100.200.20 -P 2 -t 10 -4
Resultados no host cliente:
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 5.49 GBytes 4.72 Gbits/sec 112 sender
[ 4] 0.00-10.00 sec 5.48 GBytes 4.71 Gbits/sec receiver
[ 6] 0.00-10.00 sec 5.45 GBytes 4.68 Gbits/sec 178 sender
[ 6] 0.00-10.00 sec 5.44 GBytes 4.67 Gbits/sec receiver
[SUM] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec 290 sender
[SUM] 0.00-10.00 sec 10.9 GBytes 9.38 Gbits/sec receiver
O que testar a seguir ou como encontrar gargalos?
iperf3 mostra que o link está funcionando com velocidade de 10 Gbit ou interpretei os resultados incorretamente?
Versões de software:
xe host-list params=software-version
software-version (MRO) : product_version: 8.2.0; product_version_text: 8.2; product_version_text_short: 8.2; platform_name: XCP; platform_version: 3.2.0; product_brand: XCP-ng; build_number: release/stockholm/master/7; hostname: localhost; date: 2021-05-20; dbv: 0.0.1; xapi: 1.20; xen: 4.13.1-9.11.1; linux: 4.19.0+1; xencenter_min: 2.16; xencenter_max: 2.16; network_backend: openvswitch; db_schema: 5.602
software-version (MRO) : product_version: 8.2.0; product_version_text: 8.2; product_version_text_short: 8.2; platform_name: XCP; platform_version: 3.2.0; product_brand: XCP-ng; build_number: release/stockholm/master/7; hostname: localhost; date: 2021-05-20; dbv: 0.0.1; xapi: 1.20; xen: 4.13.1-9.11.1; linux: 4.19.0+1; xencenter_min: 2.16; xencenter_max: 2.16; network_backend: openvswitch; db_schema: 5.602
software-version (MRO) : product_version: 8.2.0; product_version_text: 8.2; product_version_text_short: 8.2; platform_name: XCP; platform_version: 3.2.0; product_brand: XCP-ng; build_number: release/stockholm/master/7; hostname: localhost; date: 2021-05-20; dbv: 0.0.1; xapi: 1.20; xen: 4.13.1-9.11.1; linux: 4.19.0+1; xencenter_min: 2.16; xencenter_max: 2.16; network_backend: openvswitch; db_schema: 5.602
Outros dois servidores possuem placas HP 530FLR-SFP+:
lspci | grep net
03:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
03:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
ethtool -i eth1
driver: bnx2x
version: 1.714.24 storm 7.13.11.0
firmware-version: bc 7.10.10
expansion-rom-version:
bus-info: 0000:03:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
ethtool eth1
Settings for eth1:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseT/Full
10000baseT/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: 10000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Port: Direct Attach Copper
PHYAD: 1
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: g
Wake-on: g
Current message level: 0x00000000 (0)
Link detected: yes
Edição 1: teste de armazenamento local:
dmesg | grep sda
[ 13.093002] sd 0:1:0:0: [sda] 860051248 512-byte logical blocks: (440 GB/410 GiB)
[ 13.093077] sd 0:1:0:0: [sda] Write Protect is off
[ 13.093080] sd 0:1:0:0: [sda] Mode Sense: 73 00 00 08
[ 13.093232] sd 0:1:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 13.112781] sda: sda1 sda2 sda3 sda4 sda5 sda6
[ 13.114348] sd 0:1:0:0: [sda] Attached SCSI disk
[ 15.267456] EXT4-fs (sda1): mounting ext3 file system using the ext4 subsystem
[ 15.268750] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
[ 17.597243] EXT4-fs (sda1): re-mounted. Opts: (null)
[ 18.991998] Adding 1048572k swap on /dev/sda6. Priority:-2 extents:1 across:1048572k
[ 19.279706] EXT4-fs (sda5): mounting ext3 file system using the ext4 subsystem
[ 19.281346] EXT4-fs (sda5): mounted filesystem with ordered data mode. Opts: (null)
dd if=/dev/sda of=/dev/null bs=1024 count=1000000
1000000+0 records in
1000000+0 records out
1024000000 bytes (1.0 GB) copied, 11.1072 s, 92.2 MB/s
Isso é estranho, porque o servidor possui controlador Smart Array P420i com cache de 2 GB, hardware raid10 de 6 unidades SAS de 146 GB e 15k. iLo mostra que com armazenamento está tudo bem. Em outro servidor os resultados são semelhantes1024000000 bytes (1.0 GB) copied, 11.8031 s, 86.8 MB/s
Edição 2 (teste de armazenamento compartilhado):
Qnap (SSD Raid10):
dd if=/run/sr-mount/23d45731-c005-8ad6-a596-bab2d12ec6b5/01ce9f2e-c5b1-4ba8-b783-d3a5c1ac54f0.vhd of=/dev/null bs=1024 count=1000000
1000000+0 records in
1000000+0 records out
1024000000 bytes (1.0 GB) copied, 11.2902 s, 90.7 MB/s
MSA (ataque HP MSA-DP+):
dd if=/dev/mapper/3600c0ff000647bc2259a2f6101000000 of=/dev/null bs=1024 count=1000000
1000000+0 records in
1000000+0 records out
1024000000 bytes (1.0 GB) copied, 11.3974 s, 89.8 MB/s
Não mais que 1 rede Gigabit... Portanto, se eu transferir imagens de VM entre armazenamento compartilhado, o armazenamento local não estará envolvido. O openvswitch pode ser um gargalo?
Edição 3 (mais testes de disco):
sda = Raid10 de 6 x 146 GB 15k sas, sdb = um 146 GB 15k SAS em raid0
dd if=/dev/sdb of=/dev/null bs=1024 count=1000000
1000000+0 records in
1000000+0 records out
1024000000 bytes (1.0 GB) copied, 16.5326 s, 61.9 MB/s
[14:35 xcp-ng-em ssh]# dd if=/dev/sdb of=/dev/null bs=512k count=1000
1000+0 records in
1000+0 records out
524288000 bytes (524 MB) copied, 8.48061 s, 61.8 MB/s
[14:36 xcp-ng-em ssh]# dd if=/dev/sdb of=/dev/null bs=512k count=10000
10000+0 records in
10000+0 records out
5242880000 bytes (5.2 GB) copied, 84.9631 s, 61.7 MB/s
[14:37 xcp-ng-em ssh]# dd if=/dev/sda of=/dev/null bs=512k count=10000
10000+0 records in
10000+0 records out
5242880000 bytes (5.2 GB) copied, 7.03023 s, 746 MB/s