Introduction

Semi-recently I replaced my main self-built SuperMicro based server with a Huawei FusionServer 2288Hv3. The role of this server is running Proxmox, with a large ZFS storage array as well as participating in a SSD-only Ceph cluster. However, after the upgrade I noticed that especially the VMs running off the Ceph cluster had very poor write performance. The reads could still saturate my 10Gbit network, but especially the Windows VMs would crawl to a sub 1MB/s write speed when doing any kind of 'Windows-y' updates. In the Proxmox UI I could see that the OSDs running off the new Huawei server were having very high (400-800ms) apply latencies. I suspected I hadn't properly updated the disk controller in the Huawei server so this posts serves to describe my process in solving the performance by flashing the RAID/HBA chip to IT mode.

The Setup

The original server that the Huawei replaced was a Sandy Bridge 3 core E3 with a low end LSI HBA card with some 24GB of RAM. The new FusionServer 2288Hv3 runs a cool 48 Haswell cores with 128GB of RAM. Interestingly, the CPU that this server is provided with has CPUs I don't recognize: E5-2658A v3, which I was unable to find any information about, there seem to be no A suffix servers around. I belive all the specs match the normal E5-2658A v3. There is one (post)[https://community.intel.com/t5/Processors/What-does-quot-A-quot-suffix-in-Xeon-E5-2697A-v4-mean/m-p/283009] on the Intel community forum where someone claims 'Besides that information, the "A" means active, and that indicates that the processor needs and active fan to work properly.' which seems ridiculous.

This Huawei server comes with a 'SR310' RAID card, but if we do a lspci we can see that it is just a rebadged LSI SAS3008 chip, as it is normally the case in these OEM servers.

lspci -v

01:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
        Subsystem: Broadcom / LSI SAS9311-8i

The disks I run under Ceph are an Intel S3500 (800GB) and Intel D3-S4510 (2TB).

Initial measurements

As the performance issue is with writes and I wanted to be very sure to not accidentally write over any of my data I performed my tests from a network booted Linux with all my normal disks detached (also needed for the flashing of the HBA with minimal risk)

As all my actually good disks had data on them, I had to perform my testing with older SSDs I had laying around so I could do destructive write tests.This disk was a very old Samsung 60GB SLC SSD (MCCOE64G5MPQ-0VA03). It is a high quality SLC SSD, but with performance matching its age. Not the biggest deal, but that means that these numbers are not indicative of the performance that my production SSDs would obtain.

Important for measuring SSD performance under Ceph with fio is the --direct=1 and --fsync=1 flags, as Ceph uses those flags when writing to disk. This means that the SSD is forced to write to disk rather then letting it cache writes. I am unsure of how this interacts with RAID cards, but for SSDs without powerloss protection (aka most consumer SSDs) this absolutely destroys performance.

First a quick test with a single job, to get a baseline.

root@in-chimp:/mnt/sda# fio --filename=/dev/sda --direct=1 --fsync=1 --rw=write --bs=4k --numjobs=1 --iodepth=1 --runtim
e=60 --time_based --group_reporting --name=4k-sync-write-sda
4k-sync-write-sda: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=3767KiB/s][w=941 IOPS][eta 00m:00s]
4k-sync-write-sda: (groupid=0, jobs=1): err= 0: pid=6144: Fri May 12 15:03:49 2023
  write: IOPS=919, BW=3680KiB/s (3768kB/s)(216MiB/60007msec); 0 zone resets
    clat (usec): min=81, max=299, avg=131.87, stdev=22.27
     lat (usec): min=82, max=300, avg=132.49, stdev=22.31
    clat percentiles (usec):
     |  1.00th=[   98],  5.00th=[  105], 10.00th=[  112], 20.00th=[  119],
     | 30.00th=[  123], 40.00th=[  124], 50.00th=[  126], 60.00th=[  128],
     | 70.00th=[  133], 80.00th=[  145], 90.00th=[  157], 95.00th=[  194],
     | 99.00th=[  202], 99.50th=[  204], 99.90th=[  208], 99.95th=[  210],
     | 99.99th=[  215]
   bw (  KiB/s): min= 3288, max= 3888, per=100.00%, avg=3682.62, stdev=120.99, samples=119
   iops        : min=  822, max=  972, avg=920.66, stdev=30.25, samples=119
  lat (usec)   : 100=1.32%, 250=98.68%, 500=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=753, max=43633, avg=947.05, stdev=892.89
    sync percentiles (usec):
     |  1.00th=[  824],  5.00th=[  848], 10.00th=[  857], 20.00th=[  873],
     | 30.00th=[  881], 40.00th=[  889], 50.00th=[  898], 60.00th=[  906],
     | 70.00th=[  906], 80.00th=[  914], 90.00th=[  922], 95.00th=[  938],
     | 99.00th=[ 1696], 99.50th=[ 2409], 99.90th=[18744], 99.95th=[21103],
     | 99.99th=[33817]
  cpu          : usr=1.81%, sys=4.50%, ctx=165608, majf=0, minf=13
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,55202,0,55202 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=3680KiB/s (3768kB/s), 3680KiB/s-3680KiB/s (3768kB/s-3768kB/s), io=216MiB (226MB), run=60007-60007msec

Disk stats (read/write):
  sda: ios=0/110213, merge=0/0, ticks=0/56493, in_queue=106674, util=99.93%

Only ~900 IOPS

With a bunch more jobs, we can eek out performance (and some fine tuning of the jobs would maybe yield more):

root@in-chimp:/mnt/sda# fio --filename=/dev/sda --direct=1 --fsync=1 --rw=write --bs=4k --numjobs=20 --iodepth=1 --runti
me=60 --time_based --group_reporting --name=4k-sync-write-sda
4k-sync-write-sda: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.28
Starting 20 processes
Jobs: 20 (f=20): [W(20)][100.0%][w=14.5MiB/s][w=3706 IOPS][eta 00m:00s]
4k-sync-write-sda: (groupid=0, jobs=20): err= 0: pid=5593: Fri May 12 15:00:52 2023
  write: IOPS=4047, BW=15.8MiB/s (16.6MB/s)(949MiB/60003msec); 0 zone resets
    clat (usec): min=105, max=83598, avg=1741.91, stdev=1558.34
     lat (usec): min=105, max=83599, avg=1742.61, stdev=1558.34
    clat percentiles (usec):
     |  1.00th=[  906],  5.00th=[  963], 10.00th=[ 1012], 20.00th=[ 1106],
     | 30.00th=[ 1221], 40.00th=[ 1352], 50.00th=[ 1483], 60.00th=[ 1647],
     | 70.00th=[ 1844], 80.00th=[ 2147], 90.00th=[ 2540], 95.00th=[ 2868],
     | 99.00th=[ 6390], 99.50th=[ 7439], 99.90th=[25822], 99.95th=[31851],
     | 99.99th=[52691]
   bw (  KiB/s): min=12480, max=19432, per=100.00%, avg=16226.02, stdev=73.98, samples=2380
   iops        : min= 3120, max= 4858, avg=4056.50, stdev=18.49, samples=2380
  lat (usec)   : 250=0.07%, 500=0.04%, 750=0.04%, 1000=8.57%
  lat (msec)   : 2=67.47%, 4=22.13%, 10=1.41%, 20=0.12%, 50=0.15%
  lat (msec)   : 100=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=827, max=85519, avg=3189.79, stdev=3728.69
    sync percentiles (usec):
     |  1.00th=[ 1680],  5.00th=[ 1860], 10.00th=[ 1975], 20.00th=[ 2147],
     | 30.00th=[ 2245], 40.00th=[ 2376], 50.00th=[ 2540], 60.00th=[ 2737],
     | 70.00th=[ 2933], 80.00th=[ 3294], 90.00th=[ 3785], 95.00th=[ 4555],
     | 99.00th=[24511], 99.50th=[31065], 99.90th=[48497], 99.95th=[63701],
     | 99.99th=[71828]
  cpu          : usr=0.49%, sys=1.39%, ctx=493513, majf=0, minf=426
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,242846,0,242830 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=15.8MiB/s (16.6MB/s), 15.8MiB/s-15.8MiB/s (16.6MB/s-16.6MB/s), io=949MiB (995MB), run=60003-60003msec

Disk stats (read/write):
  sda: ios=106/388330, merge=0/97592, ticks=137/1017120, in_queue=1076797, util=99.97%

To constrast, here is a quick test with the direct and fsync flags removed, letting the RAID card and SSD both do their caching:

root@in-chimp:/mnt/sda# fio --filename=/dev/sda --rw=write --bs=4k --numjobs=1 --iodepth=1 --runtime=60 --time_based --g
roup_reporting --name=4k-sync-write-sda
4k-sync-write-sda: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=73.3MiB/s][w=18.8k IOPS][eta 00m:00s]
4k-sync-write-sda: (groupid=0, jobs=1): err= 0: pid=6454: Fri May 12 15:06:22 2023
  write: IOPS=109k, BW=427MiB/s (448MB/s)(25.0GiB/60001msec); 0 zone resets
    clat (usec): min=2, max=17872, avg= 8.77, stdev=267.36
     lat (usec): min=2, max=17872, avg= 8.82, stdev=267.36
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[    3], 80.00th=[    4], 90.00th=[    6], 95.00th=[    7],
     | 99.00th=[    9], 99.50th=[   11], 99.90th=[   21], 99.95th=[   44],
     | 99.99th=[14091]
   bw (  KiB/s): min=65472, max=1321648, per=100.00%, avg=440267.16, stdev=532902.11, samples=119
   iops        : min=16368, max=330412, avg=110066.77, stdev=133225.54, samples=119
  lat (usec)   : 4=80.08%, 10=19.34%, 20=0.48%, 50=0.06%, 100=0.01%
  lat (usec)   : 250=0.01%, 1000=0.01%
  lat (msec)   : 10=0.01%, 20=0.04%
  cpu          : usr=10.66%, sys=30.89%, ctx=2736, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,6557558,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=427MiB/s (448MB/s), 427MiB/s-427MiB/s (448MB/s-448MB/s), io=25.0GiB (26.9GB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=0/7313, merge=0/929259, ticks=0/171926, in_queue=171926, util=83.78%

109k IOPS. Obviously this is fully running cached and longer sustained writes would likely still start to see slow down, but that is not quite the point of this

Flashing the HBA to IT Mode

With some baseline measurements done it was time to flash the 'SR310' to HBA mode.

First, lets see what the firmware versions are of the device using sas3ircu:
sas3ircu output

root@in-chimp:/home/ubuntu# ./sas3ircu 0 display
Avago Technologies SAS3 IR Configuration Utility.
Version 17.00.00.00 (2018.04.02)
Copyright (c) 2009-2018 Avago Technologies. All rights reserved.

Read configuration has been initiated for controller 0
------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
  Controller type                         : SAS3008
  BIOS version                            : 8.05.00.00
  Firmware version                        : 2.00.00.00
  Channel description                     : 1 Serial Attached SCSI
  Initiator ID                            : 0
  Maximum physical devices                : 255
  Concurrent commands supported           : 3072
  Slot                                    : Unknown
  Segment                                 : 0
  Bus                                     : 1
  Device                                  : 0
  Function                                : 0
  RAID Support                            : Yes
------------------------------------------------------------------------

An absolutely abysmal Firmware version of 2, where as there are much much newer versions available of the LSI firmware.

root@in-chimp:/home/ubuntu# ./sas3flash -listall
Avago Technologies SAS3 Flash Utility
Version 15.00.00.00 (2016.11.17)
Copyright 2008-2016 Avago Technologies. All rights reserved.

        Adapter Selected is a Avago SAS: SAS3008(C0)

Num   Ctlr            FW Ver        NVDATA        x86-BIOS         PCI Addr
----------------------------------------------------------------------------

0  SAS3008(C0)  02.00.00.00    02.06.00.02    08.05.00.00     00:01:00:00

root@in-chimp:/home/ubuntu# ./sas3flash -list
Avago Technologies SAS3 Flash Utility
Version 15.00.00.00 (2016.11.17)
Copyright 2008-2016 Avago Technologies. All rights reserved.

        Adapter Selected is a Avago SAS: SAS3008(C0)

        Controller Number              : 0
        Controller                     : SAS3008(C0)
        PCI Address                    : 00:01:00:00
        SAS Address                    : 568a828-6-5a2a-e000
        NVDATA Version (Default)       : 02.06.00.02
        NVDATA Version (Persistent)    : 02.06.00.02
        Firmware Product ID            : 0x2721 (IR)
        Firmware Version               : 02.00.00.00
        NVDATA Vendor                  : LSI
        NVDATA Product ID              : SAS3008
        BIOS Version                   : 08.05.00.00
        UEFI BSD Version               : 02.00.00.00
        FCODE Version                  : N/A
        Board Name                     : SAS3008
        Board Assembly                 : N/A
        Board Tracer Number            : N/A

        Finished Processing Commands Successfully.
        Exiting SAS3Flash.

Trying to do it from linux yields can't flash IR over IT

booted to BIOS:DOS with everything on a USB

erase 6

Performance after

After flash

root@in-chimp:/home/ubuntu# fio --filename=/dev/sda --rw=write --bs=4k --numjobs=1 --iodepth=1 --runtime=10 --time_based --group_reporting --name=4k-sync-write-sda
4k-sync-write-sda: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [f(1)][100.0%][eta 00m:00s]
Jobs: 1 (f=1): [f(1)][100.0%][eta 00m:00s]

Jobs: 1 (f=1): [f(1)][100.0%][eta 00m:00s]

Jobs: 1 (f=1): [f(1)][100.0%][eta 00m:00s]
4k-sync-write-sda: (groupid=0, jobs=1): err= 0: pid=4603: Fri May 12 18:47:29 2023
  write: IOPS=353k, BW=1378MiB/s (1445MB/s)(13.5GiB/10001msec); 0 zone resets
    clat (nsec): min=2264, max=590462, avg=2572.50, stdev=765.73
     lat (nsec): min=2306, max=590524, avg=2615.55, stdev=767.53
    clat percentiles (nsec):
     |  1.00th=[ 2288],  5.00th=[ 2320], 10.00th=[ 2320], 20.00th=[ 2320],
     | 30.00th=[ 2320], 40.00th=[ 2320], 50.00th=[ 2352], 60.00th=[ 2352],
     | 70.00th=[ 2416], 80.00th=[ 2512], 90.00th=[ 3440], 95.00th=[ 3632],
     | 99.00th=[ 5152], 99.50th=[ 6304], 99.90th=[ 9152], 99.95th=[11712],
     | 99.99th=[14400]
   bw (  MiB/s): min= 1108, max= 1405, per=100.00%, avg=1382.63, stdev=69.19, samples=19
   iops        : min=283758, max=359788, avg=353952.21, stdev=17711.50, samples=19
  lat (usec)   : 4=97.28%, 10=2.65%, 20=0.07%, 50=0.01%, 100=0.01%
  lat (usec)   : 250=0.01%, 750=0.01%
  cpu          : usr=26.39%, sys=73.59%, ctx=23, majf=0, minf=16
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,3527725,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=1378MiB/s (1445MB/s), 1378MiB/s-1378MiB/s (1445MB/s-1445MB/s), io=13.5GiB (14.4GB), run=10001-10001msec

Disk stats (read/write):
  sda: ios=29/231, merge=0/29718, ticks=22/4608, in_queue=4630, util=14.47%

root@in-chimp:/home/ubuntu# fio --filename=/dev/sda --rw=write --bs=4k --numjobs=1 --direct=1 --fsync=1 --iodepth=1 --runtime=60 --ti
me_based --group_reporting --name=4k-sync-write-sda
4k-sync-write-sda: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=3907KiB/s][w=976 IOPS][eta 00m:00s]
4k-sync-write-sda: (groupid=0, jobs=1): err= 0: pid=4770: Fri May 12 18:51:14 2023
  write: IOPS=986, BW=3948KiB/s (4042kB/s)(231MiB/60001msec); 0 zone resets
    clat (usec): min=78, max=239, avg=126.21, stdev=14.14
     lat (usec): min=78, max=239, avg=126.81, stdev=14.20
    clat percentiles (usec):
     |  1.00th=[   91],  5.00th=[  100], 10.00th=[  106], 20.00th=[  120],
     | 30.00th=[  123], 40.00th=[  124], 50.00th=[  125], 60.00th=[  127],
     | 70.00th=[  130], 80.00th=[  137], 90.00th=[  147], 95.00th=[  151],
     | 99.00th=[  163], 99.50th=[  165], 99.90th=[  169], 99.95th=[  172],
     | 99.99th=[  180]
   bw (  KiB/s): min= 3496, max= 4200, per=100.00%, avg=3951.13, stdev=135.04, samples=119
   iops        : min=  874, max= 1050, avg=987.78, stdev=33.76, samples=119
  lat (usec)   : 100=4.77%, 250=95.23%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=691, max=36875, avg=879.36, stdev=740.21
    sync percentiles (usec):
     |  1.00th=[  750],  5.00th=[  775], 10.00th=[  783], 20.00th=[  807],
     | 30.00th=[  816], 40.00th=[  824], 50.00th=[  824], 60.00th=[  832],
     | 70.00th=[  832], 80.00th=[  840], 90.00th=[  848], 95.00th=[  865],
     | 99.00th=[ 2311], 99.50th=[ 3687], 99.90th=[11207], 99.95th=[18482],
     | 99.99th=[33162]
  cpu          : usr=1.88%, sys=4.73%, ctx=177645, majf=0, minf=14
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,59215,0,59215 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=3948KiB/s (4042kB/s), 3948KiB/s-3948KiB/s (4042kB/s-4042kB/s), io=231MiB (243MB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=49/118201, merge=0/0, ticks=32/56957, in_queue=107148, util=99.97%

root@in-chimp:/home/ubuntu# fio --filename=/dev/sda --rw=write --bs=4k --numjobs=8 --direct=1 --fsync=1 --iodepth=1 --runtime=60 --ti
me_based --group_reporting --name=4k-sync-write-sda
4k-sync-write-sda: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.28
Starting 8 processes
Jobs: 8 (f=8): [W(8)][100.0%][w=8980KiB/s][w=2245 IOPS][eta 00m:00s]
4k-sync-write-sda: (groupid=0, jobs=8): err= 0: pid=4841: Fri May 12 18:52:38 2023
  write: IOPS=2449, BW=9797KiB/s (10.0MB/s)(574MiB/60002msec); 0 zone resets
    clat (usec): min=98, max=37560, avg=1180.78, stdev=934.21
     lat (usec): min=98, max=37561, avg=1181.44, stdev=934.21
    clat percentiles (usec):
     |  1.00th=[  816],  5.00th=[  832], 10.00th=[  857], 20.00th=[  898],
     | 30.00th=[  922], 40.00th=[  955], 50.00th=[  996], 60.00th=[ 1020],
     | 70.00th=[ 1090], 80.00th=[ 1270], 90.00th=[ 1876], 95.00th=[ 2040],
     | 99.00th=[ 2671], 99.50th=[ 6390], 99.90th=[16057], 99.95th=[20579],
     | 99.99th=[31327]
   bw (  KiB/s): min= 8832, max=10848, per=100.00%, avg=9812.17, stdev=46.78, samples=952
   iops        : min= 2208, max= 2712, avg=2453.04, stdev=11.69, samples=952
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=51.19%
  lat (msec)   : 2=42.48%, 4=5.54%, 10=0.62%, 20=0.08%, 50=0.07%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=327, max=41605, avg=2075.18, stdev=1206.68
    sync percentiles (usec):
     |  1.00th=[ 1598],  5.00th=[ 1647], 10.00th=[ 1680], 20.00th=[ 1713],
     | 30.00th=[ 1745], 40.00th=[ 1778], 50.00th=[ 1827], 60.00th=[ 1876],
     | 70.00th=[ 1991], 80.00th=[ 2114], 90.00th=[ 2737], 95.00th=[ 2933],
     | 99.00th=[ 7177], 99.50th=[ 7832], 99.90th=[21365], 99.95th=[22676],
     | 99.99th=[33817]
  cpu          : usr=0.70%, sys=1.86%, ctx=305541, majf=0, minf=147
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,146956,0,146951 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=9797KiB/s (10.0MB/s), 9797KiB/s-9797KiB/s (10.0MB/s-10.0MB/s), io=574MiB (602MB), run=60002-60002msec

Disk stats (read/write):
  sda: ios=72/282046, merge=0/11342, ticks=90/448658, in_queue=508091, util=99.88%

Conclusion

I think there are 2 main lessons, vendors are bad at providing updates and IR mode is really bad.