PS F:\> ./fio.exe --thread --filename=fiotmp --verify_pattern=0xdeadbeef --rw=write --verify=pattern --size=100m --stone wall --bs=1m --name=layout --do_verify=0 --name=trim --rw=trim --bs=4k --name=verify --verify_only=1 --unlink=1 layout: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=windowsaio, iodepth=1 trim: (g=1): rw=trim, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=windowsaio, iodepth=1 verify: (g=2): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=windowsaio, iodepth=1 fio-3.5-94-g946f9-dirty Starting 3 threads fio: got pattern '00', wanted 'de'. Bad bits 6,w=100MiB/s,t=100MiB/s][r=0,w=100,t=25.6k IOPS][eta 49710d:06h:28m:14s] fio: bad pattern block offset 0 pattern: verify failed at file fiotmp offset 0, length 0 fio: verify type mismatch (0 media, 18 given) verify: No I/O performed by windowsaio, perhaps try --debug=io option for details? fio: pid=14824, err=42/file:io_u.c:2106, func=io_u_queued_complete, error=Illegal byte sequence layout: (groupid=0, jobs=1): err= 0: pid=10216: Thu Apr 5 22:51:38 2018 write: IOPS=3448, BW=3448MiB/s (3616MB/s)(100MiB/29msec) slat (usec): min=200, max=437, avg=243.36, stdev=44.15 clat (nsec): min=4362, max=76613, avg=39276.29, stdev=25882.35 lat (usec): min=206, max=443, avg=282.64, stdev=46.20 clat percentiles (nsec): | 1.00th=[ 4384], 5.00th=[ 4512], 10.00th=[ 4640], 20.00th=[ 4832], | 30.00th=[ 8640], 40.00th=[51968], 50.00th=[56064], 60.00th=[56576], | 70.00th=[57600], 80.00th=[59136], 90.00th=[62720], 95.00th=[64256], | 99.00th=[72192], 99.50th=[76288], 99.90th=[76288], 99.95th=[76288], | 99.99th=[76288] lat (usec) : 10=36.00%, 50=3.00%, 100=61.00% cpu : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=0 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,100,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 trim: (groupid=1, jobs=1): err= 0: pid=9132: Thu Apr 5 22:51:38 2018 trim: IOPS=70.3k, BW=275MiB/s (288MB/s)(100MiB/364msec) slat (nsec): min=7075, max=62246, avg=8742.29, stdev=2846.12 clat (nsec): min=257, max=101530, avg=5016.15, stdev=5310.88 lat (usec): min=10, max=127, avg=13.76, stdev= 6.35 clat percentiles (nsec): | 1.00th=[ 3408], 5.00th=[ 3792], 10.00th=[ 3888], 20.00th=[ 3920], | 30.00th=[ 3984], 40.00th=[ 4016], 50.00th=[ 4048], 60.00th=[ 4128], | 70.00th=[ 4384], 80.00th=[ 4576], 90.00th=[ 4832], 95.00th=[ 5600], | 99.00th=[31872], 99.50th=[51968], 99.90th=[59648], 99.95th=[62720], | 99.99th=[82432] lat (nsec) : 500=0.03%, 750=0.02%, 1000=0.01% lat (usec) : 2=0.03%, 4=36.73%, 10=60.27%, 20=1.05%, 50=1.25% lat (usec) : 100=0.62%, 250=0.01% cpu : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=0 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,0,25600,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 verify: (groupid=2, jobs=1): err=42 (file:io_u.c:2106, func=io_u_queued_complete, error=Illegal byte sequence): pid=14824: Thu Apr 5 22:51:38 2018 read: IOPS=333, BW=333MiB/s (350MB/s)(1024KiB/3msec) slat (nsec): min=14986, max=14986, avg=14986.00, stdev= 0.00 clat (nsec): min=1211.1k, max=1211.1k, avg=1211065.00, stdev= 0.00 lat (nsec): min=1226.1k, max=1226.1k, avg=1226051.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[ 1205], 5.00th=[ 1205], 10.00th=[ 1205], 20.00th=[ 1205], | 30.00th=[ 1205], 40.00th=[ 1205], 50.00th=[ 1205], 60.00th=[ 1205], | 70.00th=[ 1205], 80.00th=[ 1205], 90.00th=[ 1205], 95.00th=[ 1205], | 99.00th=[ 1205], 99.50th=[ 1205], 99.90th=[ 1205], 99.95th=[ 1205], | 99.99th=[ 1205] lat (msec) : 2=0.99% cpu : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=0 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=33.3%, 4=66.7%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,100,0,0 short=0,0,0,0 dropped=0,0,0,0 Run status group 0 (all jobs): WRITE: bw=3448MiB/s (3616MB/s), 3448MiB/s-3448MiB/s (3616MB/s-3616MB/s), io=100MiB (105MB), run=29-29msec Run status group 1 (all jobs): TRIM: bw=275MiB/s (288MB/s), 275MiB/s-275MiB/s (288MB/s-288MB/s), io=100MiB (105MB), run=364-364msec Run status group 2 (all jobs): READ: bw=333MiB/s (350MB/s), 333MiB/s-333MiB/s (350MB/s-350MB/s), io=1024KiB (1049kB), run=3-3msec