1 job

ceph@ceph8:fio-2.1.3$ ./fio 1job4k.fio
rand-read: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
fio-2.1.3
Starting 1 process
jobs pid=25332 started
scott:Jobs: 1 (f=1): [r] [100.0% done] [796.2MB/0KB/0KB /s] [204K/0/0 iops] [eta 00m:00s]

rand-read: (groupid=0, jobs=1): err= 0: pid=25332: Tue Jun  7 11:02:36 2016
  read : io=8,183.2MB, bw=843,944KB/s, iops=210,985, runt=  9,929msec

    slat (usec): min=0, max=119, avg= 1.01, stdev= 1.01
    clat (usec): min=0, max=115, avg= 0.29, stdev= 0.65
     lat (usec): min=1, max=120, avg= 1.37, stdev= 1.24

    clat percentiles (usec):
     |  1.00th=[    0],  5.00th=[    0], 10.00th=[    0], 20.00th=[    0],
     | 30.00th=[    0], 40.00th=[    0], 50.00th=[    0], 60.00th=[    0],
     | 70.00th=[    0], 80.00th=[    1], 90.00th=[    1], 95.00th=[    1],
     | 99.00th=[    1], 99.50th=[    1], 99.90th=[    1], 99.95th=[    1],
     | 99.99th=[    4]
    lat (usec) : 2=99.97%, 4=0.02%, 10=0.01%, 20=0.01%, 100=0.01%
    lat (usec) : 250=0.01%
  cpu          : usr=35.36%, sys=64.56%, ctx=861, majf=0, minf=7
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2,094,879/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=8,183.2MB, aggrb=843,943KB/s, minb=843,943KB/s, maxb=843,943KB/s, mint=9,929msec, maxt=9,929msec


8 jobs

ceph@ceph8:fio-2.1.3$ ./fio 1job4k.fio
rand-read: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
...
rand-read: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
fio-2.1.3
Starting 8 processes
jobs pid=25340 started
jobs pid=25341 started
jobs pid=25342 started
jobs pid=25343 started
jobs pid=25344 started
jobs pid=25345 started
jobs pid=25346 started
jobs pid=25347 started
scott:Jobs: 8 (f=1): [rrrrrrrr] [100.0% done] [518.1MB/0KB/0KB /s] [133K/0/0 iops] [eta 00m:00s]
rand-read: (groupid=0, jobs=8): err= 0: pid=25340: Tue Jun  7 11:08:00 2016
  read : io=5,073.1MB, bw=519,936KB/s, iops=129,983, runt=  9,993msec

    slat (usec): min=0, max=2689, avg= 1.82, stdev= 3.24
    clat (usec): min=0, max=1988, avg= 0.50, stdev= 2.18
     lat (usec): min=1, max=2690, avg= 2.43, stdev= 3.93

    clat percentiles (usec):
     |  1.00th=[    0],  5.00th=[    0], 10.00th=[    0], 20.00th=[    0],
     | 30.00th=[    0], 40.00th=[    0], 50.00th=[    0], 60.00th=[    1],
     | 70.00th=[    1], 80.00th=[    1], 90.00th=[    1], 95.00th=[    1],
     | 99.00th=[    1], 99.50th=[    1], 99.90th=[    6], 99.95th=[   13],
     | 99.99th=[   25]
    lat (usec) : 2=99.62%, 4=0.04%, 10=0.27%, 20=0.05%, 50=0.02%
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%
    lat (msec) : 2=0.01%
  cpu          : usr=8.76%, sys=27.21%, ctx=3965847, majf=0, minf=60
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1,298,929/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=5,073.1MB, aggrb=519,935KB/s, minb=519,935KB/s, maxb=519,935KB/s, mint=9,993msec, maxt=9,993msec