1 job

ceph@ceph8:fio-2.1.3$ ./fio 1job4k.fio
rand-read: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=128
fio-2.1.3
Starting 1 process
jobs pid=25326 started
scott:Jobs: 1 (f=1): [r] [100.0% done] [941.8MB/0KB/0KB /s] [241K/0/0 iops] [eta 00m:00s]

rand-read: (groupid=0, jobs=1): err= 0: pid=25326: Tue Jun 7 11:01:54 2016
read : io=9503.2MB, bw=979870KB/s, iops=244967, runt= 9931msec

    clat (usec): min=0, max=100, avg= 0.79, stdev= 0.45
     lat (usec): min=0, max=100, avg= 0.83, stdev= 0.43

    clat percentiles (usec):
     | 1.00th=[    0], 5.00th=[    0], 10.00th=[    0], 20.00th=[    0],
     | 30.00th=[    1], 40.00th=[    1], 50.00th=[    1], 60.00th=[    1],
     | 70.00th=[    1], 80.00th=[    1], 90.00th=[    1], 95.00th=[    1],
     | 99.00th=[    1], 99.50th=[    1], 99.90th=[    2], 99.95th=[    4],
     | 99.99th=[    5]
    lat (usec) : 2=99.81%, 4=0.12%, 10=0.07%, 20=0.01%, 100=0.01%
    lat (usec) : 250=0.01%
cpu          : usr=35.68%, sys=64.24%, ctx=858, majf=0, minf=6
IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2432772/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=9503.2MB, aggrb=979869KB/s, minb=979869KB/s, maxb=979869KB/s, mint=9931msec, maxt=9931msec

8 jobs

ceph@ceph8:fio-2.1.3$ ./fio 1job4k.fio
rand-read: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=128
...
rand-read: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=128
fio-2.1.3
Starting 8 processes
jobs pid=25352 started
jobs pid=25353 started
jobs pid=25354 started
jobs pid=25355 started
jobs pid=25356 started
jobs pid=25357 started
jobs pid=25358 started
jobs pid=25359 started
scott:Jobs: 8 (f=3): [rrrrrrrr] [100.0% done] [298.9MB/0KB/0KB /s] [76.5K/0/0 iops] [eta 00m:00s]
rand-read: (groupid=0, jobs=8): err= 0: pid=25352: Tue Jun 7 11:08:52 2016
read : io=3047.7MB, bw=312170KB/s, iops=78042, runt= 9997msec

    clat (usec): min=0, max=3507, avg= 1.39, stdev= 4.30
     lat (usec): min=0, max=3507, avg= 1.54, stdev= 4.31

    clat percentiles (usec):
     | 1.00th=[    1], 5.00th=[    1], 10.00th=[    1], 20.00th=[    1],
     | 30.00th=[    1], 40.00th=[    1], 50.00th=[    1], 60.00th=[    1],
     | 70.00th=[    2], 80.00th=[    2], 90.00th=[    2], 95.00th=[    2],
     | 99.00th=[    2], 99.50th=[    4], 99.90th=[   13], 99.95th=[   19],
     | 99.99th=[   32]
    lat (usec) : 2=65.22%, 4=34.25%, 10=0.38%, 20=0.10%, 50=0.04%
    lat (usec) : 100=0.01%
    lat (msec) : 2=0.01%, 4=0.01%
cpu          : usr=7.20%, sys=29.93%, ctx=4989996, majf=0, minf=50
IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=780191/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=3047.7MB, aggrb=312170KB/s, minb=312170KB/s, maxb=312170KB/s, mint=9997msec, maxt=9997msec